Mapping the Efficiency Landscape of Small Language Models

1 Introduction

In the last years, large language models (LLMs) have become ubiquitous. They have evolved to a state where they can understand and produce coherent natural language across diverse domains, and are able to perform a wide range of tasks that could previously only be accomplished by humans, such as summarizing technical texts [K¨aser et al., 2025] or engaging in complex conversations [Maharana et al., 2024]. LLMs are at the core of the emerging field of agentic AI, which encompasses AI applications performing complex tasks with a certain degree of autonomy [Acharya et al., 2025; Mohammadi et al., 2025]. LLMs are also increasingly being used in industrial settings [Raza et al., 2025]. Companies use chatbots, for example, as an interface to customers, e.g. on their webpage [Szafarski et al., 2025; Bhattacharyya, 2024].

However, the substantial resource consumption of LLMs [Berthelot et al., 2025; Plociennik et al., 2025; Plociennik et al., 2026], in particular energy demand [Ji and Jiang, 2026], scarcity of compute power, especially GPUs [Strati et al., 2024], fear of lock-in with cloud servers in different parts of the world [Wang et al., 2025; Wang, 2025], along with privacy and sustainability concerns [Raza et al., 2025], pose significant entrance barriers for companies to adopt LLMs for their business. Moreover, the broad knowledge encoded in LLMs is often not needed for industrial applications, which usually require domain- and company-specific information [Wang et al., 2025]. Hence, in this paper we argue that small, locally deployable models can be an alternative to LLMs for most common industrial tasks. Since there is no consensus on what should be considered a small language model (SLM) yet [Wang et al., 2025], we adopt a pragmatically-oriented definition: We consider every model with at most 15B parameters an SLM since those are the models that can run on consumer hardware.

Adopting SLMs in industry requires longer-term commitment and investment security. Companies must be able to choose a model that performs reasonably well over time with limited resources. This applies especially to SMEs [Bourdin et al., 2024]. Hence, the question is whether SLM development has proceeded to a stable state for informed choices of models, and which benefits they offer over LLMs.

This paper aims to provide insights into the interactions between size, performance, and energy demand of SLMs. It therefore investigates the correlation between model parameter count, inference energy consumption and model performance for 75 SLMs over 5 different benchmark tasks to determine task-specific energy efficiency. The results are set into perspective with 2 LLMs for reference. In doing so, we particularly shed light on possible trade-offs between task performance and energy consumption. By including SLMs in a timeframe from 2023 to 2025, we also include a historical perspective on the development of operational energy efficiency. This question is particularly interesting for application developers who need to select a model and are still looking for the right time to commit to the effort of customizing a self-hosted SLM for lower operational costs. With this temporal analysis, we also analyze trends in the development of SLM performance in relation to the reference LLMs.

The main contributions of this paper are:

• It explores the relations between model size, performance, and energy consumption of SLMs.
• It presents the development of SLMs over time - from 2023 to 2025.
• It benchmarks a large number of SLMs (75) on 5 different tasks. The results can be viewed as a first guideline for task-specific model selection.

The results are presented comprehensively in Section 4, structured by the following research questions (RQs):

(RQ1) What relationships and trade-offs exist between model size, energy consumption and task performance in SLMs?

(RQ2) How have efficiency and task performance evolved across model sizes and over recent years, and how do they compare to reference LLMs?

The paper is structured as follows: Section 2 discusses related work. Section 3 introduces the methodology. Section 4 presents and discusses the results. Section 5 considers limitations, while Section 6 gives some general concluding remarks.

2 RelatedWork

Belcak et al. [2025] argue for the use of SLMs instead of LLMs in agentic AI applications. They point out that specialized or fine-tuned SLMs are sufficiently powerful, more operationally suitable, more flexible and more economical than LLMs in many usecases. They stress that well-designed SLMs can even outperform the task performance of much larger models. However, the choice between a customized SLM and a service LLM can be a multi-faceted decision in actual system design, as Koraag et al. [2025] emphasize in their report on the development of a specific application in financial reporting. They report the challenges regarding the implementation of both options and specifically regarding the selection of a suitable SLM. Corradini et al. [2025] present a systematic literature review on recent developments in the SLM field. They investigate the geographical distribution of SLM research and architectural trends, and find that MMLURedux is the most frequently used SLM benchmark in the literature, followed by GSM8K.

Recent studies have shown that the influence of model size on the performance of language models is not straightforward in the sense that "bigger is always better", challenging the LLM scaling law [Korga et al., 2025; Varoquaux et al., 2025]. Xiao et al. [2025] introduce the concept of capability density to characterize the relationship between model performance and efficiency. They find that inference costs decrease exponentially over time for equivalent performance. Lee et al. [2025] investigate the influence of different quantization methods on the performance of different size models of the Llama family. They find that performance varies by quantization method and the nature of the task, and performance degradation does not necessarily increase with task difficulty.

Berthelot et al. [2025] find that energy usage is not evenly distributed across the language model lifecycle: Over time, the major share of energy is required for inference, not for training. Hence, it is key to investigate the energy consumption of language models (LMs) during inference, which is being done in several studies. Samsi et al. [2023] study the energy consumption per second of different size models (7B to 65B) from the Llama family and investigate the energy consumption per decoded token for the largest model (65B parameters). They find that energy consumption per second correlates positively with model size, and that energy consumption per decoded token for the 65B model is higher if more GPUs are used for inference. This is due to the fact that only 20-25% of GPU memory is being used at any given time. Argerich and Pati˜no-Mart´ınez [2024] find that the energy consumption per token of a 3B parameter model is only about 2/3 of that of a 7B model. A comparison of the total energy consumption per prompt is done by Jeanquartier et al. [2026] with two SLMs. They suggest that the energy consumption should be reported based on task outcome instead of per token. However, their study is limited regarding the number of included models. They also argue to include the application context in the sustainability assessment of the model. Maliakel et al. [2025] investigate the tradeoffs of performance vs. energy across 5 models of different sizes and families and 4 different benchmarks. They find that larger models consume more energy while being more accurate, and that both model architecture and input characteristics influence efficiency. Saad-Falcon et al. [2025] introduce a metric called intelligence per watt (IPW). They argue that computing transitioned from mainframes to PCs due to efficiency improvements, and that language models are about to face a similar shift - from cloud to local inference. They benchmark 20+ local LMs in terms of accuracy and energy consumption, and find that local LMs have become 5 times more energy-efficient in the last two years. Pham et al. [2025] introduce SLM-Bench, a benchmark for SLMs. They evaluate 15 SLMs on 9 NLP tasks using 23 datasets from different domains, including metrics for energy consumption and CO2 emissions. They find that there are trade-offs among the dimensions correctness, computational efficiency, and energy consumption. Hence, accuracy gains come at the cost of a higher energy usage. These are interesting results, despite the fact that the choice of benchmarks does not include coding or function calling tasks. Furthermore, the limited model selection hinders a systematic analysis of SLMs, which differ in capabilities and behavior.

3 Method

To systemically evaluate the trade-offs between task performance, model size and energy consumption across SLMs, we conducted controlled inference experiments on a large collection of SLMs and selected LLMs. The setup allows comparison across diverse tasks and model families including the analysis of efficiency differences.

3.1 Models

In the experiments we included 75 open-source SLMs. The models either do not reason, or have reasoning disabled to allow for fair comparison. The evaluated SLMs span the time period July 2023 to December 2025 and were selected to represent widely adopted and actively developed model families from major industrial and academic contributors. These include the following model families: Qwen (Alibaba), Llama (Meta AI), Gemma (Google), Mistral and Ministral (Mistral AI), OpenELM (Apple), Apertus (ETH Z¨urich), RNJ (Essential AI), Granite (IBM), SmolLM (Hugging Face), and Phi (Microsoft). All SLMs were executed using Hugging Face's transformers library on a single NVIDIA RTXA6000, deployed in a SLURM cluster, with bfloat (BF16) precision and minor model-specific loading adjustments where required. The analysis is model-centric rather than architectureisolated, reflecting complete deployed models.

To provide a reference point for comparison, we included the open-source LLMs DeepSeek V3.1 and Mistral Large 3. These models were selected for their recent release and strong performance. Both were deployed via vLLM on a cluster of 8 x NVIDIA H200 GPUs with INT8 quantization and standard vLLM hyperparameters. This method was used as it is technically not possible to deploy the LLMs like the SLMs and this represents a standard deployment for LLMs. The comparison of the single-GPU setup of the SLMs and the multi-GPU setup of the LLMs is therefore based on a realistic deployment scenario. All models were drawn from Hugging Face and are publicly available for full reproducibility.

3.2 Datasets

Models were evaluated on five benchmark datasets representing diverse tasks: General Knowledge (MMLU-Redux) [Gema et al., 2025], Math (GSM8K) [Cobbe et al., 2021], Function Calling (Berkeley Function Calling V3, in the following BFC) [Patil et al., 2025], Coding (BigCodeBench) [Zhuo et al., 2025] and Text Summarization (XSum) [Narayan et al., 2018]. From each benchmark 1000 singleturn tasks were randomly sampled. For MMLU-Redux, GSM8K, BFC and BigCodeBench, performance was measured using accuracy, with BigCodeBench requiring full pass/fail execution of provided unit tests. Accuracy is defined as the fraction of correct answers over all inputs. XSum was evaluated using mean cosine similarity between generated and reference summaries, computed with the all-MiniL6-v2 embedding model [Wang et al., 2020]. All model answers, except for XSum, were post-processed using custom, benchmark-specific answer extraction rules; primarily using regular expressions.

3.3 Experimental Setup

We evaluated single model inference on a single input, and each input was evaluated only once. All models received identical user prompts and custom, benchmark-specific system prompts describing the task. In case that models do not support native tool-calling, the tool specifications were added to the system prompt with a concise usage description. This allows us to analyze older models without native function calling as well. Inference was performed deterministically with fixed random seeds and without sampling. Models may generate output until a model-specific stop sequence occurs with a maximum of 2048 output tokens.

3.4 EnergyMeasurements

Inference-time energy consumption was measured per input, assuming an already loaded model, using CodeCarbon [Lottick et al., 2019]. The measurement window covered only the prefilling and generation step (as this uses most energy), excluding tokenization, pre- and post-processing. Analyses were focused on GPU, as it dominates energy consumption and this approach minimizes interference from background processes.

4 Results &Analysis

In this section, we present our results, structured by focused research questions.

4.1 RQ1: What relationships and trade-offs exist between model size, energy consumption and task performance in SLMs?

To analyze the relation of those three dimensions, we look at the reported performance and energy measurements as shown in Table 1. The performance varies substantially among the SLMs and while there is a general trend of better performance from bigger models, there are notable exceptions. For example, in GSM8K and BigCodeBench, the best performing models are Qwen-3-4B (93.2%) and RNJ-1-8B (55.6%), respectively, and not the most recent, largest SLMs. In addition, the differences across model sizes in the same model family are in several cases minor, e.g. Qwen-3. This suggests diminishing performance gains with increasing model sizes for some tasks and model families.

The energy consumption varies heavily among SLMs and model sizes. As expected, if models with the same number of parameters have a similar mean output length, the energy consumption is similar as well. However, the output lengths necessary to complete a task can differ widely among SLMs resulting in drastically varying energy consumption. This can result in smaller SLMs that consume more energy than bigger ones. Taking each SLM and comparing it to each strictly larger SLM shows that the smaller model consumes more energy than the larger one in 37.73% (BFC), 24.01% (GSM8K), 42.97% (MMLU-Redux), 32.12% (XSum) and 14.41% (BigCodeBench) of cases of each benchmark. This indicates that model size or joules per token alone are not sufficient parameters to determine efficient SLMs. It is necessary to consider output length or in the best case, the energy consumption as well.

Examining the relationship between energy consumption and performance reveals that higher energy demands do not reliably translate into better performance. There is no benchmark where the best-performing SLM is also the one with the highest consumption. In GSM8K, for example, the bestperforming SLM, namely Qwen-3-4B, consumes less energy than the next eight best contenders. Similar examples are scattered over all benchmarks and performance levels. However, there is an interesting phenomenon in that the highest consuming SLMs are mostly among the worst performing ones. This appears to be due to the extremely high output length. This is especially pronounced for the model family OpenELM, e.g. in XSum the one-sentence summarization has a mean output length of up to 641.77 tokens (OpenELM1-0.450B), drastically exceeding the typical length of one sentence. The high output length and low performance indicate minor instruction following as well as task capabilities. This further underlines the role of output length and energy consumption in model selection.

Given the inconsistent relation between the three dimensions, comparing models along a single dimension is insufficient. In practice, efficiency in SLMs is a multi-objective problem in which the goal is high performance and low energy consumption. To systematically capture these trade-offs and to identify models with balanced performance-energy combinations, we analyze the Pareto frontier in Figure 1. This frontier exhibits a non-linear relationship across all benchmarks. While initial increases in energy yield clear performance gains, they start to saturate quickly resulting in noticeable diminishing returns, which set in after a moderate energy budget is reached; achieving further improvements requires a substantial amount of additional energy. For example, regarding BFC, improving accuracy by 1 percentage point from Qwen-3-8B to Qwen-3-14B nearly doubles inference energy consumption. This pattern of diminishing returns is particularly expressed in XSum and BFC, where nearmaximum performance is reached at relatively low energy budgets, and additional energy consumption yields little benefit. In more reasoning-intensive tasks such as GSM8K and MMLU-Redux, the performance continues to improve with higher energy consumption, although diminishing returns remain clearly visible.

The Pareto frontier analysis also illustrates these relations by identifying the SLMs that show the best combination of performance and energy consumption. Interestingly, only a small subset of models lies on the Pareto frontier for any of the five benchmarks, and the majority are Pareto dominated. This means that alternative models exist that can achieve higher performance with lower or comparable energy consumption. Importantly, we can again identify several larger SLMs that outperform their smaller counterparts in terms of efficiency. Although this is counterintuitive, it can be explained by differences in output length as already described.

Despite the focus on SLMs, it is important to place the results regarding efficiency in the broader context of SLMs vs. LLMs. As Table 1 shows, the efficiency benefits of SLMs become particularly apparent. LLMs achieve marginal performance differences while consuming dramatically more energy in nearly all benchmarks. For example, in GSM8K, comparing the best-performing LLM, MistralLarge-3-675B, to the best-performing SLM, Qwen-3-4B, the LLM consumes approximately 32× the energy for only about a 2% absolute accuracy improvement. This indicates huge efficiency benefits of SLMs in contrast to LLMs. However, it must be noted that due to the differences in deployment, the LLMs' results may vary, though this nonetheless remains a strong indication.

RQ1 Findings: The relations between model size, performance, and energy consumption are complex and, in some cases, unintuitive. Model size alone is a weak predictor for energy consumption and performance, as smaller models can outperform larger ones but can also consume more energy. The main driver for this effect is the output length necessary for a specific SLM to complete a task, which can cause substantial efficiency differences even among SLMs with similar model sizes. The Pareto frontier highlights that only a few SLMs achieve optimal performance-energy trade-offs, while the majority are heavily Pareto dominated. Furthermore, it shows that initial increases in energy yield substantial performance gains, but diminishing returns occur with higher energy consumption. These results reveal that efficient SLM selection requires informed, task-aware model selection rather than simple size-driven decisions.

4.2 RQ2: How have efficiency and task performance evolved across model size and over recent years, and how do they compare to reference LLMs?

To answer this research question, we analyze both the temporal evolution of task performance across model sizes and the changes in inference efficiency. Figure 1 shows the yearly Pareto frontiers from 2023 to 2025 regarding energy vs. performance. Figure 2 depicts the temporal development of task accuracy across different model size categories. Together, these perspectives allow us to assess not only whether SLMs improve over time, but whether such improvements translate into more favorable energy-performance trade-offs. For the analysis, we focus on models that are Pareto optimal in regard to the trade-off accuracy to energy consumption in their release year (Figure 1) or in regard to accuracy to release date in a given model size group (Figure 2).

Across all benchmarks, we observe substantial efficiency improvements from 2023 to 2024, reflected by a clear outward shift of the Pareto frontier in Figure 1. For reasoningand code-intensive benchmarks like MMLU-Redux, BFC, and BigCodeBench, this shift is primarily driven by the emergence of new top-performing models rather than by uniform efficiency gains across all energy budgets. In particular, models with very low energy consumption exhibit only limited changes over time, whereas the high-accuracy end of the frontier expands noticeably. For all three benchmarks, the highest-accuracy models originate from 2025, indicating that recent progress mainly manifests through frontier extensions rather than broad efficiency improvements.

GSM8K shows a similar pattern with a pronounced outward shift of the Pareto frontier from 2023 to 2024. Notably, the frontiers cross each other, indicating that one model from 2023 achieves Pareto optimality over a group of 2024 Pareto optimal models. In 2025, the outward shift of the Pareto frontier is less pronounced, indicating a transition from broad efficiency improvements toward more incremental gains.

XSum shows a distinct behavior. Performance saturates early across all size categories, with most SLMs already matching the LLM reference points by 2024. Efficiency improvements are less pronounced, with no clear gains from

2024 to 2025. This suggests a task- or metric-specific ceiling, where further improvements become increasingly difficult to realize, particularly at low energy budgets.

The temporal performance analysis across model sizes in Figure 2 shows that progress is not solely driven by scale, but manifests differently across benchmarks. For MMLURedux, GSM8K, and BFC, models with at least 3B parameters exhibit consistent performance gains from 2024 to 2025, converging toward similar accuracy levels and, in the case of GSM8K and MMLU-Redux, approaching the LLM reference points. Smaller models (≤1B parameters) follow a similar trend over time but retain a considerable performance gap relative to larger models, indicating that methodical improvements do not fully eliminate scale effects for these tasks.

BigCodeBench exhibits a more differentiated pattern. While performance improves across all model size categories, clear gaps between sizes persist, and only a single mid-sized model (RNJ-1-8B) reaches performance comparable to the LLM reference points. This is in notable contrast to the previously mentioned convergence across sizes for XSum.

Overall, these results indicate a shift from broad efficiency improvements between 2023 and 2024 toward more frontierdriven progress in 2025. While some benchmarks show signs of saturation, particularly XSum, recent SLMs increasingly achieve state-of-the-art performance on reasoning, coding, and function-calling tasks - often at substantially lower energy consumption than reference LLMs.

RQ2 Findings: SLMs exhibit strong temporal improvements in both task performance and inference efficiency. From 2023 to 2024, most benchmarks show a clear outward shift of the energy-performance Pareto frontier, indicating substantial efficiency gains. From 2024 to 2025, improvements become more task and energy-budget dependent, with signs of saturation in tasks like summarization, suggesting that further frontier advances become increasingly difficult to realize under strict energy constraints, particularly for very low-energy models. Reasoning, coding, and function-calling benchmarks show continued progress, particularly driven by recent gains in smaller models. Overall, methodical advances increasingly offset scale disadvantages, allowing SLMs to approach or exceed LLM performance at significantly lower energy costs.

5 Limitations

Our study has the following limitations: First, energy measurements depend on the specific hardware configuration, runtime stack, batching behavior, and decoding parameters; absolute values may vary across systems, although relative trends should remain stable. Furthermore, surrounding pipeline components were excluded and may offer further optimization potential. Second, prompt design and evaluation protocols influence output length and accuracy and may indirectly affect energy measurements. Third, we evaluate a fixed snapshot of publicly available models and do not account for continued fine-tuning, distillation, or hardware-aware optimizations that could further shift the efficiency-performance frontier. Fourth, to enable a fair comparison between models, we only investigate non-reasoning LMs - reasoning models generate extra tokens for thinking. Hence, they tend to use more energy while exhibiting a better performance. Fifth, the quality of some of the benchmarks themselves is limited. For example, the ground truths in BFC can be flawed, and some of the tasks make certain assumptions without encoding them in the prompts/instructions for the models. And last, some benchmarks are several years old - XSum, for example, was published in 2018. This implies that some (especially newer) models might have been trained on some of the benchmark data. This phenomenon is known as Benchmark Data Contamination and is hard to eliminate [Xu et al., 2024].

6 Conclusion and FutureWork

In this paper, we have evaluated a wide range of SLMs from the years 2023 to 2025 in five different tasks in terms of performance, energy consumption, and efficiency with two popular open-source LLMs as reference points. We find that model size and energy consumption are not directly linked to performance, higher energy usage does not necessarily lead to better performance, and smaller models are not automatically more efficient. The main reason is that models need varying output lengths to answer tasks.

Analyzing the efficiency of SLMs, a non-linear Pareto frontier becomes apparent with diminishing returns at higher levels of energy consumption. Furthermore, improvements from 2023 to 2025 can be found regarding efficiency. However, this progress slows down, indicating practical limits and occasionally possible benchmark saturation. With respect to model size, performance gains can be observed among all sizes over recent years. SLMs are even approaching the performance of current frontier LLMs, making them viable replacements for some tasks. Future work may extend the comparison framework used in this study to experimental setups in which SLMs are first fine-tuned and optimized for concrete use cases. Additionally, a more complete investigation into the whole model life cycle would extend this line of research into the broader context of sustainable and circular AI development. Such strategies may reveal further performance and efficiency increases in SLMs, particularly if SLMs are customized for specific domains like agentic AI or industrial applications. Furthermore, application-oriented research may yield task-specific, efficient frontier SLMs. With more stable results, it is now also becoming easier to give industrial practitioners reliable advice as to which model to employ for a given use case and domain.

Overall, our paper demonstrates rapid advances in both efficiency and performance of SLMs. The results show that already today's SLMs are very capable, but model selection must be task-specific and efficiency-focused to fully benefit from the advantages of SLMs.

Reproducibility

All code to conduct the experiments is publicly available at https://github.com/DFKI/SLM-Efficiency-EvalSuite.