Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giant

1 Introduction

Despite the remarkable performance of recent large and small language models (LLMs and SLMs), deploying them in resource-constrained environments remains challenging. Even models like Llama-3.3-70B (released in December 2024) and Llama-3.2-1B (released in September 2024) still involve billions of parameters, making them costly to run in both server and mobile-edge scenarios. Low-bit quantization has emerged as a popular solution to reduce the memory and computational overhead of these models. In particular, Post-Training Quantization (PTQ) [Frantar et al., 2022; Lin et al., 2024; Xiao et al., 2023; Micikevicius et al.,

∗Corresponding author

Appendix available at: https://arxiv.org/abs/2409.11055

2022] is widely adopted, as Quantization Aware Training (QAT) often requires extensive retraining [Zhu et al., 2023; Wan et al., 2023].

However, existing research on quantization has largely relied on perplexity-based metrics [Frantar et al., 2022; Lin et al., 2024; Xiao et al., 2023] and older benchmarks (e.g., ARC, HellaSwag, Winogrande, MMLU) [Yao et al., 2023; Dettmers and Zettlemoyer, 2023; Liu et al., 2023; Jin et al., 2024; Li et al., 2024; Dutta et al., 2024], which have become too easy for current models and risk data contamination in recently trained LLMs. Moreover, more recent architectures like Llama-3.3, Llama-3.2, and Llama-3.1 have not been thoroughly investigated. This gap includes extreme scales, from 1B to over 405B parameters, and omits detailed category-level analysis using LLM-as-judge evaluation methods. Additionally, there has been limited manual inspection to refine and confirm evaluation results [Li et al., 2024; Dutta et al., 2024].

In this paper, we present a comprehensive evaluation of how quantization affects instruction-tuned LLMs. In particular, we aim to address the following research questions (RQs): (RQ1) Do quantized LLMs outperform smaller original models in most benchmarks, and how do they perform across diverse architectures and small language models (SLMs)? (RQ2) How do different quantization methods influence performance across a broad range of tasks, and are there significant differences in how specific approaches (e.g., GPTQ, AWQ, SmoothQuant, FP8) affect task accuracy? (RQ3) In what ways do model size and architecture affect the accuracy of quantized models? (RQ4) Does higher task difficulty necessarily correlate with greater accuracy degradation under quantization? and (RQ5) How does quantization impact the free-form conversation quality of LLMs when evaluated using the MT-Bench framework, which relies on LLMs as judges?

We perform our evaluation in a multi-cluster GPU environment, as illustrated in Figure 1. This setup consists of four servers, each with a distinct GPU configuration, and ensures consistent measurement conditions across all experiments (details in Appendix D).

Our study applies four quantization methods-GPTQ [Frantar et al., 2022], AWQ [Lin et al., 2024], SmoothQuant [Xiao et al., 2023], and FP8 [Micikevicius et al., 2022]-to instruction-tuned models ranging from 1B to 405B parameters, including Vicuna [Zheng et al., 2023], Gemma [Team et al., 2024], and the Llama family [Dubey et al., 2024]. We evaluate these models on 13 datasets covering six task categories: commonsense Q&A, complex knowledge and language understanding, instruction following, hallucination detection, mathematics, and dialogue. Further dataset details are provided in Appendix A (Table 3).

To compare our results with ongoing community efforts, we synchronize our benchmarks with Huggingface OpenLLM Leaderboard-v1 (covering April 2023 to June 2024) and OpenLLM Leaderboard-v2 (launched on June 26, 2024). However, both versions currently provide only limited data on quantized models, highlighting the need for our comprehensive evaluation. Our key findings are as follows:

• Quantized LLMs generally perform better than smaller models on most benchmarks and maintain their advantage across different architectures, showing significant improvements in both large and small language models. However, they still struggle with instruction-following (IFEval) and detecting hallucinations (TruthfulQA).
• FP8 is the most reliable method for all model sizes and tasks, especially for LLMs with 405B parameters, where SmoothQuant encounters problems. AWQ usually performs better than GPTQ in weight-only quantization, and hardware support makes FP8 even more advantageous.
• In smaller LLMs, using 4-bit quantization can lead to significant accuracy drops, especially with GPTQ. However, 70B models usually maintain good performance when quantized to 4 bits. While model size is the main factor affecting quantization difficulty, differences in LLM architecture within the same parameter size can also affect accuracy. Nevertheless, AWQ consistently outperforms GPTQ across different tasks and model types.
• Difficult tasks do not always have the biggest accuracy drops when quantized. The impact depends on the model design and the quantization method used, causing some hard tasks to remain stable while some easy tasks see bigger decreases. Overall, quantization highlights a model's existing weaknesses, especially in commonsense, logical, or mathematical reasoning.
• Quantization greatly reduces performance in Coding and STEM tasks, although it sometimes improves reasoning

accuracy. Additionally, GPT4-based evaluators can sometimes incorrectly judge wrong answers as correct.

2 RelatedWork

Quantization for LLMs. There are two main types of quantization methods for LLMs: post-training quantization (PTQ) and quantization-aware training (QAT). Due to the size and training complexity of LLMs, QAT is challenging to apply, and as a result, only limited research has been conducted in this area. Consequently, the majority of quantization research for LLMs has focused on PTQ approaches [Zhu et al., 2023; Wan et al., 2023].

LLM.int8() [Dettmers et al., 2022] is a post-training quantization method that uses 8-bit weights and activations to reduce the memory footprint of large models while maintaining performance. This dynamically adapts to ensure sensitive components of the computation retain higher precision when needed. GPTQ [Frantar et al., 2022] is a layer-wise quantization that uses inverse Hessian information to reduce the number of bits per weight while maintaining low accuracy loss. AWQ [Lin et al., 2024] proposed that preserving a small portion of important weights is a key part of reducing quantization errors. As part of an activation-aware strategy, AWQ focused on channels with larger activation magnitudes and used per-channel scaling. SmoothQuant [Xiao et al., 2023] is a method that smooths activation outliers before quantization, improving robustness in large-scale models and enabling more effective 8-bit quantization. Outlier Suppression+ [Wei et al., 2023] reduces the impact of extreme outliers in activations, allowing for more efficient quantization by normalizing problematic values without degrading model accuracy. QLoRA [Dettmers et al., 2024] combines low-rank adaptation with quantization to achieve efficient fine-tuning of large models while minimizing computational costs and memory usage.

However, these quantization algorithm works have been evaluated only on basic datasets such as perplexity, ARC, and MMLU, which were released 2-3 years ago, and they do not sufficiently take into account the recent advancements in LLMs and SLMs. Therefore, for a safe application of quantization in LLM services, a more comprehensive performance analysis is necessary.

Evaluating LLMs. Several studies have explored the effects of model quantization on the performance of LLMs, focusing on various aspects. For instance, Yao et al. [Yao et al., 2023] investigated the impact of quantization on both weights and activations in language modeling tasks. In contrast, Liu et al. [Liu et al., 2023] concentrated solely on evaluating three emergent abilities of quantized LLMs, neglecting crucial tasks such as trustworthiness, dialogue, and long-context processing.

Hong et al. [Hong et al., 2024] expanded the scope by examining trustworthiness dimensions in the assessment of LLM compression techniques. However, most studies have predominantly relied on accuracy as the primary evaluation metric, with limited attention paid to alternative metrics. For example, Zhang et al. [Zhang et al., 2024] proposed additional evaluation metrics, including fluency, informativeness, coherence, and harmlessness, alongside accuracy.

Efforts to establish more comprehensive evaluation benchmarks have also been made. Jaiswal et al. [Jaiswal et al., 2023]

developed a benchmark from existing datasets to evaluate compressed models, while Li et al. [Li et al., 2024] and Jin et al. [Jin et al., 2024] assessed various quantization techniques across different tasks. Namburi et al. [Namburi et al., 2023] explored how compression and pruning affect the parametric knowledge of LLMs.

Dutta et al. [Dutta et al., 2024] proposed a new metric from the perspective of "Flip" errors, emphasizing the importance of evaluating model robustness by accounting for inconsistencies and reversals in predictions, thereby going beyond traditional accuracy-focused metrics. Kurtic et al. [Kurtic et al., 2024] investigated the accuracy-performance trade-offs in quantizing LLMs, evaluating formats like FP8, INT8, and INT4 across various tasks and proposing practical guidelines for efficient LLM deployment at different model scales.

Xu et al. [Xu et al., 2024] explored the challenges of evaluating multilingual LLMs across diverse languages and cultures, emphasizing the development of culturally and linguistically inclusive benchmarks for fair evaluation. Liu et al. [Liu et al., 2024] provided a comprehensive examination of the generalization ability, focusing on their performance across various tasks and datasets. This study highlights the importance of designing benchmarks and metrics that accurately reflect realworld applications while identifying the limitations of current evaluation strategies.

To our knowledge, no prior study has comprehensively examined the effects of quantization across a wide range of model sizes-from 1B to 405B parameters-encompassing both SLMs and LLMs, including the latest architectures such as Llama-3.1, Llama-3.2, and Llama-3.3. Furthermore, existing research has not conducted detailed category-level analyses through cross-architecture comparisons or employed manual inspection using LLM-as-judge qualitative methods. Additionally, no work has compared the trends observed in LLMas-judge (MT-Bench) evaluations with leaderboard results to identify new trends in quantization impacts.

3 Evaluation Procedure

To handle LLMs, which cannot be processed on a single server, and to ensure fast and reliable evaluations, we developed a structured evaluation pipeline based on a multi-node cluster setup. Figure 1 presents an overview of the implemented pipeline for evaluating quantized LLMs. The evaluated LLMs include the Vicuna, Gemma, and Llama families, ranging in size from 1B to 405B. Each model is quantized using GPTQ, AWQ, SmoothQuant, and FP8 methods. The evaluation is conducted using lm-eval and MT-Bench as benchmarking tools. The multi-node cluster used for evaluation is implemented with vLLM and consists of four servers: H100-80Gx8, A100-80Gx4, RTX 6000-48Gx4, and A6000-48Gx4. Additionally, the Huggingface library is integrated into the pipeline to support model hosting and benchmarking. The evaluation is distributed across a multi-cluster environment to ensure a thorough performance assessment. If vLLM cannot be used for processing, we used the Huggingface Accelerate library instead, which is slower but shows better comparability. The versions of all tools used are provided in Appendix D.1.

4 Experimental Setup

4.1 Datasets

We conducted a comprehensive evaluation of the quantized LLMs on widely adopted benchmarks, grouped into six main categories: CommonSenseQA (ARC, HellaSwag, Winogrande), Knowledge and Language Understanding (MMLU, GPQA, MMLU-PRO, BBH, MuSR), Instruction Following (IFEval), Hallucination (TruthfulQA), Mathematics (GSM8K, MATH-Lvl-5), and Dialogue (MT-Bench). Additional information about these datasets can be found in Appendix A, and an overview of all benchmarks is provided in Table 3 (Appendix).

4.2 Reproducibility Details

Both OpenLLM Leaderboard V1 and V2 follow the same methodology outlined on the HuggingFace Leaderboard's page, including identical normalization procedures. Additional information on leaderboard calculations is provided in Appendix D.2.

We used the greedy decoding strategy to maintain deterministic output tokens across runs. The detailed configuration for the greedy decoding strategy is presented in Appendix D.1, which also lists the specific versions of each package used. Furthermore, we record all random seeds for Python, NumPy, Torch, and few-shot setups in Appendix D.1, ensuring the reproducibility of our experimental results.

4.3 QuantizationMethods and Calibration Data

We evaluated multiple PTQ methods, including GPTQ, AWQ, SmoothQuant, and FP8. GPTQ and AWQ focus on weightonly quantization, while SmoothQuant and FP8 apply to both weights and activations. For a detailed overview of each quantization method, refer to Appendix D.3, with configuration details and group sizes described in Appendix D.4.

The selection and configuration of calibration datasets are crucial for maintaining consistent performance across models. We used default settings for the number of samples and sequence lengths, as detailed in Appendix D.5 and summarized in Table 9. These settings may vary depending on the specific algorithms and tools but generally ensure stable results for our experiments.

4.4 Models

We applied quantization techniques to 12 instruction-tuned open LLMs, including the Vicuna [Zheng et al., 2023], Gemma [Team et al., 2024], and Llama [Dubey et al., 2024] families, with model sizes ranging from 1B to 405B. These models were released between June 2023 and December 2024 and were downloaded from HuggingFace's model sources.

All models were evaluated using 13 benchmark datasets, applying GPTQ, AWQ, SmoothQuant, and FP8 quantization. However, due to runtime limitations (over 30 days), we did not measure the original model accuracy or test all GPTQ configurations for Llama-3.1-405B. Also, due to space limitations, the experimental results for the Vicuna and Gemma models are provided in the Appendix B

5 Experimental Results

This section presents experimental results addressing five research questions, detailing the performance impact of quantization across 13 datasets for three model families of varying sizes. Table 1 and Table 2 summarize the experimental results from OpenLLM Leaderboard-v1 and v2, respectively.

5.1 RQ1: Do quantized LLMs outperform smaller original models on most benchmarks, and how do they fare across different architectures and SLMs?

We observe that quantized LLMs generally outperform smaller, uncompressed models across a wide range of benchmarks. For instance, a 4-bit Llama-2-13B (6.5 GB) outperforms an FP16 Llama-2-7B (14 GB) on most tasks, despite its reduced size. However, in TruthfulQA (hallucination testing) and IFEval (instruction-following), the FP16 Llama-2-7B still performs better, indicating that quantization can compromise alignment and adherence to instructions.

Similarly, quantizing Llama-3.1-405B to 4 bits (202.5 GB) yields higher accuracy than the FP16 Llama-3.1-70B (140 GB) across various tasks; yet, the instruction-following IFEval benchmark again highlights a shortfall in the quantized model. This performance gap holds across different model architectures: although Llama-3.3-70B demonstrates improvements over Llama-3.1-70B, a 4-bit Llama-3.1-405B can still outperform the uncompressed Llama-3.3-70B.

In edge-focused SLMs, quantization produces significantly larger improvements. For example, quantizing Llama-3.2-3B (SmoothQuant) improves accuracy by 13.32% on OpenLLM Leaderboard-v1 and 8.72% on OpenLLM Leaderboard-v2 compared to the FP16-Llama-3.2-1B. Such improvements exceed the margin typically observed in larger models (7B to 405B).

RQ1 Findings: Quantized LLMs consistently outperform smaller models across most benchmarks and maintain this advantage across different architectures, with significant gains observed in both large models and edge-focused SLMs. However, tasks like instruction-following (IFEval) and hallucination detection (TruthfulQA) remain challenging for quantized models.

5.2 RQ2: How do different quantization methods affect the performance of models across diverse tasks? Are there noticeable differences in how specific methods (e.g., GPTQ, AWQ, SmoothQuant, FP8) impact task accuracy?

In most cases, weight-only quantization methods (GPTQ and AWQ) and activation quantization methods (SmoothQuant and FP8) exhibit similar performance. However, for Llama3.1-405B, SmoothQuant's activation quantization resulted in a significant accuracy drop compared to other methods, with an average decrease of up to 10.86% compared to FP8 on the OpenLLM Leaderboard-v2 datasets. This decline occurs because SmoothQuant was originally designed to handle the high activation ranges observed in models up to the size of OPT-175B. Consequently, at the 405B scale of Llama-3.1, the algorithm likely did not account for certain factors that are critical at this larger scale. In contrast, for smaller models, such as SLMs with 1B and 3B parameters, the average accuracy drop remains below 1%.

When comparing weight-only quantization methods, AWQ consistently outperforms GPTQ across various LLMs on overall benchmark scores. Additionally, different implementations of GPTQ, such as AutoGPTQ and llmcompressor, demonstrate notable performance differences, with the oldest GPTQ implementation library, AutoGPTQ, still maintaining stable and consistent performance.

When both weight and activation quantization are required at 8 bits, FP8 proves to be highly effective, even on challenging tasks from the OpenLLM Leaderboard-v2. This effectiveness spans models of all sizes, from the largest, such as Llama3.1-405B, to the smallest, like Llama-3.2-1B. Therefore, FP8 offers greater stability compared to SmoothQuant and is advantageous to use when supported by hardware such as the NVIDIA H100 GPU and RTX 6000 Ada, as it provides benefits in both latency and throughput. Additionally, applying FP8 to both weights and activations allows for a reduction of the KV cache size by half, which is highly beneficial during LLM decoding phases where I/O bottlenecks are a concern. This FP8 KV cache feature is supported by vLLM.

RQ2 Findings: FP8 is the most stable option across all model sizes and tasks, particularly in large LLMs where SmoothQuant performs poorly, whereas AWQ regularly outperforms GPTQ in weight-only quantization, and specialized hardware enhances FP8's advantages.

5.3 RQ3: How do model size and architecture influence the accuracy of quantized models?

We evaluated GPTQ across 13 datasets and observed that its accuracy can degrade significantly under 4-bit quantization-a behavior not reflected in perplexity-based evaluations alone. For smaller models, such as Llama-3.2-1B, 4-bit quantization causes particularly severe accuracy drops (e.g., -25.32% on GSM8k and -16.01% on IFEval). These declines tend to be more pronounced in GPTQ than in AWQ, suggesting that SmoothQuant or FP8 at 8-bit may be necessary to maintain accuracy for 1B-scale models.

With mid-sized models like Llama-3.1-8B (GPTQ-8bit), we noted average accuracy improvements of +2.51% and +2.11% over 4-bit on OpenLLM Leaderboard-v1 and OpenLLMLeaderboard-v2, respectively. However, at the 70B scale, 4-bit outperformed 8-bit by +1.89%, indicating a reverse trend for larger models. To examine architectural differences at a fixed scale, we compared Llama-2, Llama-3.1, and Llama-3.3 at the 70B size and found that AWQ consistently surpassed GPTQ, delivering stable accuracy even at 4-bit quantization.

RQ3 Findings: In smaller LLMs, 4-bit quantization often leads to significant accuracy loss (especially with GPTQ), whereas 70B-scale models can maintain stable performance with 4-bit. Although size primarily drives quantization difficulty, architectural variations within the same scale can also influence accuracy. Nonetheless, AWQ consistently outperforms GPTQ across diverse tasks and model families.

5.4 RQ4: Does higher task difficulty always correlate with greater accuracy degradation under quantization?

Contrary to common assumptions, tasks widely considered challenging (e.g., GSM8K, MMLU, Math-Lvl-5) did not consistently show the greatest performance drops when quantized. For instance, among 12 tested models quantized with AWQ, MMLU accuracy remained largely unchanged, with some models even exhibiting slight improvements. In contrast, tasks generally regarded as less knowledge-intensive (ARC-challenge, TruthfulQA, Winogrande) occasionally experienced declines exceeding -3%.

Rather than strictly depending on task difficulty, quantization appears to magnify existing weaknesses in a model's ability to handle specific forms of reasoning, such as commonsense or mathematical inference. As highlighted in RQ3, smaller models (2B-7B) are especially vulnerable to computational reasoning tasks like GSM8K, exhibiting sharper performance drops. Moreover, as noted in RQ2, different quantization methods can produce varying degrees of accuracy loss, making it difficult to draw definitive conclusions from a single task perspective.

RQ4 Findings: Difficult tasks do not always suffer the largest accuracy loss under quantization. The impact varies by model architecture and the chosen quantization method, leading some hard tasks to remain stable while easier tasks occasionally show bigger drops. In essence, quantization amplifies a model's existing weaknesses, particularly in commonsense, logical, or mathematical reasoning.

5.5 RQ5: How does quantization impact the free-form conversation quality of LLMs when evaluated using the MT-Bench framework, which employs LLMs as judges?

Category-Level Analysis. Figure 2 presents a detailed breakdown of three models (Llama-3.1, Llama-3.2, and Llama3.3) across MT-Bench categories. Quantized LLMs suffer the largest score degradation in Coding and STEM. For Coding, manual inspections reveal that GPT4 often assigns lower scores when the generated code contains fewer examples or insufficient comments. In STEM tasks, correctness matters when a definite solution exists, and concise logical explanations are important when an exact answer does not exist. Quantized models frequently either provided overly verbose justifications or produced incorrect statements, leading to lower scores. In contrast, the Reasoning category showed an increase of about 1 point with quantization. Manual checks reveal that concise answers tend to receive higher GPT4 scores, and quantized models often responded more concisely than their original counterparts. The detailed results of this manual inspection, along with the full text responses, can be found in Appendix C.4. Also, Table 6 in the Appendix C.1 lists category-wise MTBench scores for all models.

Limitations of GPT4-Based Evaluation. Consistent with the findings in MT-Bench, GPT4 sometimes misjudges incorrect responses as correct, particularly for math and reasoning tasks. Although reference-guided judging and chainof-thought prompting can mitigate such errors [Zheng et al., 2023], they do not eliminate them entirely. In Reasoning task, GPT4 erroneously considered a wrong answer correct, boosting quantized models' scores. Conversely, there were STEM questions where both original and quantized models provided accurate answers, yet GPT4 mistakenly marked them as incorrect. These misjudged cases are described in the Appendix C.5.

For models like Llama-3.3-70B, categories such as Humanities consistently scored near or at the maximum (10 points). In these cases, quantized versions also achieved scores in the high 9-point range, making it difficult to discern meaningful quality differences through manual inspection. This is because it is difficult to clearly understand the reasoning behind the results by only looking at GPT4's judgment statements, especially when the score differences are marginal, such as 1-2 points. Hence, for the latest large-scale models, more sensitive metrics or superior judging models may be required to evaluate subtle quality gaps.

Multi-Turn Analysis. Table 7 in the Appendix C.2 presents the average scores of multi-turn conversations across 12 models. Among smaller models (e.g., 2B, 7B, 8B), some exhibit slight score improvements; however, this trend is not consistent across all small-scale models. In contrast, larger models (e.g., 13B, 70B) generally experience score declines, although there are exceptions where AWQ enhances performance. Additionally, accuracy losses become more noticeable in the second turn of multi-turn interactions.

These observations indicate that establishing a clear trend based solely on model size or type is challenging, as the impact of quantization depends on a combination of factors, including model architecture, quantization method, and task complexity. Furthermore, when comparing quantization methods, AWQ typically outperforms GPTQ, which is consistent with the results observed on OpenLLM Leaderboard-v1 and v2.

Comparison with Leaderboard Results. The trends observed in MT-Bench do not always align with leaderboard outcomes. For instance, the Quantized Llama-3.1-405B model outperformed a newer Llama-3.3-70B-FP16 model on certain leaderboards, yet scored similarly or slightly lower in MTBench. Unlike the leaderboard tasks, which may not be particularly sensitive to Coding and Math challenges, the free-form and demanding nature of MT-Bench conversations highlights performance drops in more complex categories such as Coding and STEM. Thus, although computational metrics suggest that quantization does not uniformly degrade accuracy (RQ4), MTBench's qualitative LLM-based evaluation reveals significant performance reductions in tasks known to be difficult.

RQ5 Findings: We observe that quantization considerably reduces performance in Coding and STEM tasks, while occasionally improving reasoning. The impact of quantization on multi-turn conversation quality does not consistently correlate with model size or type Additionally, GPT4-based assessments sometimes misjudge incorrect answers as correct.

6 Conclusion

We evaluated instruction-tuned quantized LLMs across 13 datasets and 6 task types, using models ranging from 1B to 405B and 4 quantization methods, including GPTQ, AWQ, SmoothQuant, and FP8. We found that quantized LLMs generally outperformed smaller models in most tasks, except for hallucination detection and instruction-following. Performance varied by quantization method and precision, with weightonly quantization performing better in the 405B model. Task difficulty had little impact on accuracy loss. Our MT-Bench evaluation revealed that quantization significantly reduces performance in Coding and STEM tasks while occasionally enhancing reasoning. Additionally, GPT4-based assessments can misjudge incorrect answers as correct.

Acknowledgments

We thank the former Neubla ML team members - Minwook Ahn, Minho Park, Jinsol Kim, Raegeun Park, and Byonghwa Oh - for their valuable discussions and feedback. This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No.RS-2023-00277060, Development of open edge AI SoC hardware and software platform, No.RS-2024-00459797, Development of ML compiler framework for on-device AI, RS-2025-02214497, Development of low-level optimization programAPI technology for AI semiconductors)

Ethical Statement

There are no ethical issues.