MPPQ: Enhancing Post-Training Quantization for LLMs via Mixed Supervision, Proxy Rounding, and Pre-Searching

1 Introduction

Large language models (LLMs) [OpenAI, 2023; Touvron et al., 2023a; Touvron et al., 2023b] excel at complex natural language processing (NLP) tasks, yet they also come with high computational resource and parameter storage demands. Consequently, it is almost not feasible to deploy LLMs on constrained hardware. Quantization, especially post-training quantization (PTQ) [Dettmers et al., 2022; Xiao et al., 2023; Shao et al., 2024], has recently gained preference from both industry and academia due to its ability to operate with limited computational overhead and calibration data.

∗Dong Wang is the corresponding author

The PTQ of LLMs presents unique challenges, along with the inherent flaws of the technique. On the one hand, outlier channels [Dettmers et al., 2022] in the intermediate activations of LLMs are matters that need to be taken seriously. A feasible solution [Xiao et al., 2023; Wei et al., 2023; Shao et al., 2024; Zeng et al., 2024; Zhao et al., 2024] is to migrate it to the corresponding weights, and then apply clipping in weight quantization to relieve the impact of outliers. In terms of quantization tuning, prevalent PTQ methods [Shao et al., 2024] for LLMs generally utilize the mean squared error (MSE) [Choukroun et al., 2019] between the outputs of full-precision and quantized blocks as a supervised signal for updating learnable parameters. This optimization technique, however, offers limited supervision and falls short in aligning block-level and layer-level output activations. On the other hand, AdaRound [Nagel et al., 2020] demonstrated that the round-to-nearest (RTN) operation is suboptimal and proposed a learnable quantization rounding in the form of element-wise addition. Meanwhile, recent progress [Han et al., 2015; Lee et al., 2023b; Lee et al., 2024] also argued that weights with large magnitudes are relatively important and should be allowed to be quantized to discrete values further from themselves. Clearly, existing PTQ for LLMs methods [Xiao et al., 2023; Shao et al., 2024; Ma et al., 2024; Zeng et al., 2024; Zhao et al., 2024; Liu et al., 2024a] have not yet given sufficient consideration to this. Moreover, [Gong et al., 2024] suggested that asymmetric quantization should be paired with asymmetric clipping, and vice versa. Unfortunately, previous methods often incorrectly initialized the weight clipping factor, leading to an abnormal combination of quantization and clipping patterns.

To this end, we propose MPPQ, an accurate and efficient low-bitwidth PTQ framework tailored for LLMs. Specifically, MPPQ first introduces a novel Mixed Metric Supervision (MMS) approach that integrates multi-granularity and multi-dimensional regularization into block-wise reconstruction. In terms of granularity, it includes both block-wise and layer-wise loss, and in terms of dimension, it considers both magnitude similarity and direction similarity. Such approach enhances the consistency of quantized blocks with full-precision while also leading to more robust parameter learning. Inspired by the Low-Rank Adaptation (LoRA) [Hu et al., 2022] paradigm, we propose a Proxy-based Adaptive Rounding (PAR) scheme that employs two learnable lowrank matrices and optimizes them under the MMS. By performing the Hadamard product of the low-rank matrices with the corresponding weights, each element of the weights can be modified with varying intensities relative to their individual magnitudes. Notably, this scheme rectifies the inaccuracies caused by the RTN operator and leverages the efficiency of LoRA fine-tuning. Furthermore, our MPPQ proposes a factor coarse pre-searching (FCP) mechanism. The mechanism engages a grid search algorithm to iteratively identify the optimal clipping factors based on minimizing quantization errors across each weight layer, before the training begins. In this way, the proper combination of quantization and clipping patterns is ensured, while also promoting the stable convergence of weight clipping factors in optimization.

The contributions of this paper are summarized as follows:

1. We propose a novel mixed metric supervision approach under block-wise reconstruction, ensuring the distributions of quantized blocks align with full-precision counterparts while providing better regularization for parameter learning.
2. A proxy-based adaptive rounding scheme for weight quantization is constructed with the aim of reducing the quantization error caused by the RTN operator without incurring any extra inference overhead.
3. We introduce a factor coarse pre-searching mechanism for better weight clipping, which ensures the correct combination of quantization and clipping patterns and provides a better initial learning state for parameters.
4. Extensive experiments across models and datasets verify the robust performance of MPPQ, particularly in extremely low quantization bitwidths (e.g., W4A4, W6A6, and W2A16) scenarios. For example, our average accuracy for the 4/4-bit LLaMA-2-7B model is 54.87% (2.17% ↑ vs 52.70% in ABQ-LLM). Similarly, on the W2A16 configuration of LLaMA-13B, our perplexity on the C4 dataset is 9.79 (2.67 ↓ vs 12.46 in AffineQuant).

2 RelatedWork

2.1 Network Quantization

Quantization means to represent the weights, activations, and even gradients with low-bitwidth, leading to smaller models and faster inference. This technique can be largely divided into two categories: quantization-aware training (QAT) and post-training quantization. Thanks to the straight-through estimator (STE) [Bengio et al., 2013], QAT is capable of meeting most of the demands but requires high GPU effort and leverages the entire dataset. PTQ offers a more friendly alternative, enabling to quickly determine the optimal quantizer for a network using a small amount of data and has recently brought competitive outcomes. AdaRound [Nagel et al., 2020] showed that the RTN operation is not often the optimal solution and proposed reconstructing in a manner based on per-layer learning. BRECQ [Li et al., 2021] extended AdaRound to per-block reconstruction by utilizing the Fisher information. [Wei et al., 2022a] proposed QDrop, which randomly discards quantized activation values during PTQ to further enhance low-bit performance. PD-Quant [Liu et al.,

2023] incorporated global prediction differences into reconstruction and adjusted the distribution of activations to alleviate the issue of overfitting.

2.2 Quantization on LLMs

LLMs quantization can be broadly divided into weight-only quantization [Frantar et al., 2022; Lin et al., 2024; Cheng et al., 2023; Kim et al., 2024; Chee et al., 2024; Lee et al., 2023a] and weight-activation quantization [Dettmers et al., 2022; Yao et al., 2022; Yuan et al., 2023; Xiao et al., 2023; Wei et al., 2023; Shao et al., 2024; Liu et al., 2024b; Zeng et al., 2024; Zhao et al., 2024] based on the quantization objects. The former typically employs techniques such as compensation, scaling, rotation, and mixed-precision to compress weights. However, matrix operations at the hardware level are still executed with high precision, meaning that weight-only quantization fails to truly accelerate the inference of LLMs. Conversely, the latter can invoke specific hardware units during the inference phase, but outliers in activations are also a tough issue that must be addressed. To this end, SmoothQuant [Xiao et al., 2023] utilized mathematical equivalent transformations to shift outliers to corresponding weights. [Wei et al., 2023] further expanded channel-wise shifting while [Shao et al., 2024] employed a differentiable approach to learn the optimal parameters mentioned above, as well as weight clipping factors. QLLM [Liu et al., 2024a] proposed to reallocate the outliers to other channels to mitigate their impact on the quantization range. LRQuant [Zhao et al., 2024] sought better quantization performance by refining the initialization of the learnable smoothing parameters. In a notable difference, our proposed MPPQ focuses on loss construction, rounding optimization, and pattern matching to overcome the poor performance of existing works in extremely low-bitwidth quantization.

3 Preliminaries

3.1 b-bit Quantization

In this paper, we adopt hardware-friendly uniform asymmetric quantization for both weights and activations. For the matrix W ∈ Rm×n with floating-point values and a quantization bitwidth of b, the quantization and de-quantization procedures can be expressed as follows:

where s and zp represent the quantization step size and the zero point, respectively:

⌊·⌉ represents the RTN function and clamp(x, xmin, xmax) denotes the operation that constrains the value x within [xmin, xmax]. Weight-only quantization enables memory footprint savings, while weight-activation quantization can further accelerate inference. Previous studies [Dettmers et al., 2022; Xiao et al., 2023; Wei et al., 2022b] have revealed that several outlier channels are prevalent in the activations of LLMs, which enlarges the quantization range and thus hinders the accurate mapping of floating-point values with varying magnitudes. Therefore, the current consensus is that what must be done before activation quantization is to suppress the widespread outliers.

3.2 Scaling Transformation

Given the input activation Xl-1 ∈ Rm×n , a mainstream approach to suppress outliers is to first determine a diagonal matrix Λ ∈ Rn×n , based on the heuristic principle. Then, left-multiply it by the weight and right-multiply the activation with its inverse, respectively, to ensure computational invariance.

By inserting such transformations into LLMs, the impact of outliers on the quantization errors is effectively mitigated. In parallel, Λ can be incorporated intoW⊤, thereby eliminating the runtime burden. The diagonal element Λi within Λ is computed by the following formula:

where α represents the migration strength and is often set manually. Building on this point, [Wei et al., 2023] further introduced an additional channel-wise shifting operation:

where δ ∈ R1×n and ∆ ∈ R1×n represent matrices composed of per-channel shifting and scaling, respectively.

4 Approach

In this section, we present the details of our proposed quantization framework. The overall pipeline of MPPQ is illustrated in Figure 1 and mainly consists of three components.

MPPQ adheres to scaling transformation [Xiao et al., 2023; Wei et al., 2023; Shao et al., 2024] for handling outliers in activation, then it performs the Hadamard product with the weight and two learnable low-rank matrices, in anticipation of altering the quantization rounding of the weights. This is followed by determining the optimal initial values of weight clipping factors using grid search. Finally, MPPQ utilizes the proposed mixed metric supervision to calculate quantization loss and update all learnable parameters until convergence.

4.1 MixedMetric Supervision

Previous findings [Li et al., 2021; Wei et al., 2022a] have suggested that it is necessary to optimize layers with dependencies together in PTQ. Based on this, a block-wise quantization reconstruction paradigm has been widely applied in current PTQ methods for LLMs. The paradigm typically employs MSE to compute the loss between the outputs of the full-precision and quantized blocks, and then uses it as the sole metric to guide the quantization process:

where F(·) represents the mapping function for a transformer block in LLMs, Qw(·) and Qa(·) are the weight and activation quantizers, respectively, with parameters θw and θa. ∥·∥F denotes the Frobenius Norm.

Eq. (6) can be further simplified to:

where Xfp and Xq are the inputs of the full-precision and the quantized block, respectively, which also correspond to the outputs of their respective previous blocks. Weights are omitted in Eq. (7).

• (a) 18th transformer block.
• (b) 23th transformer block.

Figure 2: Visualization of the output distribution of different blocks from LLaMA-2-7B on WikiText2 under W4A4 quantization.

As observed in Figure 2, the significant difference between the output distributions of full-precision and quantized blocks suggests that relying solely on MSE loss for block-wise quantization reconstruction may not be sufficient to effectively align them. To address this, we propose a novel reconstruction loss function based on mixed metric supervision to impose stronger constraints on the distributional similarity between the layer-wise and block-wise outputs, thereby mitigating quantization errors.

Concretely, we first align the input of the quantized transformer block with its full-precision counterpart and devise a novel variant that differs from the losses used in prior works:

Next, for Eq. (7) and Eq. (8), we add cosine similarity (CS) loss to capture the directional differences as well as to achieve a more robust parameter optimization.

Furthermore, we impose additional constraints on the feature representation of certain layers within the quantized block, ensuring that its overall distribution is closer to that of the full-precision block while helps to smooth outliers in the feature space. In the end, we integrate these loss functions to formulate the final optimization objective Ltotal as follows:

where O(1) and O(2) represent the loss functions. The common procedure of both, as well as the part unique to O(2) are as follows, respectively:

In which, F and Q denote the full-precision and quantized block outputs, with fi and qi representing the full-precision and quantized outputs at layer i, respectively. ∥·∥2 denotes the Euclidean Norm. DKL(·) represents the Kullback-Leibler divergence [Kullback and Leibler, 1951] and σ is the softmax function.

4.2 Proxy-Based Adaptive Rounding

PTQ typically assigns each weight to its nearest fixed-point representation, whereas AdaRound [Nagel et al., 2020] has demonstrated that this approach is not optimal. It introduced a method to adaptively determine the best rounding for each weight of the model through back-propagation, as illustrated below:

where W and ˆW represent the pre-trained weight and its quantized version, respectively. h(V) ∈ {0, 1}m×n is the learnable rounding matrix and its size must be consistent with W. Directly applying AdaRound to LLMs quantization is not feasible, as LLMs typically have billions of parameters, which makes solving h(V) on a small calibration dataset challenging. Moreover, previous studies [Lee et al., 2024; Han et al., 2015] have demonstrated that the fixed-point representations of weights with large magnitudes in LLMs should be allowed to deviate more from their floating-point values. Therefore, there is an urgent need to propose a new rounding method within the aforementioned limitations.

Inspired by the parameter sharing of the LoRA structure, we explore a proxy-based adaptive rounding scheme for weight quantization. As shown in Figure 1, two learnable low-rank matrices are deployed and their product serves as the proxy rounding matrix, which is then merged with the weight by performing the Hadamard product. PAR achieves element-wise scaling of weights with varying intensities prior to quantization, and thus is able to adjust the quantization rounding values accordingly for each weight.

The aforementioned process can be summarized as follows:

In which, P is the proxy rounding matrix that is defined as Pij = exp(B·A)ij . B ∈ Rm×r and A ∈ Rr×n are low-rank matrices with a rank of r and initialized by random Gaussian and zeros, respectively. γ serves for balancing.

On the flip side, the gradient of Ltotal with respect to Pij can be derived as follows 1 :

As expected, Eq. (14) indicates that the scaling degree of each pre-trained weight is proportional to its own magnitude, which matches the findings of [Lee et al., 2023b]. In other words, the largerWij is, the greater the chance that ˆWij deviates fromWij. Notably, after the reconstruction, the lowrank matrices can be absorbed by the network without introducing any extra inference costs.

1 The detailed proof can be found in Appendix A.

Algorithm 1 Factor Coarse Pre-searching

Input: Pre-trained FP transformer block

Parameter: Number of weight layers N, grid search iterations m, starting search point p, initial loss and clipping factors L∗ , θ∗max, and θ∗min

Output: Optimal clipping factors θ∗max and θ∗min

1: Let L∗ to +∞, θ∗max and θ∗min to -1. 2: while n < N do 3: while j <mdo 4: Calculate the θmax and vmax by Eq. (15) and Eq. (16). 5: while k <mdo 6: Calculate the θmin and vmin by Eq. (15) and Eq. (17). 7: Quantize weight layer n based on Eq. (1). 8: Calculate L using Eq. (18). 9: if L∗ > L then 10: L∗ ← L, θ∗max ← θmax, θ∗min ← θmin. 11: end if 12: end while 13: end while 14: end while 15: return θ∗max and θ∗min

4.3 Factor Coarse Pre-searching

Reducing errors caused by outliers in weights or activations in the quantization of LLMs is crucial. Previous studies, including [Shao et al., 2024], have implemented weight clipping with clipping factors that are symmetrically initialized by hand. Obviously, this method fails to consider the crucial compatibility between quantization and clipping patterns, as well as whether the clipping factors are in the optimal initial state before training, which is also highly questionable. In this section, we present a factor coarse pre-searching mechanism for weight quantization, which can serve as a preoperation and skillfully overcome the aforementioned issues.

The running procedure of the mechanism is summarized in Algorithm 1 and can be roughly divided into four steps. We first exploit a grid search algorithm and set its hyperparameters, including the total number of iteration rounds m and the starting search point denoted as p ∈ [0, 1]. Next, the maximum and minimum values of all quantization intervals for the weight W to be updated under the current iteration round λ ∈ [0, m).

whereWi represents the i-th quantization interval. θimax and θimin represent its maximum and minimum clipping factors, respectively. Subsequently, we use the Eq. (1) to quantize Wi and then calculate the quantization error using the MSE distance:

The final step involves adjusting the optimal maximum or minimum clipping factor of the quantization interval if the quantization error in the current round is less than that in the previous round.

5 Experiments

5.1 Settings

Models and Datasets. We evaluate MPPQ on LLaMA-1 (7B, 13B), LLaMA-2 (7B, 13B), and OPT 2 (125M, 1.3B, 2.7B, 6.7B, 13B) models. In zero-shot question-answering (QA) tasks, we report the accuracy of models on several datasets: PIQA [Bisk et al., 2020], ARC [Clark et al., 2018], HellaSwag [Clark et al., 2018], WinoGrande [Sakaguchi et al., 2021], and BoolQ [Clark et al., 2019]. For language generation tasks, we measure the perplexity (PPL) on datasets including WikiText2 [Merity et al., 2016] and C4 [Raffel et al., 2020].

Quantization Settings. In terms of the quantization granularity, we employ per-token and per-channel quantization for activation and weight, respectively. Regarding the objects of quantization, we perform weight-only and weightactivation quantization. Meanwhile, the attention probabilities are maintained in full-precision for alignment with prior research. We also focus on quantization at low-bitwidth, including W2A16 and W4A4, among others.

Baseline Methods. Several recently prominent methods have been selected for benchmarking against MPPQ to highlight its superiority, including GPTQ [Frantar et al., 2022], AWQ [Lin et al., 2024], OmniQuant [Shao et al., 2024], AffineQuant [Ma et al., 2024], and ABQ-LLM [Zeng et al., 2024] for weight-only quantization and SmoothQuant [Xiao et al., 2023], OmniQuant, QLLM [Liu et al., 2024a], ABQLLM, and LRQuant [Zhao et al., 2024] for weight-activation quantization.

Implementation Details. We randomly sample 128 segments with a sequence length of 2048 from WikiText2 as our calibration training dataset. During the training phase, we adopt the AdamW optimizer with zero weight decay. Each transformer block is employed for 20 epochs with a batch size of 1 to complete the quantization reconstruction. Besides, the learning rates for the learnable outlier transformation factors and weight clipping factors are set to 5e-3 and 5e-4, respectively. For the learnable matrices in the PAR, we set the rank r to 128 and the learning rate to 1e-4. The entire training procedure is completed on a NVIDIA RTX 6000 48G GPU and we use the LM Evaluation Harness Toolbox for evaluation.

5.2 Experimental Results

Experiments on Language Generation Tasks. Table 1 presents a comparison of the PPL results for LLaMA-1 and LLaMA-2 models. As can be seen, with the reduction in quantization bitwidth or the model size becomes smaller, the enhancements shown by MPPQ become more pronounced. In particular, in the W6A6 quantization of the LLaMA-2-7B model, the margin of PPL between the MPPQ and FP16 baseline has narrowed to 0.14 and 0.23, respectively, which underscores the effectiveness of our approach. Furthermore, MPPQ reduces the PPL of the LLaMA-7B model on the WikiText2 dataset from 11.48 (ABQ-LLM) to 8.91 under the W2A16 quantization configuration, which was previously unattainable. These consistent superiority at low-bitwidth settings highlights our proficiency in maintaining generative capability under aggressive compression rates, without being dependent on any quantization-specific.

2 All results about OPT models are reported in Appendix B.

Experiments onZero-shot QATasks. We further evaluate the performance of MPPQ on multiple zero-shot benchmarks using the accuracy metric. As illustrated in Figure 3, on almost all datasets and models, MPPQ has significantly outperformed existing SOTA baseline methods, showcasing its impressive and powerful generalization ability. For instance, in Figure 3a, our average accuracy outperforms LRQuant by approximately 4% while for larger models (e.g., LLaMA-213B), this advantage still remarkable exists.

Combining the consistent gains elaborated in the above two subsections, we deduce that our MPPQ has established a new SOTA in any quantization scenario.

5.3 Ablation Study

We conduct detailed ablation study to thoroughly analyze the efficacy of the proposed MMS, PAR, and FCP. The experimental results in Table 2 and Figure 4 indicate the contributions of each component.

It can be observed from Table 2 that enabling FCP reduces the perplexity from 5.97 to 5.82 (0.15 ↓), emphasizing the critical importance of properly aligning quantization and clipping patterns, an aspect often overlooked in prior work. Building on this, the inclusion of PAR further improves accuracy by 0.78 under W3A16 as shown in Figure 4, demonstrating that adaptive rounding is one of the keys to significantly enhancing quantization quality. Finally, the combination version delivers the best overall performance and is selected as our default configuration.

5.4 Discussions

The Effectiveness of MMS. Figure 2 indicates that our proposed MMS mitigates the inconsistency between distributions, while its ability to smooth outliers is shown in Figure 5.

It is evident that the Vanilla reflects the original distribution of activation, which is highly lacking in smoothness and is thus not preferred for quantization. The SmoothQuant and OmniQuant are able to yield relative improvements, yet both still have distinct limitations. In contrast, MPPQ provides more precise smoothing for outliers and thus obtains betters gains in Table 1 and Figure 3.

Effectiveness Analysis of FCP. As depicted in Figure 6a, there are over 20% learned factors close to 1, indicating that numerous outliers still exist, which further leads to suboptimal quantization results. In contrast, our clipping factors are mostly concentrated between 0.6 and 0.75, implying effective clipping of outliers.

6 Conclusion

In this study, we have introduced MPPQ, a novel PTQ framework for LLMs that targets extremely low-bitwidth scenarios. MPPQ relies on scaling transformation for outlier handling and consists of three innovative optimizations. Firstly, MPPQ introduces a mixed metric supervision approach for block-wise quantization reconstruction to effectively alleviate the deviation between quantization and full-precision distributions, while providing better regularization guidance for parameter learning. Subsequently, we have successively proposes a proxy-based adaptive rounding scheme and a factor coarse pre-searching mechanism for weight quantization. The primary objective of the former is to minimize errors caused by the RTN operator, while the latter aims to ensure proper alignment between weight quantization and clipping patterns. Both can play a role in tackling their respective challenges. Extensive experimental results in language generation and zero-shot QA tasks indicate the superiority of the proposed method. For example, it reduces the gap in PPL to less than 0.5 with FP16 for W3A16 quantization on the LLaMA7B model. Moreover, the models quantized by MPPQ, without any additional structures, facilitate their subsequent deployment on edge devices.

Figure 6: Visualization of the learned clipping factors in W3A16 quantization of the LLaMA-2-7B model. Here, for a fair comparison, our MPPQ activates only the FCP component.

Ethical Statement

Our proposed method does not further amplify biases and contravene any ethical standards.

Acknowledgments

This work was supported by Beijing Natural Science Foundation under Grant L244050.