FBQuant: FeedBack Quantization for Large Language Models

1 Introduction

Large Language Models (LLMs) [Touvron et al., 2023] have demonstrated remarkable capabilities in natural language processing, driving significant advancements in the field. while their extensive parameter counts enable this high-level performance, they also pose substantial deployment challenges related to memory access, storage, and computation. These issues challenges even more critical for personal, localized applications, where privacy concerns often preclude the use of cloud-end LLM APIs, and where most personal devices lack high-performance accelerators

✉ denotes the corresponding author.

(e.g., top-tier GPUs). Furthermore, as AI systems increasingly rely on LLM-based agents [Talebirad and Nadiri, 2023; Li et al., 2023b] to handle ever more complex tasks, the computational demands continue to intensify, raising unprecedented barriers to on-device deployment.

One defining characteristic of the on-device LLMs is the consistently small batch size (in most cases, a batch size of one) [Lin et al., 2024b]. Consequently, inference is predominantly a memory-bandwidth-bound operation constrained by weight loading. Under these conditions, weight-only quantization emerges as a highly effective optimization strategy (as shown in Fig. 1). By storing weights in INT3/4 while retaining inputs and outputs in floating-point, memory requirements and bandwidth overhead are substantially reduced. Moreover, this technique can be broadly applied to various hardware platforms, such as desktops, laptops, and mobile phones, offering a cost-effective solution that balances performance and affordability.

Existing optimization methods of weight-only quantization for LLMs can be broadly classified into three categories, as illustrated in Fig. 2, Clamping, Rotation, and Sub-branching: (a) Clamping methods [Lin et al., 2024b; Shao et al., 2023] refine the quantization scale by restricting the value range, improving the representation of smaller weights while sacrificing outliers. (b) Rotation-based approaches [Liu et al., 2024b; Lin et al., 2024a] apply rotationequivalent transformations to shift quantization challenges from weights (which are quantized) to activations (which remain in high precision). (c) Sub-branching [Li et al., 2023a; Li et al., 2024] methods introduce a parallel branch alongside the quantized main path to compensate for quantization errors. Despite extensive research into clamping and rotation techniques, the quantization-induced accuracy degradation remains a critical concern. Recently, sub-branching methods have gained attention as an orthogonal solution for mitigating quantization errors. The key idea is to split each quantized layer into two parallel paths: a main path that retains the quantized weights and a sub-branch that compensates for quantization losses. Recent approaches, such as CALDERA [Saha et al., 2024] and EoRA[Liu et al., 2024a], construct sub-branches using the LoRA [Hu et al., 2021] framework, but these methods can overfit to limited calibration data or rely on ill-posed optimization objectives (see Sec. 3). Another significant challenge of sub-branching methods is the unexpected increase in inference latency, despite the sub-branches only introducing a small number of operations. This delay is primarily driven by memory access bottlenecks, as the sub-branch frequently reads and writes input activations, intermediate results, and layer outputs.

In this work, we propose a novel method called FeedBack Quantization (FBQuant), tackling both the optimization problem and the latency overhead introduced by subbranches. Drawing inspiration from negative feedback mechanisms in automatic control [Franklin et al., 2002], FBQuant feeds the sub-branch weights back into the main path, ensuring that the reconstructed weights remain inherently bounded by the quantization process (see Sec. 4.1). With this mechanism, FBQuant is able to fine-tune sub-branches with a limited amount of calibration data, and prevent overfitting. The pipeline is both straightforward and effective as illustrated in Fig. 3, and we provide rigorous mathematical proofs in Secs. 3 and 4. To address the latency challenge, we develop the CUDA kernel fusion implementation on the sub-branches. By reducing the repeated read and write operations in the constructed layers, this implementation saves 60% of extra inference time compared to conventional sub-branches.

Experiments demonstrate that FBQuant outperforms existing methods across various tasks and model families. On 3-bit Llama2-7B, FBQuant improves zero-shot accuracy by 1.2%. We further apply FBQuant to instruction-tuned versions of these models, showing consistent gains over other quantization techniques. Wall-clock time evaluations further reveal significant token throughput improvements over traditional implementations, while matching the latency of quantized models without sub-branches.

Our contributions are summarized as follows:

• Feedback-Based Sub-Branch Optimization. We introduce a novel sub-branch quantization pipeline, FBQuant, which feeds sub-branch signals back into the main path to establish more effective reconstruction objectives. This design inherently upper-bounds the reconstructed weights, thereby preventing overfitting to calibration noise.
• Efficient CUDA Kernel Integration. We develop a tailored CUDA kernel that fuses the sub-branch operations with the main path, significantly reducing latency by minimizing repeated reads and writes to inputs, intermediates, and outputs.
• Superior Performance Across Models. Extensive experiments demonstrate that FBQuant consistently improves perplexity and zero-shot accuracy over existing quantization approaches across a variety of model families and parameter sizes.

2 RelatedWorks

Quantization Methods are commonly divided into two categories: Post-Training Quantization (PTQ) [Dong et al., 2019; Liu et al., 2023a] and Quantization-Aware Training (QAT) [Liu et al., 2023b]. PTQ offers training-free solutions by reconstructing the quantized model after it has been fully trained. For example, AdaRound [Nagel et al., 2020] optimizes the rounding direction inspired by a Hessian-induced reconstruction objective, while Hawq [Dong et al., 2019] leverages Hessian information to identify weights sensitive to quantization. In contrast, QAT incorporates quantization during the training process, mitigating the accuracy loss associated with quantization. In the case of large language models (LLMs), PTQ is generally preferred for its simplicity and lower computational overhead. In this work, we focus on PTQ for LLMs and introduce FBQuant.

Weight-only Quantization focuses solely on quantizing model weights, reducing memory usage and speeding up memory-bound operations. For instance, [Dettmers et al., 2022] introduce the INT8 weight quantization method. QLoRA[Dettmers et al., 2024] proposes the NF4 data format to better align with LLM weight distributions. GPTQ[Frantar et al., 2022] uses layer-wise quantization combined with Optimal Brain Compression[Frantar and Alistarh, 2022], leveraging inverse Hessian information to guide quantization process. AWQ[Lin et al., 2024b] identifies salient weights by the corresponding activations and applies scaling techniques to protect them during quantization. OmniQuant[Shao et al., 2023] refines quantization via clamping and rotation values learned per layer. However, these methods suffer from significant accuracy degradation under low-bit quantization. In contrast, FBQuant employs an orthogonal sub-branch mechanism that effectively compensates for quantization errors, leading to improved performance.

Weight-Activation Quantization handles both weights and activations. SmoothQuant [Xiao et al., 2022] and Outlier Suppression [Wei et al., 2022] achieve W8A8 quantization by balancing the quantization challenges of weights and activations. QServe [Lin et al., 2024c] employs a W4A8 scheme, offering a trade-off between accuracy and speed. Quarot [Ashkboos et al., 2024] and DuQuant [Lin et al., 2024a] further enhance accuracy by introducing Hadamard rotation on weights and activations. RPTQ[Yuan et al., 2023] and LLM-QAT[Liu et al., 2023b] achieve W4A4 quantizaiton. However, weight-activation quantization generally faces insufficient support on general devices, making weightonly methods more applicable in practice.

Sub-branches Compensation introduces parallel pathways alongside the quantized layer to offset quantization errors. LoftQ [Li et al., 2023a] directly decompose quantization errors using Singular Value Decomposition and construct low-rank residual paths correspondingly. CALDERA [Saha et al., 2024] relies on quantized low-rank sub-branches that are fine-tuned to further reduce quantization errors, while EoRA [Liu et al., 2024a] projects compression errors into the eigenspace of input activations, focusing on reconstructing the most impactful components. SVDQuant [Li et al., 2024], originally developed for diffusion models but adaptable to LLMs, observes that high-rank components capture most of the outliers, leaving the remaining components simpler to quantize. However, these methods lack robust optimization strategies or employ suboptimal reconstruction objectives, often resulting in diminished performance or overfitting to calibration data noise. In contrast, our FBQuant introduces a feedback-driven optimization mechanism that effectively addresses the issue.

3 Motivation

This section discusses our insights on the ill-posed optimization of existing sub-branching methods (Sec. 3.1), and the inference delay induced by the sub-branches (Sec. 3.2).

3.1 Ill-posed Optimization

Sub-branch compensation reconstructs the weights of a layer can be mathematically represented as:

where W is the original weight matrix, WQ is its quantized counterpart produced by quantizer Q, and Σ is a low-rank linear projection derived from the sub-branch. Although this design aims at correcting quantization errors, directly optimizing Σ can lead to overfitting and dependence on calibration data. To see why, let's consider the layer-wise reconstruction objective:

where

∥ · ∥F denotes the Frobenius norm, X represents layer inputs, and r is the rank constraint for the sub-branch. Letting ∆ = W-WQ, assume Σ∗ yields a minimal value ϵ1 for L1, then

where tr(·) denotes the trace operation. In addition, from the Singular Value Decomposition (SVD), Σ∗ can be expanded in terms of the top-r singular vectors:

We can subsequently consider an alternative solution Σ′ :

with Nr sharing the same dimensionality as Vr, and a scalar α. Because typical calibration data are limited, X⊤X is positive semidefinite but not strictly full-rank [Huang et al., 2024], implying there exists non-zero Nr orthogonal to X⊤X, that is,

Then,

Hence, the construction loss ϵ′ which is achieved by Σ′ is equal to ϵ1:

where gray terms are 0 due to orthogonality. This equation demonstrates that Σ′ is also a valid solution. However, for any w ∈ W, which is reconstructed by σ′ ∈ Σ′ , the difference between the original and the reconstructed weights is:

With determined {w,Q, σ∗ }, the unbounded term ασN may significantly deviate the reconstructed weights, leading to less meaningful values, which highlights the potential for overfitting and performance degradation.

3.2 Delay Attribution of Sub-branches

We observe that inference latency can be significantly impacted by the inclusion of sub-branches. Theoretically, the computational overhead introduced by sub-branches constitutes only a small fraction of the total computation in the main path. However, in practice, it can lead to a substantial increase in inference latency. As illustrated in Fig. 4, consider a linear layer in the Llama2-7B model with a weight matrix W ∈ Rd×d , a layer input and output dimension of d = 4096, and a sub-branch implemented by the LoRA [Hu et al., 2021] approach with a rank value of 128. The inclusion of the subbranch results in a 6.25% increase in Multiply-Accumulate Operations (MACs), but causes the decoding process to be four times slower inside the layer. This dramatic slowdown is primarily attributable to memory access bottlenecks. The sub-branch computations involve intensive access to input activations in the down projection (i.e., A), multiple writes to intermediate results before the up projection (i.e., B), and additional writes to the layer outputs.

4 Methodology

Building on the insights from Sec. 3, we propose FBQuant, which includes: (1) feedback integration to upper-bound reconstructed weights and prevent overfitting (see Sec. 4.1), and (2) a differentiability strategy ensuring gradient propagation for the sub-branch parameters to process layer-wise reconstruction (see Sec. 4.2). (3) Furthermore, we introduce an efficient CUDA kernel implementation to mitigate the extra latency induced by the sub-branch (see Sec. 4.3).

4.1 Feedback and Upper Bound

Conventional sub-branch methods overlook on how far the reconstructed weights can deviate from the original. To address this limitation, we introduce a novel feedback mechanism, hereafter referred to as FBQuant, which incorporates negative sub-weights into the quantization process. As illustrated in Fig. 3, the weights of the main path are Q(W-Σ), and the weights of the sub-branch are Σ, then, the feedback-based reconstructed weights WF are redefined as:

Letting w and σ be elements of W and Σ respectively, the difference between the original and reconstructed weights is defined as:

When Q(·) functions as round-to-nearest, the deviation in Eq. (12) is bounded as:

where s is the quantizer's scaling factor. Some toy examples are shown in Fig. 3 (left). Consequently, FBQuant naturally limits reconstructed-weight deviations, which is a critical property that prevents the sub-branch from overfitting calibration data noise.

4.2 Differentiability for Layer-wise Reconstruction

We fine-tune sub-branch weights based on layer-wise reconstruction. Before that, we must ensure the differentiability of the reconstruction objectives. Let's first look at the loss function derived by FBQuant:

Although Straight-Through Estimator (STE) [Bengio et al., 2013] is commonly used to approximate the derivative of the quantizer as

it yields zero gradients for the sub-branch weights Σ in FBQuant, that is

To overcome this issue, we detach the feedback signal from the back-propagation graph, while allowing gradients to flow only through the sub-branch. This modification yields

Then, we obtain:∂LF

which enables optimizing Σ by gradient descent. In practice, we implement Σ using low-rank adapters [Hu et al., 2021], such that Σ = B·A. The sub-branches are applied to the linear layers in LLMs, such as Query, Key, Value, and Out projections in attention blocks, as well as Down, Gate, and Up projections in feed-forward networks (FFNs). During the optimization process, calibration data is fed into the model and each layer is optimized using gradient descent to minimize the reconstruction loss in Eq. (14). More details of layer-wise reconstruction by FBQuant are described in algorithm 1.

4.3 Kernel Fusion

To address the latency challenge induced by sub-branches describe in Sec. 3.2, we propose the kernel fusion method, illustrated in Fig. 5. The reconstructed layer consists of four kernels: de-quantization and linear projection in the main path, as well as down-projection and up-projection in the sub-branch. We observe that while the down-projection requires relatively minimal time, the other operations dominate the overall latency. To mitigate this, we integrate weight de-quantization, activation-weight multiplication, and the upprojection into a single CUDA kernel. This reduces the total number of kernels from four to two, significantly lowering the kernel launch overhead. Additionally, in the fused

Algorithm 1 Layer-wise Reconstruction by FBQuant

Require: Xl: input activations of the l-th layer

Wl: original weights of the l-th layer

Q: linear quantizer

Σl, Al, Bl: Σl ← BlAl, weights forming the low-rank sub-branch

• r: rank of the sub-branch satisfies rank(Σ) ≤ r

Ensure: A′l , B′l : optimized low-rank sub-branch weights

• 1: Al ← instantiate Al ∼ N(0, σ2 I);
• 2: Bl ← instantiate Bl with 0;
• 3: for l = 1 -→N do
• 4: repeat
• 5: obtain FP16 output: Zl ←WlX⊤l
• 6: obtain quantization output:

Z′l ←Q(Wl -BlAl)X⊤l +BlAlX⊤l

• 7: calculate reconstruction loss: L← ∥Zl -Z′l ∥F
• 8: back propagation:
• ∂L ∂Σ with detachedWl and Q(Wl -BlAl)
• 9: update Al, Bl via gradient descent
• 10: until Convergence
• 11: A′l ,B′l ←Al,Bl
• 12: end for
• 13: return {A′l ,B′l }Nl=1

kernels, the up-projection in the sub-branch shares the same output tensor as the linear projection in the main path. This optimization minimizes redundant writes, which would otherwise occur if the operations were handled separately. As a result, these modifications collectively reduce the additional inference delay caused by the sub-branches by 60%.

5 Experiments

In this section, we present the experimental setup of models, baselines, datasets, metrics and implementation details in Sec. 5.1. Then, we demonstrate the perplexity and zero-shot accuracy of various quantization methos in Sec. 5.2, followed by the performance of instruction-tuned models and the wallclock latency on real devices.

5.1 Experimental Setup

Models. We benchmark our methods using the model frameworks and checkpoints from HuggingFace [Jain, 2022], which includes Llama2 [Touvron et al., 2023], Llama3 [Dubey et al., 2024], and Qwen2.5 [Yang et al., 2024] families, with parameter sizes ranging from 7 billion to 70 billion. Specifically, we use pre-trained versions to generate the main results as shown in Tabs. 1 and 2, while instructionfinetuned versions are utilized for the pairwise competition as shown in Fig. 6.

Baselines. We compare FBQuant against Round-ToNearest (RTN), GPTQ [Frantar et al., 2022], AWQ [Lin et al., 2024b], OmniQuant [Shao et al., 2023], CALDERA [Saha et al., 2024], and SVDQuant [Li et al., 2024]. GPTQ, AWQ, and OmniQuant are implemented using their publicly released codebase. For CALDERA and SVDQuant, we follow their papers and codebases to produce the results, as CALDERA only support Lattice quantizer and SVDQuant is originally designed for diffusion models. Additionally, we include unquantized models using the float16 datatype as a baseline for a fair comparison.

Datasets and Metrics. Following previous works [Frantar et al., 2022; Lin et al., 2024b], we employ 128 samples with a sequence length of 2048 in the subset of WikiText2 [Merity et al., 2016] training data for calibration. The perplexity results are tested on the WikiText2 validation set. The zeroshot evaluation is conducted using the open-source toolkit, i.e., Language Model Evaluation Harness [Gao et al., 2024], which has been utilized by other baselines. The evaluation datasets include Arc-Challenge [Clark et al., 2018], ArcEasy [Clark et al., 2018], HellaSwag [Zellers et al., 2019], MMLU [Hendrycks et al., 2021], PIQA [Bisk et al., 2020], WinoGrande [Sakaguchi et al., 2019], and BoolQ [Wang et al., 2019]. We report the averaged accuracy.

Implementation Details. All experiments are conducted using A100 and RTX 3090 GPUs. Both the A100 and 3090 GPUs are utilized for optimizing the sub-branches, while only the 3090 GPU is used for testing latency, as it is commonly available for personal use. In the main results, we set the rank parameter to 128. The total number of optimization epochs is set to 20. A group size of 128 is used in all quantization methods. Sub-branches are integrated into all linear layers in LLMs, such as Query, Key, Value, and Out projections in Attention blocks, as well as Down, Gate, and Up projections in Feed-Forward Networks.

5.2 Results and Analysis

Performance Comparison. Tab. 1 and Tab. 2 present the evaluation results for perplexity on WikiText2, and zero-shot accuracy on seven public benchmarks, across different quantization methods, tested on various LLM architectures and model sizes. The quantization methods evaluated include RTN, GPTQ, AWQ, and OmniQuant, which rely on clamping and rotation techniques; as well as CALDERA, SVDQuant, and FBQuant, which adopt sub-branch approaches. Our FBQuant achieves state-of-the-art results in both perplexity and zero-shot accuracy across various models. Specifically, for the 3-bit Llama3-8B model, FBQuant reaches a perplexity of 6.78, marking a significant improvement of 0.85 over the second-best method AWQ. For the Llama2-7B model, FBQuant attains a zero-shot accuracy of 64.68%, outperforming OmniQuant by 1.20%. Three key observations can be made from the results: (a) 4-bit quantization generally offers good performance, while 3-bit quantization remains a little gap compared to the floating-point models. (b) New models present greater challenges for quantization. For instance, the 3-bit Llama3-8B model poses significant challenges for AWQ, which suffers a substantial degradation in perplexity by 1.88. (c) Existing sub-branch approaches occasionally produce poor performance, as observed with CALDERA on 3-bit version of Llama3-8B, Llama3-70B, and Qwen2.5-7B, as it relies on the ill-posed reconstruction objective as explained in Sec. 3. Besides, SVDQuant performs poorly on 3-bit Llama3-8B, as it only focuses on the weight outliers, ignoring the layer output error.

Quantization of Instruction-tuned Models. Most realworld applications are powered by instruction-tuned models, motivating us to benchmark the performance of our method on the instruction-tuned versions of the corresponding LLMs. Following the experiment setting in [Lin et al., 2024b; Shao et al., 2023], we conduct pairwise comparisons among AWQ, OmniQuant, CALDERA, SVDQuant, and FBQuant under 3bit settings. We use the GPT-4 evaluation protocol [Chiang et al., 2023] to assess performance on the Vicuna benchmark [Chiang et al., 2023], which comprises 80 questions. The results of the comparisons are reported as win, tie, and loss percentages. A higher percentage of wins and ties indicates better performance, highlighting the superior results achieved by FBQuant. To negate position bias [Zheng et al.,

2023], we also test pairs in exchanged position, totaling 160 trails per competition. As shown in Fig. 6, in the Llama38B-Chat model, our FBQuant demonstrates best performance against other quantization methods. For instance, FBQuant achieves 79.3% win-tie rate against to AWQ, and 90.0% wintie rate against to SVDQuant.

Wall-clock Latency on Real Devices. We evaluate the token throughput, measured in tokens per second (tk/s), of the FBQuant kernels on the RTX 3090 GPU, and compare it to floating-point (FP16), INT4, and INT4 with conventional sub-branch (INT4-Sub) implementations. As shown in Fig. 7, INT4-Sub exhibits a significant slowdown, achieving only 46 tk/s, slightly slower than FP16, which achieves 48 tk/s. In contrast, FBQuant demonstrates a substantial improvement against baselines, achieving 61 tk/s and significantly increasing token throughput.

6 Conclusions

In this paper, we introduced FBQuant, a novel feedbackbased optimization approach that addresses key challenges in the sub-branching quantization for LLMs. By incorporating a feedback mechanism inspired by automatic control systems, FBQuant effectively optimizes sub-branches without overfitting to calibration data, ensuring robust reconstruction of quantized weights. Additionally, we developed an efficient CUDA kernel fusion implementation to tackle the latency overhead introduced by sub-branches, significantly reducing inference delays caused by memory access bottlenecks. Our experiments demonstrated that FBQuant consistently outperforms existing quantization techniques across various tasks and model families on LLMs, achieving superior accuracy and perplexity while improving inference efficiency. Specifically, FBQuant enhances zero-shot accuracy for 3-bit models, such as Llama2-7B, by 1.2%, and achieves significant throughput improvements in wall-clock evaluations. These results highlight the practical utility of FBQuant for on-device deployment of LLMs, where computational and memory constraints are critical.

Figure 7: Token throughput of the Llama2-7B model using FP16, INT4-Sub, INT4 and INT4-FBQuant tested on RTX 3090 GPU. Experiments were conducted with a batch size of 1. The rank parameter for INT4-Sub and FBQuant was set to 128. The token throughput is measured by prefilling 256 and decoding 64 tokens.

Acknowledgments

This work was supported in part by the National Key Research and Development Program of China under Grant No. 2022YFB4400900, in part by the Strategic Industries and Key Technologies Project of Jiangsu Province under Grant BE2023020-3 and the Basic Research Program of Jiangsu Province under Grant BK20243042.