QFlash: Bridging Quantization and Memory Efficiency in Vision Transformer Attention

1 Introduction

Transformer self-attention provides strong representational power, but its computation and memory cost grow as O(N 2 ) with sequence length. This growth incurs a memory bottleneck in Vision Transformers (ViTs), where intermediate tensors must be transferred between on-chip and off-chip memory. FlashAttention [Dao et al., 2022] was proposed to reduce this problem by splitting the sequence into tiles. Each tile is computed on-chip, and only the final outputs are written to off-chip memory.

However, FlashAttention and later works [Dao et al., 2022; Dao, 2023; Shah et al., 2024] cannot compute the softmax in one step because of the tile-based design. They update the row-wise maximum for each tile, rescale the past results, and then add the new tile results. This process needs numerical stability and therefore depends on floating-point operations.

†Corresponding author

* Appendix available at: https://arxiv.org/abs/2604.25306

Other works on quantizing attention include I-BERT [Kim et al., 2021], I-ViT [Li and Gu, 2023], QAttn [Kluska et al., 2024], and INT-FlashAttention [Chen et al., 2024]. Although I-BERT and I-ViT quantize the Softmax operation, they do not consider a tile-based fused attention design like FlashAttention [Dao et al., 2022], leaving memory bottlenecks unresolved. QAttn and INT-FlashAttention quantize only the matrix multiplications, while the softmax still runs in floating point. As a result, there has been no work on a fully integeronly fused attention that reduces memory cost and removes floating-point operations.

Through our analysis of integer-only FlashAttention, we identify three key challenges. (C1) Scale Explosion: In tilewise accumulation, approximation errors and scale changes from integer exponential operations propagate across tiles, causing numerical instability. (C2) GPU Inefficiency: The shift-based exponential approximation requires integer division to separate integer and fractional parts, which is significantly slower than multiplication or shift operations on GPUs. (C3) Quantization Granularity: Fused attention requires uniform scales across tiles for direct integer comparison. pertoken quantization introduces scale mismatches that necessitate costly dequantization.

In this paper we propose QFlash, an integer-only fused attention on tile-based computation that addresses these challenges. As shown in Figure 1, QFlash keeps the tile-based design but quantizes all operators including the softmax, thus the full kernel runs in the integer domain. First, we implement online softmax [Milakov and Gimelshein, 2018] with shift-based exponential approximation and row-wise max updates, addressing C1 by carefully managing scale propagation across tiles. Second, we optimize the shift-based exponential computation to minimize GPU inefficiency (C2) through efficient integer arithmetic. Third, we adopt per-tensor quantization granularity to maintain uniform scales across the entire attention computation, enabling direct integer comparison without dequantization (C3). This full kernel fusion reduces off-chip memory use, improves scheduling, and raises GPU utilization. QFlash therefore keeps the benefits of fused attention while providing a fully integer path that improves both latency and energy efficiency.

To validate QFlash, we implement it as an integer-only Triton kernel and evaluate it on an NVIDIA RTX 5090 across seven attention workloads (A1-A7) from ViT, DeiT, and Swin. QFlash achieves up to 6.73× speedup over I-ViT on ViT/DeiT and 8.53× on Swin, reduces energy by 18.8% over FP16 FlashAttention-2, and maintains Top-1 accuracy close to FP32 on ViT/DeiT. QFlash also improves SQNR by up to 6.7dB over I-ViT, indicating reduced quantization noise. We also observe that QFlash's IMMA utilization is 8.0%, lower than FlashAttention-2's HMMA utilization of 13.08%, but since IMMA provides twice the peak throughput on RTX 5090, QFlash delivers higher absolute performance, demonstrating that integer-only design achieves high efficiency without sacrificing accuracy.

2 Background &RelatedWork

2.1 FlashAttention

Standard self-attention requires computing the similarity between all query key pairs and storing the results in a matrix, leading to memory and computational complexity of O(N 2 ) with respect to the sequence length.

As a result, both the computation cost and GPU memory usage increase rapidly when processing long sequences. FlashAttention was proposed to address this issue by leveraging the GPU memory hierarchy to avoid storing the intermediate similarity matrix in memory and instead performing computations in a tile-based manner. The key idea is to use online softmax [Milakov and Gimelshein, 2018], which incrementally updates the softmax normalization as each block is processed. In standard softmax, all similarity values must be collected at once to compute the denominator.

In contrast, online softmax processes similarity values block by block, updating the running maximum and exponential sum as new blocks are computed. This approach maintains numerical stability while eliminating the need to store the entire similarity matrix in memory. Thanks to this design, FlashAttention performs computations directly in high-speed GPU memory and writes results to global memory only when necessary. As a result, the number of memory accesses is significantly reduced, and the overall memory complexity is relaxed from O(N 2 ) to O(N).

2.2 Vision Transformer Quantization

Prior work on Vision Transformer (ViT) compression has predominantly targeted linear operators, notably matrix multiplications [Li et al., 2023; Yuan et al., 2022; Zhong et al., 2024; Ramachandran et al., 2024]. Subsequent studies extended quantization to non-linear components (Softmax, GELU, and LayerNorm) using polynomial, log-domain, or bit-shift approximations [Kim et al., 2021; Lin et al., 2021; You et al., 2024; Hu et al., 2024; Kim et al., 2024]. These contributions, however, remain at the algorithmic level and do not translate the approximations into kernel-level speedups.

I-ViT [Li and Gu, 2023] advances the state of the art by bitshift approximating Softmax and GELU and applying TVMbased intra-operator optimizations. Nevertheless, it forgoes inter-operator fusions akin to FlashAttention, leaving memory traffic largely unoptimized and thereby preserving considerable headroom for further latency reduction in quantized ViTs.

There have been some efforts to optimize quantized attention kernels by fusing attention operators [Kluska et al., 2024; Chen et al., 2024]. These approaches reduce computation and memory usage by quantizing the key matrix multiplications in attention. However, the core operation of attention-Softmax-is still computed in the floating-point domain, which limits their applicability for integer-only inference.

More recent works accelerate attention and nonlinear operators through hardware-software co-design. Fused Tensor Core [Jahadi et al., 2025] fuses matrix multiplication and Softmax on GPUs by offloading max and sum operations to Tensor Cores, but still relies on floating-point exponentiation. PICACHU [Qin et al., 2025] and QUARK [Zhao et al., 2025] accelerate Softmax, GELU, and LayerNorm using dedicated hardware or CGRA-based designs. While effective, these approaches either keep Softmax in the floating-point domain or depend on specialized hardware support, limiting their generality for software-only, integer-only attention on commodity GPUs.

To address these limitations, this work proposes a fully integer-only FlashAttention kernel that quantizes all attention operations, including Softmax, using a software-only design on commodity GPUs.

2.3 Advantages of Integer-only Quantization

Peak performance of CUDA cores and Tensor Cores. For CUDA cores, the latency of integer and floating-point operations is almost the same, so full quantization is not very useful [Arafa et al., 2019]. On the RTX 5090, however, measurements show that Tensor Cores can run 174K operators per cycle with HMMA (FP16) and 348K with IMMA (INT8), which is two times higher 1 . Applying quantization to all operators therefore increases IMMA use and reduces data communication, giving a clear benefit.

Performance per watt of Tensor Cores. Half-precision matrix multiply and accumulate (HMMA) uses less energy per operation, but the instruction overhead is still high, which reduces the overall performance per watt. Integer matrix multiply (IMMA) may consume a bit more energy per operation, but the instruction overhead is lower, so it reaches higher compute density in real workloads [Dally, 2023]. From this view, quantizing the model to use IMMA improves both energy efficiency and performance density. In this work, we show that using IMMA with quantization improves not only latency and throughput but also energy efficiency.

1 Calculated based on 170 SMs configuration of RTX 5090 architecture.

Use of integer-only accelerators. In mixed-precision designs, matrix multiplications are quantized, but key operations such as softmax still depend on floating-point units. This causes extra communication cost between integer and floating-point hardware, which limits the acceleration and adds inefficiency from floating-point units. Low-cost integeronly accelerators cannot support such mixed-precision needs, so extra floating-point units must be added in a heterogeneous chip. This increases chip area and power budget, which raises the cost of model deployment.

3 Challenges of Fully Integer FlashAttention

3.1 C1: Scale Explosion in Tile-wise Integer Accumulation

The first challenge is how to manage scaling in a tile-wise accumulation setting to avoid overflow and severe accuracy degradation. FlashAttention computes attention outputs tile by tile and incrementally accumulates each partial result into the running output. Since the accumulated result from previous tiles is reused as input for subsequent tiles, numerical stability becomes critical once quantization is applied. In integer-based FlashAttention, the exponential function is replaced with an integer approximation. The resulting approximation errors and scale variations are repeatedly injected into the accumulation process across tiles, which can lead to rapid scale growth and instability if not carefully controlled. A detailed comparison of scale management strategies is provided in Appendix B.1.

3.2 C2: Inefficiency of Shift-based Exp/Softmax on GPU

The integer softmax proposed in I-ViT [Li and Gu, 2023] typically transforms the exponential function into an exp 2(·) form and implements it using shift operations combined with linear approximations. While this approach is theoretically efficient, it introduces practical inefficiencies on GPUs when applied to integer inputs.

Specifically, computing exp2(x) requires decomposing the input into integer and fractional parts. For quantized integer inputs, this decomposition involves integer division. On GPUs, integer division has significantly higher latency than multiplication or shift operations, making it a major performance bottleneck for softmax computation. Implementation details of our GPU-friendly solution are discussed in Appendix B.2.

3.3 C3: Quantization Granularity in Integer Fused Attention

Attention inputs have a multi-dimensional structure, and quantization granularity can be applied at the tensor, head, or token level. The choice of granularity directly affects both accuracy and performance.

In FlashAttention, stable exponential computation requires subtracting the maximum value from QK⊤ scores, with the maximum being updated at every tile. When this process is performed in an integer-only setting, updating the maximum requires either all tiles to be computed under the same scale or the values to be dequantized into floating point for comparison. However, dequantization introduces additional computation and memory accesses, which significantly undermines the performance benefits of fused attention kernels.

For this reason, we aim to maintain a shared scale that enables direct comparison of integer values without dequantization. From this perspective, per-token quantization generates different scales for each token, making it difficult to apply in integer fused attention during max updates and accumulation. As a result, per-tensor and per-head quantization are the practical granularity choices for integer-based FlashAttention. A visual illustration of granularity constraints is presented in Appendix B.3.

4 Integer-Only Fused AttentionMethod

In this section, we describe in detail how the proposed QFlash algorithm operates. We first define the notation used in the description. Let b denote the bit-width, and let Iint(b) denote the set of signed b-bit integers as defined in Equation (1). Let sX be the scale factor, and let ˆX denote the integer quantization of X.

For a real matrix X ∈ Rm×n , the scale factor sX and the quantized matrix ˆX are defined as in Equation (2).

QFlash maintains the tile-based fused attention computation of FlashAttention [Dao et al., 2022] while quantizing all operators, including the softmax, to achieve both data size reduction and compute efficiency. The detailed procedure follows Figure 2 and Algorithm 1. One execution of the outer loop in Algorithm 1 computes a single output tensor ˆOi. As shown in Figure 2, the process consists of the following eleven steps: 1 Compute Query-Key Score → 2 Compute Row-Max → 3 Update Max → 4 Compute Score Intensity → 5 Compute Max-Offset Factor → 6 Requantization → 7 Compute Row-Sum → 8 Compute Value MatMul → 9 Accumulate Denominator → 10 Accumulate Numerator → 11 Normalize.

4.1 MatMul

QFlash performs integer matrix multiplications for 1 Compute Query-Key MatMul and 8 Compute Value MatMul. These operations can be expressed as the dot product of two quantized matrices ˆA ∈ IBr×d int8 and ˆB ∈ Id×Bc int8 . Here, Br denotes the size of the query block, Bc denotes the size of the key/value block, and d represents the inner dimension of the multiplication. The inputs are given in int8 format, and the results are accumulated in int32 format. Equation (3) defines the scaling relationship corresponding to this integer matrix multiplication.

This operation can be highly optimized on GPUs and dedicated accelerators, replacing floating-point multiplications to significantly reduce both computational cost and memory access overhead.

4.2 Compute Row-Max and Update Max

For a matrix ˆX ∈ IBr×Bc int32 , the 2 Compute Row-Max step calculates the maximum value of each row to obtain a vector ˆm ∈ IBr int32 . Subsequently, in the 3 Update Max step, these values are compared with those from the previous tile to update the row-wise maxima progressively.

Equation (4) defines the extraction of the maximum value from each row, which serves as the reference point for the subsequent softmax computation. By subtracting the rowwise maximum from all elements, potential overflow in the exponential calculation is prevented. In addition, Update Max incorporates the running maximum across tiles during block-wise operations, ensuring stable numerical computation throughout the entire sequence. This tile-wise max update requires consistent scaling for direct integer comparison, which motivates Challenge C3.

4.3 Compute Score Intensity andMax-Offset Factor

In QFlash, the exponential operation is used in step 4 Compute Score Intensity and step 5 Compute Max-Offset Factor. The Score Intensity step computes the exponential term of softmax in integer form to measure attention strength. The Max-Offset Factor step uses the Row-Max value to keep the exponential operation stable.

We adapt the ShiftExp algorithm from I-ViT to approximate the exponential function of softmax with integer arithmetic. The key of softmax is the ex computation. By multiplying with the constant log2(e), we can rewrite it as a power of two. The input matrix is ˆX ∈ Im×n int32 with scale factor sX ∈ R. After Row-Max is applied, ˆX always has negative values. therefore, the exponential is applied only to negative inputs. This process is given in Equation (5).

Here, ˜ sX = sX · log2 e. Equation (6) rewrites the input ˆX as an integer part Q ∈ Z≤0 and a fractional part R. This decomposition typically requires integer division on GPUs, which motivates Challenge C2.

Since the fractional part R lies in (-1, 0], it can be replaced with a simple linear approximation as in Equation (7).

Note that ⌊ 1 ˜ sX ⌉ is a constant determined only by ˜ sX. Thus, it is precomputed outside the kernel, and the kernel performs

Algorithm 1 QFlash Algorithm

• 1: Require: ˆQ, ˆK, ˆV ∈ IN×d int8 , scale factors sQ, sK, sV ∈ R and s = sQ · sK · d-1/2 · log2 e, block sizes Bc,Br, multiplier Mr, shift bits r. 2: Divide ˆQ into Tr = ⌈N/Br⌉ blocks ˆQ1, . . . , ˆQTr of size Br×d, and divide ˆK, ˆV into Tc = ⌈N/Bc⌉ blocks ˆK1, . . . , ˆKTc and ˆV1, . . . , ˆVTc of size Bc × d each. 3: Divide output ˆO ∈ IN×d int8 into Tr blocks ˆO1, . . . , ˆOTr of size Br ×d each. 4: for i = 1 to Tr do 5: Load ˆQi from ˆQ[(i -1)Br : iBr, :] from off-chip to SRAM 6: On chip, initialize ˆOi = 0 ∈ ZBr×d , ˆl(0) i = 0 ∈ ZBr , ˆm(0) i = -221 ∈ ZBr 7: for j = 1 to Tc do 8: Load ˆK⊤j from ˆK⊤[:, (j -1)Bc : jBc] and ˆVj from ˆV[(j - 1)Bc : jBc, :] from off-chip to SRAM 9: On chip, compute ˆS(j) i ← ˆQi ˆK⊤j ▷ 1 10: On chip, compute ˆm(j) i ←max( ˆm(j-1) i , rowmax(ˆS(j) i )) ▷ 2 , 3 11: On chip, compute ˆα(j) i ← ShiftExp2( ˆm(j-1) i - ˆm(j) i , s) ▷ 4 12: On chip, compute ˆP(tiled) i ← ShiftExp2(ˆS(j) i - ˆm(j) i , s) ▷ 5 13: On chip, compute ˆP(j) i ←Requantization(ˆP(tiled) i , M, b) ▷ 6 14: On chip, compute ˆl(j) i ← ScaleRelease(ˆl(j-1) i , ˆα(j) i , s) + rowsum(ˆP(j) i ) ▷ 7 , 9 15: On chip, compute ˆO(j) i ← ScaleRelease( ˆO(j-1) i , ˆα(j) i , s) + ˆP(j) i ˆVj ▷ 8 , 10 16: end for 17: On chip, compute ˆOi ← ⌊ ˆOTc i /ˆlTc i ⌋ ▷ 11 18: Write ˆOi to off-chip as the i-th block of ˆO 19: end for 20: Return the output ˆO ∈ IN×d int8 and the output scale factor sO = sV .

only element-wise integer operations. Finally, Equation (8) shows the combination of the integer part and the approximated fractional part to compute the exponential.

This formulation enables an efficient integer-only approximation of the exponential operation using simple arithmetic and bit-shift operations.

4.4 Requantization

The result of the matrix multiplication is stored as ˆX ∈ IBr×Bc int32 . To be used in subsequent operations, this value must be converted back into the range of ˆY ∈ IBr×Bc int8 through the 6 Requantization step. This is achieved by applying a scaling transformation that reduces the integer range. First, we define a fixed-point multiplier as in Equation (9). Here, the scale ratio is converted into a binary logarithm to compute n, and the shift size r is set by considering the bit-width b.

Based on this, the requantization is performed as shown in Equation (10).

We precompute n, r, and ⌊ sX sY · 2r ⌉ outside the kernel, since they depend only on the scale factors and bit-width and are independent of the input ˆX.

4.5 Compute Row-Sum

For softmax normalization, the denominator requires the sum of each row. Thus, for the matrix ˆX ∈ IBr×Bc int8 , the 7 Compute Row-Sum step calculates the element-wise sum of each row.

Equation (11) describes the computation of the row-wise sums, where ˆ si represents the sum of the elements in the i-th row. These values serve as the key denominator terms in the softmax normalization.

4.6 Accumulate Numerator and Denominator

For softmax normalization, steps 9 Accumulate Denominator and 10 Accumulate Numerator must be performed. If these accumulations are carried out directly in the quantized domain, overflow may occur. A common approach dequantizes values before accumulation, but this introduces floating-point operations and weakens integer-only benefits. Related to Challenge C1, we approximate the inverse scale as an integer to perform division, enabling stable numerator/denominator accumulation and integer-only softmax normalization without overflow. We refer to this operation as ScaleRelease, whose definition and analysis are provided in Appendix B.1.

4.7 Normalization

After the inner loop, step 11 performs normalization. Both the numerator and denominator are accumulated as integers, and their ratio corresponds to softmax normalization without explicitly materializing the full score matrix. Since the scale factors cancel out during normalization, dequantization is implicitly handled. Importantly, we implement this division entirely in integer arithmetic, eliminating floating-point operations while producing int8 outputs directly usable for downstream computation.

5 Evaluation

All experiments were run on an NVIDIA GeForce RTX 5090 GPU with TVM 0.8, PyTorch 2.7.1 (CUDA 12.8), and Triton 3.3.1. We evaluated latency, energy, and accuracy at the operator level and then tested Top-1 accuracy against IViT, I-BERT, FQ-ViT, FlashAttention-2, INT-FlashAttention (Full/Half), and QAttn.

We extract seven unique attention workloads from different ViT/DeiT/Swin model variants, as summarized in Table 1. In addition, we evaluate both batch size 1 and batch size 8 to cover real-time inference and throughput-oriented inference settings, respectively.

5.1 Inference Latency

Latency comparison. Figure 3 compares the latency of different attention kernels across the seven attention workloads listed in Table 1. Since I-ViT is a representative integer-only attention kernel, we measure speedup primarily against I-ViT to assess the effectiveness of our fully integer QFlash design. In the batch-1 case, QFlash achieves up to 4.54× and 7.54× speedup over I-ViT on ViT/DeiT and Swin workloads, respectively. In the batch-8 case, QFlash is up to 6.73× and 8.53× faster than I-ViT on ViT/DeiT and Swin workloads, respectively. Despite the overhead of fully integer execution, QFlash consistently outperforms the integer-only baseline and remains generally faster or comparable to FP16 and mixed-precision kernels across workloads.

Tensor Core utilization. To analyze the latency improvement of our kernel code, we used NVIDIA Nsight Compute [Leinhauser et al., 2021] to profile Tensor Core utilization. We report utilization as a ratio to the peak throughput, using two workloads (A2 and A7) with batch size 8. The results are summarized in Table 2. The key observation is that on the RTX 5090, Tensor Cores achieve 174K operators per cycle with HMMA (FP16) and 348K with IMMA (INT8), meaning the peak throughput of IMMA is exactly twice that of HMMA. Therefore, even though the IMMA utilization of QFlash is only 8.0% compared to 13.08% HMMA utilization in FlashAttention-2, the higher peak throughput of IMMA means that QFlash still delivers greater overall performance. The same conclusion holds for A7.

5.2 Energy Consumption

We compared the energy consumption of five kernels on workload A2 with batch size 8. Power consumption was recorded using nvidia-smi [Yu et al., 2023; Bridges et al., 2016], and the number of operations was measured with Nsight Compute. All kernels executed the same total amount of computation. As listed in Table 3, QFlash shows the lowest energy consumption. There are two reasons for this result. First, QFlash achieves shorter execution time for the same workload, which reduces the integrated energy. Second, QFlash consistently uses the IMMA instruction for Tensor Cores, which provides higher energy efficiency per operation compared to HMMA [Dally, 2023; Horowitz, 2014]. The combination of these two effects makes QFlash the kernel with the lowest energy consumption under identical conditions.

5.3 Accuracy Evaluation

Quantization-level accuracy. Accuracy was measured with Signal-to-Quantization-Noise Ratio (SQNR), which indicates the relative signal-to-noise ratio after quantization, and Mean Squared Error (MSE), which quantifies the absolute error. Table 4 reports the results for two attention workloads, A2 and A7, evaluated with batch size 8. For both input sizes, QFlash shows SQNR above 30dB and MSE around 10-3 , which means the distortion from quantization is small. The quality is higher than the baseline integer method I-ViT, but slightly lower than INT-Flash-Half and INT-Flash-Full due to their mixed-precision design. QFlash chooses a design that favors speed and energy efficiency, while still giving acceptable accuracy and a balanced trade-off.

Top-1 evaluation setup. We measure end-to-end Top-1 accuracy on ViT, DeiT, and Swin models by quantizing only the attention modules while keeping the remaining layers in floating point to isolate the effect of attention quantization. For activation scaling, we use dynamic quantization, where scaling factors are computed on-the-fly during inference. Since IViT, I-BERT, FQ-ViT, INT-FlashAttention, and QAttn adopt different quantization granularities, we employ their original implementations without modification for a fair comparison: I-ViT, I-BERT, FQ-ViT, and QAttn apply ▽ per-tensor quantization, while INT-FlashAttention applies △ per-token and ◦ per-head quantization. Finer granularity requires more scaling factors, leading to a trade-off between accuracy and efficiency.

Top-1 accuracy results. As reported in Table 5, QFlash maintains accuracy close to FP32 on ViT and DeiT models and exceeds I-ViT, I-BERT, and FQ-ViT on ViT backbones. On Swin, QFlash yields lower Top-1 accuracy than I-BERT and FQ-ViT. Nevertheless, QFlash achieves an efficient integer-only attention kernel with minimal scaling overhead via per-tensor quantization.

6 Conclusion

In this paper, we proposed QFlash to address the dependence of Transformer attention on floating-point operations and the off-chip memory bottleneck from intermediate tensors. QFlash implemented the full attention process, including softmax, using INT8/INT32 integer-only operations with shift-based log/exp approximations and integer normalization. Experiments showed that QFlash achieved higher speed and precision than I-ViT and improved energy efficiency compared to FP16 FlashAttention, providing a practical solution for efficient and accurate integer-only attention in large-scale Transformer inference.

Ethical Statement

There are no ethical issues.

Acknowledgments

This work was supported by the Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (RS2025-02214497, Development of low-level optimization program API technology for AI semiconductors) (No.RS-202300277060, Development of open edge AI SoC hardware and software platform).