Overflow Aware Quantization: Accelerating Neural Network Inference by Low-bit Multiply-Accumulate Operations

1 Introduction

To date, as a powerful machine learning system architecture, deep neural network (DNN) has been applied on numerous applications, e.g., image classification [He et al., 2016], object detection [Ren et al., 2015; Liu et al., 2016], and semantic segmentation [Chen et al., 2018]. However, to achieve high-end performance on complicated problems, most DNN systems require heavy computational resources for model inference, which inevitably limits DNN's deployment on lowcost processors. Such processors are extensively used in billions of commercial products, such as mobile phones, drones, and Internet-of-Things (IoT) devices, which makes DNN acceleration a critical problem in both academia and industry.

To allow DNN acceleration, researchers have developed various approaches, which can be roughly divided into two groups. The first group of methods focus on designing or searching for more compact and efficient network structures, which allow for reduced number of parameters and computations while achieving comparable performance. Representative methods in this category include MobileNet [Howard et al., 2017], EfficientNet [Tan and Le, 2019], ProxylessNAS [Cai et al., 2019], and so on. The second group aims at improving the efficiency of arithmetic computation, i.e., the multiply-accumulate (MAC) operation, as it dominates most computations during the DNN model inference. One widely used method is to approximate the original floating-point calculation using fixed-point operation to achieve computation acceleration. This type of method is well-known as quantization [Jacob et al., 2018]. Representative visualization of quantization is shown in Figure 1(a), where 8-bit fixed-point integers are used to approximate floating-point values and 32bit fixed-point variables are used to hold MAC results. Moreover, in addition to speeding up the MAC operations, quantization also achieves better parallel computing based on the capability of modern CPUs.

Figure 1: A representative example to show that replacing a 32-bit accumulator with a 16-bit one leads to a double amount of multiplyadds operations at the same time.

By comparing Figure 1(b) against Figure 1(a), it can be shown that if 16-bit fixed-point variables are used to hold the MAC result, the degree of parallelism will be doubled and the I/O times will be halved. However, when 16-bit holder is used, numerical overflow on MAC results becomes a frequently-happening problem that must be explicitly considered. A straightforward solution is to use low-bit quantization (<8-bit) for all operands, which however leads to loss of quantization precision and significantly reduced the performance. In addition, low-bit quantization still needs to take up more physical bits (e.g., 4-bit quantization still requires 8-bit physical operands) in most modern CPUs, making the computational resources partially wasted.

To summarize, existing quantization methods utilize fixed number of bits to represent float values, while both highbit and low-bit representation have limitations. The former suffers from numerical overflow problems and the latter one leads to model precision degradation. To tackle this problem, we propose a novel method to adaptively determine the quantization precision in DNNs, by optimizing the number of bits for operands while prohibiting overflow on the low-bit MAC result holders. To achieve this, we introduce a trainable quantization range mapping factor α into each layer of a DNN network, which automatically scales the quantized result to prevent the undesirable overflow. In addition, we propose a quantization-overflow aware training framework for learning the quantization parameters, to minimize the performance loss caused by post quantization [Krishnamoorthi, 2018]. To verify the effectiveness of our method, we conducted tests on a couple of state-of-the-art light-weighted DNNs for a variety of tasks on different benchmarking datasets. Specifically, our experiments include image classification, object detection, and semantic segmentation, which are tested on ImageNet, Pascal VOC, and COCO datasets, respectively. Experimental results demonstrate that, compared with state-of-theart quantization methods, the proposed method can achieve comparable performance while speeding up the inference efficiency by about 2 times.

The main contributions of this paper are listed as follows:

1. We propose an overflow aware quantization (OAQ) algorithm for accelerating DNNs, that is able to adaptively maximize the number of bits for operands while prohibiting the numeric overflow.
2. To ensure optimized performance, we design a quantization overflow aware training framework (QOAT), to automatically learn the parameters used by the proposed OAQ algorithm.
3. We conduct extensive experiments on three public datasets using state-of-the-art light-weight DNNs. The results verify the effectiveness of our method on a variety of tasks including image classification, object detection, and semantic segmentation.

2 RelatedWork

In this work, we focus on quantization methods that accelerate inference on off-the-shelf hardware platforms. While non-uniform quantization methods [Stock et al., 2019; Gao et al., 2019] are also shown to be effective, they do not allow efficient implementing on modern CPUs.

A representative early method is binary quantization [Rastegari et al., 2016], which quantizes both weights and activations to one bit, by using bit-shift and bit-count instead of multiply-adds operators to speed up. This method achieves acceptable performance on common over-parameterized networks, like AlexNet [Krizhevsky et al., 2012], but leads to substantial performance degradation on light-weight networks, e.g. ResNet-18 [He et al., 2016] and MobileNet [Howard et al., 2017]. As of today, one of the most widely used quantization methods is 8-bit quantization [Jacob et al., 2018; Krishnamoorthi, 2018; Jain et al., 2019], which is extensively applied in different applications on various hardware platforms. 8-bit quantization converts the inference process into integer-only operations, that could result in 2× ∼ 3× faster inference process on mobile CPUs. However, when tacking resource-demanding large networks or deployed on resource-constrained platforms, additional DNN acceleration is still required.

To allow further DNN acceleration, low-bit quantization techniques are under active exploration, in which a key problem is to balance the inference speed and model performance. [Choi et al., 2018] proposes PACT to train not only the weights but also the clipping parameters for clipped ReLU using gradient descent. [Louizos et al., 2018] presents RQ to optimize the quantization part with gradient descent. However, both methods suffer from performance degradation on lightweight networks. In fact, when quantizing both weights and activations to 4 bits, PACT [Choi et al., 2018] leads to accuracy reduction from 70.9% to 62.44%. Also, quantizing MobileNet using RQ [Louizos et al., 2018] to 6-bit achieves 68% accuracy only. Additionally, existing low-bit techniques only focus on the classification task. Evaluation results on other popular tasks, e.g., object detection or semantic segmentation, are limited on literatures. Inference accelerating using properties of processors is another research direction. [Zeng et al., 2019] proposes to decrease computational load of multiplications, by exploiting the parallel computing capability of modern CPUs. [Gong et al., 2019] implements 2-bit fast integer arithmetic with ARM NEON technology and achieves 1.7× speed up over NCNN [Tecent.Inc, 2017]. However, the reported results still suffer from significant performance loss, e.g., 4-bit quantized MobileNet-v2 leads to performance drop from 71.8% to 64.8%.

Moreover, there are a number of mixed-precision quantization methods [Wang et al., 2019; Wu et al., 2018; Dong et al., 2019], which focus on searching for an optimal bit-width setup that can achieve high-level acceleration on customized hardware platforms while avoiding performance drop. Compared to those methods, the proposed OAQ framework focuses on off-the-shelf devices, and addresses the numerical overflow problem by designing trainable parameters in each layer of a network.

3 Quantization Prerequisites

In this section, we first present the general algorithmic framework of quantization methods. Subsequently, we analyze the inherent overflow problem of the general framework and provide mathematical conditions that allow overflow aware quantization (OAQ). Detailed steps on performing OAQ approach are discussed in the next section.

3.1 Standard QuantizationMethod

To convert a floating-point real number r ∈ R to a fixedpoint quantized number q ∈ Z, the following affine mapping function is typically used [Jacob et al., 2018]:

where S is the scale factor and Z is the zero-point parameter. By denoting q(i,k) a the element at ith row and kth column of matrix A, the matrix multiplication of C = A × B in quantized domain can be computed element-wise as:

where the multiplier P is defined as

which can be implemented by fixed-point multiplication and efficient bit-shift [Jacob et al., 2018]. By expanding terms in (2), we are able to obtain:

where

In (4), the majority of computation costs are the core integer matrix multiplication accumulation:

To compute for (6), the standard method is to accumulate products of 8-bit values (signed or unsigned) with a 32-bit integer accumulator (also see Figure 1(a)):

where Zy represents the space of y-bit representable number. As a result, a 32-bit register is required for caching the intermediate result. A NEON instruction can compute multiple multiply-adds at the same time, but it is limited by the register size and the number of multiplying and summing units on board. The limited register resource is usually the main bottleneck. Moreover, we note that under most ARM architectures, there is no instruction to implement (7). To this end, existing popular mobile inference engines (e.g., TFLite and NCNN) typically rely on

as a replacement, i.e., VMAL.S16 on ARM architecture.

Additionally, we note that loading data between register and memory is heavy operations. By applying (8), more register space is used to implement a bigger micro-kernel 1 . This largely reduces the frequency of transferring intermediate results between register and memory. To show more details, we evaluated the performance of convolutions on MTK8167s CPU with different implementations. Using VMLAL.S8 and 4x8 micro-kernel, the computation efficiency is improved by 36%. While applying a 4x16 micro kernel, we achieved 73% speed up.

3.2 Optimize Quantization Operations

To optimize the efficiency of current quantization scheme, we seek to use 16-bit integer as accumulator

Compared to (7), one NEON instruction here (VMLAL.S8 on ARM architecture) is able to compute double amount of multiply-adds operations, as shown in Figure 1(a) and Figure 1(b).

However, by directly using (9), numerical overflow becomes an unavoidable problem. This is one of the core problem we seek to resolve in this work. To make this possible, we first re-expand (2) by assuming qa as the quantized inputs and qb as the quantized weights:

which can be rewritten as

where

In above equations, Za, Zb, qb are all be constant values once training is complete, and thus B and ˆ q(j,k) b can be computed in advance to improve inference efficiency.

Based on above equations, we point out that, to allow efficient computation under (7), the following three conditions

must be satisfied:⎧⎪⎨⎪⎩

1 https://engineering.fb.com/ml-applications/qnnpack/

It is important to note that the last condition in (14) should hold for both final number and all intermediate numbers. We also point out that, the second and last conditions in (14) are not always naturally true. Without taking special consideration, numerical overflow will frequently happen. To guarantee (14) in a DNN system, additional algorithms need to be designed and implemented.

4 OverflowAware Quantization Framework

This section describes the details of our overflow aware quantization algorithm, including both representation and training, to ensure the important three conditions in (14).

4.1 Adaptive Integer Representation

One straightforward method to reduce accumulation overflow is to narrow the range of each quantized value. For example, by using 4-bit quantization instead of 8-bit quantization, real values are mapped to [-8, 7] instead of [-128, 127]. As a result, numerical overflow becomes significantly less likely to happen, at a cost of wasting a large number of bits and reducing accuracy. To this end, we propose an adaptive floatbit-width method to fully utilize the representation capability without arithmetic overflow.

Specifically, we use a float quantization range mapping factor α to adjust the affine relationship between the real range and quantized range, as shown in Figure 2. Scaled by α, the original 8-bit (the biggest bit-width can be used) quantization range [-128, 127] is mapped to [⌊ -128 α ⌋, ⌊ 127 α ⌋]. By enlarging α, we are able to narrow down the quantized value range until the arithmetic overflows are eliminated.

To present this mathematically, the affine function (1) can be written as

By applying the scale factor α on S

the quantized value is narrowed to

where rmin and rmax are the minimum and maximum limits of the real value, and b is the number of bits for the quantized value.

We name this as float-bit-width method since it differs from the traditional integer low-bit representation whose quantization ranges have to be chosen from the limited bit-width set, e.g, 3-bit for [-4, 3] or 4-bit for [-8, 7]. By using the proposed method, it is feasible to utilize different quantization range mapping factor α in each layer, to maximize the representation capability while prohibiting overflow. Additionally, since α is continuous, it can also be easily integrated into training process of DNNs. Details on adaptively learning α for weights and activations of each layer will be discussed in the next subsection.

In addition to the range of quantized value, the value distribution is also critical. We expect the quantized values to be centered and gathered around zeros. As a result, The accumulated number will be more likely to be away from overflow. Initializing weights with a normal-distributed-like initializer and applying L1-L2-Normalization are representative methods that can be used.

4.2 Learning Quantization Range Mapping Factor

To find proper α for each layer to simultaneously prohibit arithmetic overflow and retain the model's performance, we propose a quantization-overflow aware training framework. As shown in Figure 3, inspired by simulating quantization effects in forwarding pass [Jacob et al., 2018], we add the overflow aware module that simulates neural operations, e.g., Convolution, FullyConnect using 16-bit accumulator for capturing arithmetic overflow in the forward pass.

In the forward pass, 8-bit quantization simulates quantized inference by implementing in floating-point arithmetic which is called FakeQuantization (Qfake) [Jacob et al., 2018]. To adaptively learn α, capturing the amount of arithmetic overflow in the quantized inference process is required. Therefore, we insert a Quantization node (Qreal) into each layer of the computation graph in addition to Qfake. Qreal requires rmin and rmax to produce 8-bit quantized values which are the same with inference engine. rmin and rmax are scaled by α before being passed into Qreal and Qfake. rmin and rmax in activations are aggregated by exponential moving averages (EMA) with the smoothed parameter close to 1, such that the observed ranges are smoothed, allowing the network to enter a more stable state. The created convolution operation with 16-bit accumulator Conv-INT16 takes the 8-bit quantized values as input to simulate the integer-only-inference on the inference engine. This calculates the amount of arithmetic overflow No that accumulates in the inference process, including all types of overflow defined in the overflow-free conditions (14). An easy way to capture overflow signals from the popular training framework, e.g., TensorFlow or PyTorch is to compare the results between the regular 32-bit convolution and the 16-bit convolution. As the process of getting No is integer-only computation, gradient descent becomes not a proper method in back-propagation. To address this problem, we use a simple rule to compensate for this.

Specifically, when No is bigger than zero, increasing α by:

where lc is the fixed maximum learning rate. Additionally, lri is the dynamic learning rate for increasing α, which decays with the steps of training. After a large number of iterations, lri is decayed to a quite small value to stabilize the training state.

Alternatively, if No is zero, we decrease α by:

where lrd is the learning rate for decreasing α, whose properties and physical representation are both similar to lri. For improving training efficiency, calculating No and updating α are executed every M steps, e.g., 10 or 50 during iterations.

The strategy of inserting Qreal is slightly different from Qfake. As shown in Figure 3, fake quantization for input or output is inserted after the activation function or a bypass connection, e.g. adds or concatenates as they change the real min-max ranges. For the operations such as max-pooling, up-sampling or padding, Qfake is not required. But the 16bit neural operation, e.g., Conv-INT16 only accepts quantized integer inputs. The outputs of Qreal has to be directly passed into it. That means Qreal must be inserted ahead of ConvINT16 in the computation graph. Finally, Qreal and Qfake share the same rmin, rmax and α, for strictly simulating the same inference process.

5 Experiments

In experiments, we first show in-depth analysis on the proposed framework of adaptive scale parameter. Subsequently, we compare our proposed OAQ framework against representative state-of-the-art methods, to provide metrics on our algorithmic advantages.

5.1 Overflow Study

Since our key algorithmic contribution is a method to adaptively learn the quantization range mapping factor α for DNN's each layer, it is also important to demonstrate the distribution of α across networks in our experiments. Figure 4 provides two representative results of per-layer α values, adaptively trained by using the proposed quantizationoverflow aware training framework. The detailed operations per-layer are listed in Table 1. From this figure, we can observe that the computed scale factor varies across layer, emphasizing our 'scale-per-layer' adaptive design. Additionally, we note that the majority of activation factor α fall into the range of [2, 4], indicating that the quantized values are mostly between 6bits and 7bits.

Additionally, we estimate the overflow ratio of low-bit quantization with a 16-bit accumulator by random multiplyadds simulation. Specifically, we uniformly sampled N values from the quantization value range and tried to capture an overflow signal. The signal is computed by applying consecutive multiply-adds operators of the N numbers on a 16-bit holder. Since 3x3 depth-wise convolution and 1x1 point-wise convolution are widely used in light-weight DNN architectures, we chose N to be {9, 64, 256, 1024} representatively. Subsequently, the rough Non-Overflow ratio was calculated by 100000 independent simulation runs. As shown in Figure 5, 6-bit quantization could retain overflow free in most cases, while 7-bit and 8-bit methods are risky.

Both Figure 4 and Figure 5 indicate that at most 6-bits could be used if we apply low bit quantization to address the overflow problem. Therefore, we will focus on comparing OAQ against state-of-the-art 6-bit quantization methods.

5.2 Overall Performance Evaluation

To demonstrate the performance of our proposed OAQ framework, we evaluate both inference accuracy/recall and runtime characteristics on representative public benchmarking datasets.

ImageNet

The first experiment is to benchmark MobileNet-v2 [Sandler et al., 2018], MoibleNet-v1 [Howard et al., 2017], and MobileNet-v1 with depth-multiplier 0.25 on the ILSVRC2012 ImageNet. This dataset consists of 1000 classes, 1.28 million training images, and 50K validation images. We finetuned MobileNet models from pre-trained model zoo (TFslim 2 ). All of those models were re-trained for 20 epochs. During training, the inputs were randomly cropped and resized to 224x224 before being fed into the network. Since the inputs of the first layer are not suitable for scaling, we skipped learning the quantization range mapping factor of the first layer's weights. This rule was applied to all the following experiments, including PACT [Choi et al., 2018], which always uses 8-bit weights in the first layers. We report our evaluation results using Top-1 and Top-5 accuracy.

2 https://github.com/tensorflow/models/tree/master/research/slim

Figure 4: Provides two representative results of per-layer factor values, implying that the quantized values are mostly between 6bits and 7bits.

In our tests, we focus on comparing OAQ against state-ofthe-art 6-bit quantization methods, including PACT [Choi et al., 2018], RQ [Louizos et al., 2018], and SR+DR [Gysel et al., 2018]. Comparison against other selective state-of-theart methods were also conducted. As shown in Table 2, OAQ even outperforms 8-bit Quantization-Aware Training (QAT) [Jacob et al., 2018] in certain cases. From Table 2, we observe that when quantizing MobileNet-v1, OAQ outperforms RQ [Louizos et al., 2018] and SR+DR[Gysel et al., 2018] by large margins. Although PACT [Choi et al., 2018] is better on Top-5, the proposed method consistently performs better in other metrics, especially on MobileNet-v1-0.25, i.e., 1.36% higher than PACT. We also take DSQ [Gong et al., 2019] into comparison, as it also achieved 1.7× speed up over NCNN on an ARM Cortex-A53 CPU. The result shows, on MobileNetv2 our results are significantly better, e.g., 6.84% higher.

Detection on VOC and COCO

To illustrate the applicability of our method to object detection, we applied OAQ on MobileNet-v1-SSD [Howard et al., 2017; Liu et al., 2016] and MobileNet-v2-SSDLite [Sandler et al., 2018] and evaluated on the 2012 Pascal VOC object detection challenge and 2017 MSCOCO detection challenge. We implemented our experiments with TensorFlow and finetuned models from TensorFlow Object Detection API 3 (only backbone since the SSD header differs with tasks). We first trained them with 32-bit floating-point precision to achieve state-of-the-art performance, and subsequently fine-tuned on VOC for 40, 000 steps with batch size 32 and 60, 000 steps on COCO with batch size 48 respectively.

The results on VOC and COCO are listed in Table 3 and Table 4 respectively. On both of VOC detection challenge and COCO detection challenge, the proposed method outperforms the 6-bit PACT [Choi et al., 2018] significantly. In addition, our method achieves comparable performance with 8-bit QAT [Jacob et al., 2018] and is of small mAP drop from the original model, i.e., about 1% mAP drop in VOC and less than 2% mAP drop in COCO.

Segmentation on VOC

To demonstrate the generalization of our method to semantic segmentation, we applied OAQ on DeepLab [Chen et al., 2018] with MobileNet-v2 backbone (depth-multiplier 0.5 and

3 https://github.com/tensorflow/models/blob/master/research/ object detection/g3doc/detection model zoo.md

1.0). The performance was evaluated on the Pascal VOC segmentation challenge, which contains 1464 training images and 1449 validation images. The results are shown in Table 5. When quantizing the original model to 6 bits with PACT [Choi et al., 2018], there is a significant drop in performance, e.g., MobileNet-v2-dm0.5 backbone dropped 10.9% in mIOU. By comparison, the proposed OAQ method achieved comparable performance with QAT, and only drop 1.8% and 0.4% on MobileNet-v2-dm0.5 and MobileNet-v2 backbone respectively compared to the original model.

Inference Efficiency Benchmark

Finally, we demonstrate the capability of DNN acceleration of the proposed method on different low-cost hardware platforms. Specifically, we benchmarked computational efficiency on two selectively platforms, i.e., Allwinner V328 and MTK8167, whose processor architectures are ARMCortex-A7 and ARM-Cortex-A35 respectively. In this test, MobileNet-v1 and ResNet-18 were used as representative DNN models to conduct inference. To run the DNN inference, three popular neural network inference engines for lowcost platforms were selected, i.e., TFLite, MNN [Jiang et al., 2020], and NCNN, under single-threaded implementation within one core.

The experimental results are listed in Table 6, which clearly demonstrate that the proposed method outperforms competing methods by wide margins. In fact, on both MTK8167s and Allwinner V328, our method achieves 2× faster runtime than TFLite and 1.85× faster than NCNN.

6 Conclusion

In this paper, we propose an overflow aware quantization method to allow significant DNN inference time acceleration, and minimize the loss of accuracy. To achieve this, we propose to adaptively adjust the number of bits used for representing quantized fixed-point integers. This scheme is also incorporated into a novel training framework, to adaptively learn the overflow-free quantization range while maintaining high-end performance. By using the proposed method, an extremely light-weight neural network can achieve comparable performance with the 8-bit quantization method on the ImageNet classification challenge. Comprehensive experiments were also conducted to verify that our method can also be applied to various dense prediction tasks, e.g., object detection, and semantic segmentation by outperforming competing state-of-the-art methods.

Acknowledgements

Thanks for Shuo Zhang, Huanghao Ding, Conggang Hu, and Baitao Shao for their valuable input to this article. Specifically, Shuo Zhang helped us on maturing and structuring the idea of utilizing the int16 accumulator for accelerating the inference time. Huanghao Ding and Conggang Hu helped us on inference engine code design. Finally, Baitai Shao joined us on paper modification and proof-ready process.