Direct Quantization for Training Highly Accurate Low Bit-width Deep Neural Networks

1 Introduction

Deep Convolutional Neural Networks (CNNs) have been playing a crucial role in the recent tremendous successes of a variety of computer vision tasks including image classification, object detection, segmentation, image retrieval, to name a few. However, unfortunately, this benefit comes at the cost of an excessive amount of memory and expensive computational resources, which impedes its application in embedded devices. Therefore, how to make CNN lightweight and practical in terms of memory and computation is an important task and has attracted a great amount of effort.

An effective approach is to use low bit-width weights and/or low bit-width activations. This approach not only can reduce the memory footprint, but it also achieves a significant gain in speed. In particular, using binary weights with full-precision (FP) activations can reduce the model size by 32× and achieve ∼2× faster during inference [Rastegari et al., 2016]. When also using low bit-width activations, the speedup is significantly greater; as the most computationally expensive convolutions can be done by bitwise operations [Rastegari et al., 2016]. Even though there are great improvements achieved by the existing quantization-based methods [Cai et al., 2017; Hou and Kwok, 2018; Li et al., 2017; Rastegari et al., 2016; Wan et al., 2018; Zhang et al., 2018; Zhou et al., 2016; Zhu et al., 2017], there are still noticeable accuracy gaps between the quantized CNNs and their FP counterparts, especially in the challenging cases of 1 or 2 bitwidth weights and activations.

Most of the current state-of-the-art network quantization methods [Wan et al., 2018; Zhang et al., 2018; Zhou et al., 2016; Zhu et al., 2017; Zhou et al., 2018] learn the FP weights and the quantizers to quantize FP weights. The learning can be sequential or joint learning. In the sequential learning, the models firstly learn FP weights. They then learn the quantizers to quantize FP weights. In the joint learning, the models learn the FP weights and their quantizers simultaneously. However, both learning approaches cause mismatch during training, i.e., the gradient descent process is only used to update the FP weights, but not the quantized weights. To address this problem, we propose a novel method that allows gradients to update the quantized weights with learnable quantization levels directly, i.e., the quantized weights are learned without requiring FP weights. In specific, the quantized weights are learned via learning two auxiliary variables: binary weight encodings and quantization basis vectors.

Another important aspect of training a low bit-width network is the activation quantization. Earlier works use simple, hand-crafted quantizers (e.g., uniform or logarithmic quantization) [Gupta et al., 2015; Lin et al., 2016; Zhou et al., 2016; Aojun et al., 2017; Hubara et al., 2017] or pre-computed quantizers fixed during network training [Cai et al., 2017]. Recently, [Zhang et al., 2018; Jung et al., 2019] proposed to learn the quantizers for activations during training adaptively. However, all existing methods consider all activation channels of an activation map equally and do not pay attention to the fact that different channels of activations contain different amount of information. As a result, the learned quantizer could be biased toward a few channels with high variances; while losing information from a large number of lower-variance channels. To handle this problem, we propose a novel activation quantization method which tries to minimize the information loss in the majority of channels, which cumulatively have noticeable impacts on outputs, via learning channel-wise quantizers (i.e., one quantizer per channel). Our main contributions can be summarized as follows:

• Our work is the first one to propose a novel method to learn the quantized weights with learnable quantization levels directly by using gradient descent such that they directly minimize the cost function. Hence, our method could avoid the training mismatch (i.e., the quantized weights are not updated by gradients) of existing methods.
• To avoid the activation quantizers being biased toward a few high variance activation channels, we propose to learn quantizers that focus to minimize quantization errors of the majority of channels. This is obtained via learning channel-wise quantizers.
• A detailed analysis on the inference efficiency of the proposed method is provided. The proposed method also achieves the state-of-the-art image classification accuracy when comparing to other low bit-width networks on CIFAR-100 and ImageNet datasets with AlexNet, ResNet and MobileNetV2 architectures.

2 RelatedWorks

How to improve inference efficiency of CNN for practical applications has been an active research field. Existing approaches can be roughly divided into three main categories: compact network design, parameter reduction, and network quantization. In this section, we discuss recent works on the last category, to which our work belongs.

Weight-only Quantization. Earlier works have applied the basic form of weight quantization to directly constrains the weight values into the binary space without or with a scaling factor, i.e., {-1, 1} in BinaryConnect [Courbariaux et al., 2015] or {-α, α} in Binary-Weight-Networks (BWN) [Rastegari et al., 2016]. TernaryWeight Networks (TWN) [Li and Liu, 2017] proposed to quantize weights into a ternary form of {-α, 0, α}. Trained Ternary Quantization (TTQ) [Zhu et al., 2017] generalized TWN by constraining the weights to asymmetric ternary values {-αn, 0, αp}. In comparison to the binary weights, the ternary weights can help to improve performance, while achieving a similar speedup during inference [Wan et al., 2018]. Hou and Kwok proposed loss-aware quantized networks which can further improve the accuracy by considering the loss in learning the binary/ternary weights. Aojun et al. presented incremental network quantization (INQ) to incrementally convert a pretrained FP CNN model to a low-precision one.

Weight and Activation Quantization. Since quantization of activations can substantially reduce complexity further, by allowing the dot products to be implemented with bitwise operations (i.e., xor and popcount) [Tang et al., 2017; Rastegari et al., 2016], this approach attracts more and more attention [Rastegari et al., 2016; Zhou et al., 2016;

Zhang et al., 2018]. In early works [Tang et al., 2017; Rastegari et al., 2016], the authors proposed to binarize both weights and activations to {-1, 1}. Although these methods can be highly efficient in reference time, there are considerable accuracy drops compared to FP networks. To address this problem, the generalized low bit-width quantization was further studied in [Zhou et al., 2016; Miyashita et al., 2016]. By allowing to use more bits, this approach can provide a trade-off between accuracy and inference complexity. A popular choice of early works is the uniform quantization [Zhou et al., 2016; Hubara et al., 2017]. Miyashita et al. later proposed logarithmic quantization which can help to improve the inference efficiency via bit-shift operations. Park et al. used weighted-entropy to learn quantizers that are more concentrated on the values that are neither too small nor too large. Cai et al. proposed to pre-compute a single activation quantizer for all activation layers by fitting the probability density function of a half-wave Gaussian distribution. As opposed to the fixed, handcrafted quantization schemes of the aforementioned works, LQ-Nets [Zhang et al., 2018] proposed to jointly train a quantized CNN and its associated quantizers. Jung et al. proposed to train quantizers with parameterized intervals, which simultaneously performs both pruning and clipping. Gu et al. proposed Bayesian optimized 1-bit CNNs (BONN) which incorporates the prior distributions of FP kernels and features into the Bayesian framework to construct 1bit CNNs. Besides, recent works on designing new structures for binary networks achieve promising results. Bi-Real Net [Liu et al., 2018] introduced a new variant of residual structure to preserve the real activations before the sign function. The shortcut can help to increase the representational capability of the 1-bit convolutional block. Zhuang et al. proposed Group-Net, in which a set of binary convolution branches can effectively replace a FP convolution.

3 ProposedMethod

3.1 Learned QuantizedWeights

Apart from existing network quantization works, which train the weight quantizers from the full-precision (FP) network weights either jointly or in advance [Zhou et al., 2016; Wan et al., 2018; Zhang et al., 2018]; in this work, we propose to directly learn the quantized weights.

Specifically, for a general case of representing a network weight Wq of Nw elements by Kw-bits, we learn a binary weight encoding Swb ∈ {-1,+1}Nw×Kw and a quantization basis vector vw = [vw 1 , · · · , vwKw ] ∈ RKw . Consequently, the network weight can be directly replaced by Swb vw in the cost function, i.e., Wq = Swb vw .

where Θ = {Swbi , vw i }∀i is the set of all network parameters and L is a cost function, e.g., cross-entropy for classification.

To handle the binary constraint on the weight encoding Swb , inspired by Courbariaux et al., we introduce a fullprecision encoding matrix Sw (i.e., Sw ∈ RNw×Kw ) for parameter updates, and obtain Swb using sign function (i.e.,

Algorithm 1: Learned Quantized Weights (LQW).

L is the cost function for a mini-batch.

• Requires : Learning rate η and learning-rate scaling factors {γv, γs}

Parameters: Encoding matrix Sw, basis vector vw 1 Forward propagation:

• 2 Swb = sign(Sw )
• 3 Wq = Swb vw

4 Backward propagation and Parameter update:

• 5 Given ∂L ∂Wq obtained using the chain rule of gradient descent, compute ∂L ∂Swb and ∂L ∂vw
• 6 vw ←vw -γvη ∂L ∂vw

Swb = sign(Sw )) only during forward and backward propagations. Furthermore, in order to update Sw using gradient descent, we adopt the straight-through estimator (STE) ∂ sign(z) ∂z = 1|z|≤1 [Hinton, 2012], i.e., ∂L ∂Swb = ∂L ∂Sw for Sw ∈ [-1, 1], to approximate the gradients propagating through the sign function. Regarding the quantization basis vectors vw , as they have no constraint, they can be updated normally using the standard gradient descent process. The proposed method allows the quantized weights with learnable quantization levels to be updated (via the weight encodings and the quantization basis vectors) such that they directly minimize the final cost function through the gradient descent process. Additionally, we want to emphasize that other works (e.g., LQ-Nets, TBN) achieve quantized weights from FP weights during a joint training process. More specifically, their FP weights are updated via back-propagation, while the quantized weights (Wq) are updated to minimize the ℓ2-loss with the learning FP weights (i.e., argminWq ‖Wq -W‖2 ). This process potentially causes training mismatch, as the gradients cannot directly affect the quantized weights.

Algorithm 1, dubbed as Learned Quantized Weights (LQW), presents the procedure of our proposed method to train a quantized network weight. Note that, to maximize the flexibility of the low bit-width CNN, while ensuring its compatibility with bitwise operations, we learn one quantization basis vector and weight encoding per filter.

3.2 Activation Channel-Wise Averaged Quantizer

Scalar Quantizer. Given a full-precision ReLU activation A ∈ RB×C×W×H ≥0 (i.e., the set of non-negative real values), our goal is to represent A using Ka-bits with minimized information loss. We consider A as a vector a of Na-dimension (Na = BCWH). We aim to find an optimal quantization basis vector va = [va 1 , · · · , vaKa ] ∈ RKa and an optimal binary encoding Sa ∈ {0, 1}Na×Ka that minimize the following quantization error:

This problem can be solved by alternatively updating Sa and va : (i) Given va fixed, the optimal Sa can be found

Algorithm 2: Channel-wise Averaged Quantizer

(CAQ). L is a cost function for a mini-batch.

Inputs

: A mini-batch activation A with C channels, Moving-average factor µ, and number of iteration T .

Outputs : The quantized activation Aq.

Parameters: A set of quant. basis vectors {v(j) }Cj=1 .

1 if in the training stage, then

2 Forward propagation:

3

4

5

6

7

8

for j = 1 to C do

Set v(j)

0 = v(j)

for t = 1 to T do

Given v(j)

t by looking up

t-1 , compute S(j)

the index of the nearest quant. level;

Given S(j)

t , compute v(j)

t by using Eq. 3.

10 Compute Aq by looking up the nearest quantization level ∈ va for each element.

11 Backward propagation:

12 Given ∂L ∂Aq obtained using back-propagation;

13

∂A = ∂L

∂Aq

// STE

14 else // inference stage

16 Compute Aq by looking up the nearest quantization level ∈ va for each element.

by looking up the index of the nearest quantization level. Note that, for a Ka-bit quantization, there are 2Ka quantization levels qa = [qa i , · · · , qa 2(Ka) ] ∈ R2(Ka) and qa i = 〈Dec2BinKa (i), va 〉 , where 〈·〉 is a dot product and Dec2BinKa (i) is the function convert the decimal value i (0 ≤ i < 2Ka ) to the equivalent Ka-bit binary vector, e.g., Dec2Bin3(6) = [0, 1, 1]. (ii) Given Sa fixed, we have the closed-form solution for va as follows:

Channel-Wise Averaged Quantization. The Batch Normalization layer [Ioffe and Szegedy, 2015] includes two steps: (i) normalizing and (ii) shifting and scaling. While the normalizing step tries to make each channel of A to have zero-mean and unit-variance, the scaling and shifting step makes the variances of channels different. The variances can be very large or very small, depending on the corresponding scaling factor of the i-th channel. Hence, when a single scalar quantizer is used for A as in previous works (e.g., LQNets [Zhang et al., 2018]) (called as baseline), few channels which have high variances can significantly affect the quantization basis values. Consequently, the resulting quantization basis vector potentially causes significant information loss in many channels with low variances, which cumulatively have noticeable impacts on outputs. Additionally, slightly less focus on large activation values might help to reduce overfit-

∂L

(a) 2-nd quant. level

(c) 4-th quant. level

(d) MSE relative changes

Figure 1: Fig. 1a, 1b, and 1c are respectively the distributions of the optimal 2-nd, 3-rd, and 4-th channel-wise quantization levels of the first activation layer of AlexNet with Ka = 2. t indicates the proposed channel-wise averaged quantization levels - Mean Square Error (MSE) = 7.376 (using 100 random ImageNet images); and t indicates the quantization levels when considering the whole activation as the input for a scalar quantizer (e.g., [Zhang et al., 2018]) (called as baseline) - MSE = 6.664. Fig. 1d shows the histogram of the relative changes (%) in channel-wise MSE of our method compared to the baseline. We can observe that, in the baseline approach, the quantization levels tend to be biased toward large values. Although our proposed method results in higher quantization error overall (i.e., 7.376 > 6.664), the majority of channels (68 out of 96 channels in Fig. 1d) achieve smaller quantization errors (the relative changes < 0) compared to the baseline.

ting. To address this problem, first, we propose to learn a scalar quantizer for each channel of activation A. Then, the quantization basis values for the layer-wise quantizer can be obtained by taking the average of all quantization basis values of channel-wise quantizers, i.e., va i = 1C ∑Cj=1 va(j) i , where va(j) i is the i-th quantization basis value of j-th channel. With this proposed approach, the layer-wise quantization levels minimize the quantization errors on the majority of channels. Figure 1 provides an example to illustrate our proposed method. Besides, noticeably, only a mini-batch of input is available at each forward/backward step, we necessarily apply exponential moving average to update the channel-wise quantization level vectors to ensure stability. Algorithm 2, dubbed as Channel-wise Averaged Quantizer (CAQ), summarizes the procedure for training and inference of our proposed quantizer for activations. Note that during inference, only the channel-wise averaged quantizer is required. Hence, our method has the same time complexity as the baseline does.

4 Experiments

To evaluate the performance of the proposed methods, we conduct experiments on CIFAR-100 [Krizhevsky, 2009] and ImageNet (ILSVRC2012) [Deng et al., 2009] datasets using the two representative network architectures, AlexNet [Krizhevsky et al., 2012], ResNet [He et al., 2016], and MobileNetV2 [Sandler et al., 2018]. We adopt the standard data splits for both datasets, i.e., 50K training and 10K test images for CIFAR-100, about 1.2 million training and 50K test images for ImageNet. Additionally, we use a variant of AlexNet architecture by adding Batch Normalization layers after each convolutional layer and Fully-Connected layer and removing the Local Response layers as commonly used in recent works [Zhou et al., 2016; Zhuang et al., 2018].

4.1 Implementation Details

We follow the common settings in previous works [Zhou et al., 2016; Zhang et al., 2018]: We first resize the shorter side of the images to 256. During training, we randomly crop 227 × 227/224 × 224 patches for AlexNet/ResNet and MobileNetV2 from training images or their horizontal flips (called as basic augmentation). Following Zhang et al., we adopt the basic augmentation in most of our experiments, except for the cases of ResNet on ImageNet dataset with Kw/Ka = 2/2, 3/3, where the augmentation strategy of the ResNet Torch implementation 1 is adopted. We report top-1 and top-5 classification accuracy using single-center crops of test images.

In all experiments, we use Nesterov SGD with a momentum of 0.9 and mini-batch size of 256. The starting learning rate is set at 0.02/0.1/0.02 for AlexNet/ResNet/MobileNetV2 respectively. We adopt the polynomial learning rate scheduler with the annealing power of 2 and the final learning rate of 10-6 after 120 epochs. Regarding the weight decay, for FP weights, we set weight decay as 5 × 10-4 /5 × 10-5 /10-5 for AlexNet/ResNet/MobileNetV2. For the quantized weights, the weight decay has a stronger effect; hence, we crossvalidate within {10-6 , 5 × 10-6 , 10-5 , 5 × 10-5 } using 1% of the training set as the validation set. We scale the learning rate of the binary weight encodings as suggested by Courbariaux et al.. We also scale the learning rate of quantization basis vectors by a factor of γv = 1 50 to avoid unstable training. The moving-average factor µ and the number of iteration T in CAQ algorithm are fixed as 0.9 and 1, respectively. Following previous works [Rastegari et al., 2016; Zhou et al., 2016; Zhang et al., 2018], we do not quantize the first and last layers. Since the speedup, benefited by bitwise operations, is low due to their small input-channel number or filter size [Zhou et al., 2016; Rastegari et al., 2016]. All networks using our proposed method are trained from scratch.

4.2 Inference Efficiency Analysis

We first show that the inner products between our quantized weight and activation vectors can be efficiently computed by bitwise operations. Let a weight vector W ∈ RN be encoded by Sw = {sw1 , · · · , swKw } ∈ {-1, 1}N×Kw and the learned basis vector vw = {vw 1 , · · · , vwKw } ∈ RKw . Similarly, we encode an activation vector A ∈ RN by Sa = {sa1 , · · · , saKa } ∈ {0, 1}N×Ka and va = {va 1 , · · · , vaKa } ∈ RKa . To utilize the bitwise inner products [Zhou et al., 2016]; first we need to convert Sa ∈ {0, 1}N×Ka to ˆSa ∈ {-1, 1}N×Ka by Sai = 0.5ˆSai + 0.5. Then, with some manipulation, we can derive that:

1 https://github.com/facebook/fb.resnet.torch

where Q = ∑Kw i=1 ∑Ka j=1 vw i va j 2 (swi ⊙1), 1 is a Kadimension vector with all elements are 1, and ⊙ denotes the inner product with bitwise operations, xnor and popcnt. Noting that during inference, Sw , vw , and va are fixed; so Q is a scalar constant and can be pre-computed. Similar to [Zhou et al., 2016; Zhang et al., 2018], the computation complexity of Eq. 4 w.r.t. Kw,Ka is O(KwKa), i.e., directly linearproportional to the bitwidths of weights and activations.

Considering a matrix multiplication (MatMul) ˜C = ˜W˜A, where ˜W ∈ Rn×p , ˜A ∈ Rp×m and ˜C ∈ Rn×m. To calculate ˜C with FP ˜W, ˜A, there are nmq multiply-accumulate operations (MACs) required. Following Wan et al., the theoretical speedup ratio for the bitwise MatMul of our method, which requires KwKamn MACs and 2(KwKa)mn binary operations, is given as follows:

where L is the number of bits binary operation in one clock cycle, and γ = 1.91 is ratio between the speed of performing a L-bit binary operation and a MAC [Wan et al., 2018].

However, we found that Eq. 5 may not accurately reflect the actual speedup ratio of MatMul using bitwise operations in comparison with using FP. First, Eq. 5 over-simplifies the bitwise MatMul by only including xnor and popcount operations. In fact, additional operations are required, including sum to accumulate the popcount results, operations to convert from integer (popcount results) to FP numbers to multiply with quantization basis values. Second, Eq. 5 under-utilizes the CPU capability with FP computation. The modern CPU can perform 4 MACs in a single clock by using Single Instruction Multiple Data (SIMD) instructions. Multi-threading can also be utilized to further speed up computation by dividing the workload into multiple separate cores. Third, data transferring operations (e.g., loading, storing) is not considered. Using quantized weights and activations or SIMD can help to reduce the number of loading/storing operations. Finally, the additional time for quantizing activations should be considered in computing the speedup ratio.

We provide a CPU implementation of the bitwise MatMul (Eq. 4) to measure its actual CPU speedup in comparison with FP MatMul. We fix n = 256 (i.e., the number of output channels Co), m = 14 × 14 × 100 (i.e., an 100-sample mini-batch of 14 × 14 size), and 3 × 3 kernel. Noting that p = 3 × 3 × Ci, where Ci is the number of input channels. The experiment is conducted on a quad-core CPU (i76700HQ@2.60GHz) with L = 64. In Figure 2, we present the empirical speedup ratio of the bitwise MatMul in compared with the FP MatMul (with different implementations of FP MatMul) as the number of input channels Ci varies.

Firstly, from Eq. 5, we can compute the theoretical speedup for Kw/Ka = 1/1 is ∼ 60×. However, our CPU implementation of bitwise MatMul, with all overheads including quantization and memory allocation, can achieve up to ∼ 200× speedup (Figure 2a) (when SIMD is not utilized for FP MatMul). When SIMD is utilized for FP MatMul, bitwise MatMul can achieve about ∼ 23× speedup for Kw/Ka = 1/1 (Figure 2b). Additionally, when multithreading (8 threads) is also considered for both FP (CBLAS library) and bitwise MatMul, bitwise MatMul can gain ∼ 4.3× speedup for Kw/Ka = 1/1. While for Kw/Ka = 2/2, we obverse only a small speedup for bitwise MatMul (for Ci > 64) (Figure 2c). Furthermore, the empirical speedup ratios confirm that the complexity of our bitwise MatMul is proportional to the bit-widths of weights and activations.

Additionally, we presented in Table 1 the inference time per image using bitwise convolution for the quantized AlexNet (for both weight and activation) models. We also include the inference time using FP convolution operation for comparison. We can see that when using AlexNet with Kw/Ka = 1/2, our proposed method with bitwise convolution can speed up ×1.7 and ×6.8 in comparison with FP convolution with and without using SIMD, respectively.

4.3 Ablation Analysis

In this section, we first conduct experiments to analyze the effect of our proposals, i.e., (i) directly learning the quantized weights (LQW) and (ii) the channel-wise average quantizer

(CAQ) for activations. For easy comparison, we consider the learned quantizer proposed by Zhang et al., which is used for both weights and activations, as the baseline. Table 2 shows the classification accuracy of AlexNet on CIFAR-100 with different combinations of baseline method and our proposals for Kw/Ka =2/2. The experimental results demonstrate the effectiveness of our proposals, i.e., LQW and CAQ, in training networks with low bit-width weights and activations. Additionally, LQW helps achieve a slightly higher improvement gain compared to CAQ (+0.1%).

We additionally conduct an experiment on CIFAR-100 using AlexNet with different numbers of CAQ iteration T. We repeat the experiment 5 times and report the results (mean±std) in Table 3. We observer that using T = 2, 3 does not result in any significant difference on Top-1 accuracy compared to T = 1. This is mainly because, for each mini-batch, CAQ starts from proper initial quantization basis values, which are the good results of the last mini-batch.

4.4 ComparisonWith State of The Art

In this section, we compare the performance of our proposed method with existing methods including XNOR-Net [Rastegari et al., 2016], DoReFa-Net [Zhou et al., 2016], HWQG [Cai et al., 2017], PQ+TS+Guided [Zhuang et al., 2018], TBN [Wan et al., 2018], LQ-Nets [Zhang et al., 2018], BiReal Net [Liu et al., 2018], QIL [Jung et al., 2019], DSQ [Gong et al., 2019], BONN [Gu et al., 2019], RCBN [Liu et al., 2019], and Group-Net [Zhuang et al., 2019]. Noting that, our method is generic and can be used with any bit-width and architecture.

Comparison on CIFAR-100. Table 4 presents the top1/top-5 classification accuracy on CIFAR-100 dataset of different network quantization methods on AlexNet, ResNet-18, and MobileNetV2. In all compared settings for AlexNet and ResNet-18, our method consistently outperforms the state-ofthe-art method LQ-Nets by large margins, i.e., ≥ 0.7% for Kw/Ka = 1/2 and ≥ 0.6% for Kw/Ka = 2/2 in term of top-1 accuracy. Regarding the efficient architectures MobileNetV2, at bit-widths of Kw/Ka = 3/3, there is only a small accuracy drop compared with FP model. Even at lower bit-width Kw/Ka = 2/2, Kw/Ka = 1/2, our proposed method still achieves good performance for MobileNetV2, which is already very efficient. This confirms the effectiveness of our proposed method. Furthermore, we can observe from Figure 3a that the training loss on CIFAR-100 of LQNets with ResNet-18 architecture is unable to approach 0 as our proposed method does. LQ-Nets also suffers from unstable test error (Figure 3b). This result is potentially due to the mismatch in the training process of LQ-Nets, as the gradient descent does not directly update the quantized weights. Consequently, the quantized weights are unable to obtain the desirable changes. This demonstrates the necessity of direct updating the quantized weights using gradients as our proposed method can do.

Comparison on ImageNet. The classification results on ImageNet dataset for AlexNet and ResNet-18/34 are presented in Table 5. Our proposed method (LQW + CAQ) outperforms the compared methods, e.g., DoReFa-Net, TBN, HORQ, HWGQ, DSQ, in the majority of settings for both AlexNet and ResNet-18/34. Our method outperforms LQNets in term of Top-1 accuracy by clear margins in the majority of experiments (i.e., ≥ 0.4%); except for ResNet-34 with Kw/Ka = 1/2, where our method achieves comparable performance with LQ-Nets. For Kw/Ka = 1/1, our method gains 0.5% and 0.2% Top-1 improvement over BiReal Net for ResNet-18 and ResNet-34 respectively. Our method still under-performs BONN and RBCN for ResNet18. However, BONN is designed specifically for 1-bit CNN, it is nontrivial to adopt this method to higher bit-widths for better trade-off between the computational speed and accuracy, as can be done easily in our proposed method. Besides, in BONN, in addition to quantization loss and the essential cross-entropy loss, BONN also has an additional feature loss to enhance the intra-class compactness which improves the classification accuracy. For our work, because we aimed to learn the quantized network without any modification in the network architecture, in addition to cross-entropy, we only introduce the quantization loss. Regarding RBCN, this method requires joint training of a FP model, a quantized model, and discriminators for distillation from the FP model to the a quantized model. Consequently, this method requires significantly high computational cost and memory. In comparison to QIL, our method achieves comparable performance for Kw/Ka = 2/2. Specifically, our method has 0.3% Top-1 accuracy lower than QIL for AlexNet, while achieves 0.3% higher than QIL for ResNet-18. For ResNet-34, the accuracy of our method is marginally lower than QIL's (i.e., -0.1% top1 accuracy). We note that QIL usually achieves the best performance with trainable but complex quantizer functions. It is unsure if the computational complexity of QIL during inference can be linearly proportional to the bit-widths of weights and activations (no efficiency analysis is provided by Jung et al.). In our work, we carefully design the method to ensure the complexity linearly proportional to the bit-widths. Additionally, in comparison with Group-Net (4 bases) on ResNet18, our method has 0.3% lower in top-1 accuracy. Nevertheless, Group-Net requires additional computational costs for the skip-connection in every binary convolution and for computing the soft connection between binary groups.

Table 5: Top-1/Top-5 classification accuracy (%) of AlexNet and ResNet-18/34 on ImageNet dataset under different bit-widths.

5 Conclusion

In this paper, we have presented a novel method to train CNN with low bit-width weights and activations. Firstly, we proposed to directly learn the quantized weights with learnable quantization levels using gradient descent, instead of learning to quantize full-precision learned weights as proposed in existing works. Secondly, by considering the quantization errors of all activation channels, our proposed quantizer for activations minimizes the information loss in the majority of channels, instead of being biased toward a few high-variance channels. The experiment results demonstrate the effectiveness of our proposed method in training low bit-width CNN

to achieve the competitive classification accuracy for various network architectures.