Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks

1 Introduction

The superior performance of deep CNNs usually comes from the deeper and wider architectures, which cause the prohibitively expensive computation cost. Even if we use more efficient architectures, such as residual connection [He et al., 2016a] or inception module [Szegedy et al., 2015], it is still difficult in deploying the state-of-the-art CNN models on mobile devices. For example, ResNet-152 has 60.2 million parameters with 231MB storage spaces; besides, it also needs more than 380MB memory footprint and six seconds (11.3 billion float point operations, FLOPs) to process a single image on CPU. The storage, memory, and computation of this cumbersome model significantly exceed the computing limitation of current mobile devices. Therefore, it is essential to maintain the small size of the deep CNN models which has relatively low computational cost but high accuracy in realworld applications.

∗Corrsponding Author

Recent efforts have been made either on directly deleting weight values of filters [Han et al., 2015b] (i.e., weight pruning) or totally discarding some filters (i.e., filter pruning) [Li et al., 2017; He et al., 2017; Luo et al., 2017]. However, the weight pruning may result in the unstructured sparsity of filters, which may still be less efficient in saving the memory usage and computational cost, since the unstructured model cannot leverage the existing high-efficiency BLAS libraries. In contrast, the filter pruning enables the model with structured sparsity and more efficient memory usage than weight pruning, and thus takes full advantage of BLAS libraries to achieve a more realistic acceleration. Therefore, the filter pruning is more advocated in accelerating the networks.

Nevertheless, most of the previous works on filter pruning still suffer from the problems of (1) the model capacity reduction and (2) the dependence on pre-trained model. Specifically, as shown in Fig. 1, most previous works conduct the "hard filter pruning", which directly delete the pruned filters. The discarded filters will reduce the model capacity of original models, and thus inevitably harm the performance. Moreover, to maintain a reasonable performance with respect to the full models, previous works [Li et al., 2017; He et al., 2017; Luo et al., 2017] always fine-tuned the hard pruned model after pruning the filters of a pre-trained model, which however has low training efficiency and often requires much more training time than the traditional training schema.

To address the above mentioned two problems, we propose a novel Soft Filter Pruning (SFP) approach. The SFP dynamically prunes the filters in a soft manner. Particularly, before first training epoch, the filters of almost all layers with small ℓ2-norm are selected and set to zero. Then the training data is used to update the pruned model. Before the next training epoch, our SFP will prune a new set of filters of small ℓ2norm. These training process is continued until converged. Finally, some filters will be selected and pruned without further updating. The SFP algorithm enables the compressed network to have a larger model capacity, and thus achieve a higher accuracy than others.

Contributions. We highlight three contributions: (1) We propose SFP to allow the pruned filters to be updated during the training procedure. This soft manner can dramatically maintain the model capacity and thus achieves the superior performance. (2) Our acceleration approach can train a model from scratch and achieve better performance compared to the state-of-the-art. In this way, the fine-tuning procedure and the overall training time is saved. Moreover, using the pretrained model can further enhance the performance of our approach to advance the state-of-the-art in model acceleration. (3) The extensive experiment on two benchmark datasets demonstrates the effectiveness and efficiency of our SFP. We accelerate ResNet-110 by two times with about 4% relative accuracy improvement on CIFAR-10, and also achieve stateof-the-art results on ILSVRC-2012.

2 RelatedWorks

Most previous works on accelerating CNNs can be roughly divided into three categories, namely, matrix decomposition, low-precision weights, and pruning. In particular, the matrix decomposition of deep CNN tensors is approximated by the product of two low-rank matrices [Jaderberg et al., 2014; Zhang et al., 2016; Tai et al., 2016]. This can save the computational cost. Some works [Zhu et al., 2017; Zhou et al., 2017] focus on compressing the CNNs by using lowprecision weights. Pruning-based approaches aim to remove the unnecessary connections of the neural network [Han et al., 2015b; Li et al., 2017]. Essentially, the work of this paper is based on the idea of pruning techniques; and the approaches of matrix decomposition and low-precision weights are orthogonal but potentially useful here - it may be still worth simplifying the weight matrix after pruning filters, which would be taken as future work.

Weight Pruning. Many recent works [Han et al., 2015b;

2015a; Guo et al., 2016] pruning weights of neural network resulting in small models. For example, [Han et al., 2015b] proposed an iterative weight pruning method by discarding the small weights whose values are below the threshold. [Guo et al., 2016] proposed the dynamic network surgery to reduce the training iteration while maintaining a good prediction accuracy. [Wen et al., 2016; Lebedev and Lempitsky, 2016] leveraged the sparsity property of feature maps or weight parameters to accelerate the CNN models. A special case of weight pruning is neuron pruning. However, pruning weights always leads to unstructured models, so the model cannot leverage the existing efficient BLAS libraries in practice. Therefore, it is difficult for weight pruning to achieve realistic speedup.

Filter Pruning. Concurrently with our work, some filter pruning strategies [Li et al., 2017; Liu et al., 2017; He et al., 2017; Luo et al., 2017] have been explored. Pruning the filters leads to the removal of the corresponding feature maps. This not only reduces the storage usage on devices but also decreases the memory footprint consumption to accelerate the inference. [Li et al., 2017] uses ℓ1-norm to select unimportant filters and explores the sensitivity of layers for filter pruning. [Liu et al., 2017] introduces ℓ1 regularization on the scaling factors in batch normalization (BN) layers as a penalty term, and prune channel with small scaling factors in BN layers. [Molchanov et al., 2017] proposes a Taylor expansion based pruning criterion to approximate the change in the cost function induced by pruning. [Luo et al., 2017] adopts the statistics information from next layer to guide the importance evaluation of filters. [He et al., 2017] proposes a LASSO-based channel selection strategy, and a least square reconstruction algorithm to prune filers. However, for all these filter pruning methods, the representative capacity of neural network after pruning is seriously affected by smaller optimization space.

Discussion. To the best of our knowledge, there is only one approach that uses the soft manner to prune weights [Guo et al., 2016]. We would like to highlight our advantages compared to this approach as below: (1) Our SPF focuses on the filter pruning, but they focus on the weight pruning. As discussed above, weight pruning approaches lack the practical implementations to achieve the realistic acceleration. (2) [Guo et al., 2016] paid more attention to the model compression, whereas our approach can achieve both compression and acceleration of the model. (3) Extensive experiments have been conducted to validate the effectiveness of our proposed approach both on large-scale datasets and the state-ofthe-art CNNmodels. In contrast, [Guo et al., 2016] only had the experiments on Alexnet which is more redundant the advanced models, such as ResNet.

3 Methodology

3.1 Preliminaries

We will formally introduce the symbol and annotations in this section. The deep CNN network can be parameterized by {W(i) ∈ RNi+1×Ni×K×K , 1 ≤ i ≤ L} W(i) denotes a matrix of connection weights in the i-th layer. Ni denotes the number of input channels for the i-th convolution layer. L

denotes the number of layers. The shapes of input tensor U and output tensor V are Ni ×Hi ×Wi and Ni+1 ×Hi+1 × Wi+1, respectively. The convolutional operation of the i-th layer can be written as:

where Fi,j ∈ RNi×K×K represents the j-th filter of the i-th layer. W(i) consists of {Fi,j, 1 ≤ j ≤ Ni+1}. The Vi,j represents the j-th output feature map of the i-th layer.

Pruning filters can remove the output feature maps. In this way, the computational cost of the neural network will reduce remarkably. Let us assume the pruning rate of SFP is Pi for the i-th layer. The number of filters of this layer will be reduced from Ni+1 to Ni+1(1 - Pi), thereby the size of the output tensor Vi,j can be reduced to Ni+1(1-Pi) ×Hi+1 ×Wi+1. As the output tensor of i-th layer is the input tensor of i + 1-th layer, we can reduce the input size of i-th layer to achieve a higher acceleration ratio.

3.2 Soft Filter Pruning (SFP)

Most of previous filter pruning works [Li et al., 2017; Liu et al., 2017; He et al., 2017; Luo et al., 2017] compressed the deep CNNs in a hard manner. We call them as the hard filter pruning. Typically, these algorithms firstly prune filters of a single layer of a pre-trained model and fine-tune the pruned model to complement the degrade of the performance. Then they prune the next layer and fine-tune the model again until the last layer of the model is pruned. However, once filters are pruned, these approaches will not update these filters again. Therefore, the model capacity is drastically reduced due to the removed filters; and such a hard pruning manner affects the performance of the compressed models negatively.

As summarized in Alg. 1, the proposed SFP algorithm can dynamically remove the filters in a soft manner. Specifically, the key is to keep updating the pruned filters in the training stage. Such an updating manner brings several benefits. It not only keeps the model capacity of the compressed deep CNN models as the original models, but also avoids the greedy layer by layer pruning procedure and enable pruning almost all layers at the same time. More specifically, our approach can prune a model either in the process of training from scratch, or a pre-trained model. In each training epoch, the full model is optimized and trained on the training data. After each epoch, the ℓ2-norm of all filters are computed for each weighted layer and used as the criterion of our filter selection strategy. Then we will prune the selected filters by setting the corresponding filter weights as zero, which is followed by next training epoch. Finally, the original deep CNNs are pruned into a compact and efficient model. The details of SFP is illustratively explained in Alg. 1, which can be divided into the following four steps.

Filter selection. We use the ℓp-norm to evaluate the importance of each filter as Eq. (2). In general, the convolutional results of the filter with the smaller ℓp-norm lead to relatively lower activation values; and thus have a less numerical impact on the final prediction of deep CNN models. In term of this understanding, such filters of small ℓp-norm will be given high priority of being pruned than those of higher ℓpnorm. Particularly, we use a pruning rate Pi to select Ni+1Pi

Algorithm 1 Algorithm Description of SFP

Input: training data: X, pruning rate: Pi

Output: The compact model and its parametersW∗

unimportant filters for the i-th weighted layer. In other words, the lowest Ni+1Pi filters are selected, e.g., the blue filters in Fig. 2. In practice, ℓ2-norm is used based on the empirical analysis.

Filter Pruning. We set the value of selected Ni+1Pi filters to zero (see the filter pruning step in Fig. 2). This can temporarily eliminate their contribution to the network output. Nevertheless, in the following training stage, we still allow these selected filters to be updated, in order to keep the representative capacity and the high performance of the model.

In the filter pruning step, we simply prune all the weighted layers at the same time. In this way, we can prune each filter in parallel, which would cost negligible computation time. In contrast, the previous filter pruning methods always conduct layer by layer greedy pruning. After pruning filters of one single layer, existing methods always require training to converge the network [Luo et al., 2017; He et al., 2017]. This procedure cost much extra computation time, especially when the depth increases. Moreover, we use the same pruning rate for all weighted layers. Therefore, we need only one hyper-parameter Pi = P to balance the acceleration and accuracy. This can avoid the inconvenient hyper-parameter search or the complicated sensitivity analysis [Li et al., 2017]. As we allow the pruned filters to be updated, the model has a large model capacity and becomes more flexible and thus can well balance the contribution of each filter to the final prediction.

Reconstruction. After the pruning step, we train the network for one epoch to reconstruct the pruned filters. As shown in Fig. 2, the pruned filters are updated to non-zero by back-propagation. In this way, SFP allows the pruned model to have the same capacity as the original model during training. In contrast, hard filter pruning decreases the number of feature maps. The reduction of feature maps would dramatically reduce the model capacity, and further harm the performance. Previous pruning methods usually require a pretrained model and then fine-tune it. However, as we integrate the pruning step into the normal training schema, our approach can train the model from scratch. Therefore, the fine-tuning stage is no longer necessary for SFP. As we will show in experiments, the network trained from scratch by SFP can obtain the competitive results with the one trained from a well-trained model by others. By leveraging the pretrained model, SFP obtains a much higher performance and advances the state-of-the-art.

Obtaining Compact Model. SFP iterates over the filter selection, filter pruning and reconstruction steps. After the model gets converged, we can obtain a sparse model containing many "zero filters". One "zero filter" corresponds to one feature map. The features maps, corresponding to those "zero filters", will always be zero during the inference procedure. There will be no influence to remove these filters as well as the corresponding feature maps. Specifically, for the pruning rate Pi in the i-th layer, only Ni+1(1 - Pi) filters are non-zero and have an effect on the final prediction. Consider pruning the previous layer, the input channel of i-th layer is changed from Ni to Ni(1 - Pi-1). We can thus re-build the i-th layer into a smaller one. Finally, a compact model {W∗(i) ∈ RNi+1(1-Pi)×Ni(1-Pi-1)×K×K } is obtained.

3.3 Computation Complexity Analysis

Theoretical speedup analysis. Suppose the filter pruning rate of the ith layer is Pi, which means the Ni+1 × Pi filters are set to zero and pruned from the layer, and the other Ni+1 × (1 - Pi) filters remain unchanged, and suppose the size of the input and output feature map of ith layer is Hi×Wi and Hi+1 ×Wi+1. Then after filter pruning, the dimension of useful output feature map of the ith layer decreases from Ni+1 × Hi+1 × Wi+1 to Ni+1(1 - Pi) × Hi+1 × Wi+1. Note that the output of ith layer is the input of (i + 1) th layer. And we further prunes the (i + 1)th layer with a filter pruning rate Pi+1, then the calculation of (i + 1)th layer is decrease from Ni+2 × Ni+1 × k2 × Hi+2 × Wi+2 to Ni+2(1 -Pi+1) ×Ni+1(1 -Pi) × k2 ×Hi+2 ×Wi+2. In other words, a proportion of 1-(1-Pi+1)×(1-Pi) of the original calculation is reduced, which will make the neural network inference much faster.

Realistic speedup analysis. In theoretical speedup analysis, other operations such as batch normalization (BN) and pooling are negligible comparing to convolution operations. Therefore, we consider the FLOPs of convolution operations for computation complexity comparison, which is commonly used in previous work [Li et al., 2017; Luo et al., 2017]. However, reduced FLOPs cannot bring the same level of realistic speedup because non-tensor layers (e.g., BN and pooling layers) also need the inference time on GPU [Luo et al., 2017]. In addition, the limitation of IO delay, buffer switch and efficiency of BLAS libraries also lead to the wide gap between theoretical and realistic speedup ratio. We compare the theoretical and realistic speedup in Section 4.3.

4 Evaluation and Results

4.1 Benchmark Datasets and Experimental Setting

Our method is evaluated on two benchmarks: CIFAR10 [Krizhevsky and Hinton, 2009] and ILSVRC-2012 [Russakovsky et al., 2015]. The CIFAR-10 dataset contains 50,000 training images and 10,000 testing images, which are categorized into 10 classes. ILSVRC-2012 is a large-scale dataset containing 1.28 million training images and 50k validation images of 1,000 classes. Following the common setting in [Luo et al., 2017; He et al., 2017; Dong et al., 2017a], we focus on pruning the challenging ResNet model in this paper. SFP should also be effective on different computer vision tasks, such as [Kang et al., 2017; Ren et al., 2015; Dong et al., 2018; Shen et al., 2018b; Yang et al., 2010; Shen et al., 2018a; Dong et al., 2017b], and we will explore this in future.

In the CIFAR-10 experiments, we use the default parameter setting as [He et al., 2016b] and follow the training schedule in [Zagoruyko and Komodakis, 2016]. On ILSVRC-2012, we follow the same parameter settings as [He et al., 2016a; 2016b]. We use the same data argumentation strategies with PyTorch official examples [Paszke et al., 2017].

We conduct our SFP operation at the end of every training epoch. For pruning a scratch model, we use the normal training schedule. For pruning a pre-trained model, we reduce the learning rate by 10 compared to the schedule for the scratch model. We run each experiment three times and report the "mean ± std". We compare the performance with other stateof-the-art acceleration algorithms, e.g., [Dong et al., 2017a; Li et al., 2017; He et al., 2017; Luo et al., 2017].

4.2 ResNet on CIFAR-10

Settings. For CIFAR-10 dataset, we test our SFP on ResNet20, 32, 56 and 110. We use several different pruning rates, and also analyze the difference between using the pre-trained model and from scratch.

Results. Tab. 1 shows the results. Our SFP could achieve a better performance than the other state-of-the-art hard filter pruning methods. For example, [Li et al., 2017] use the hard pruning method to accelerate ResNet-110 by 38.6% speedup ratio with 0.61% accuracy drop when without fine-tuning. When using pre-trained model and fine-tuning, the accuracy drop becomes 0.20%. However, we can accelerate the inference of ResNet-110 to 40.8% speed-up with only 0.30% accuracy drop without fine-tuning. When using the pre-trained model, we can even outperform the original model by 0.18% with about more than 40% FLOPs reduced.

These results validate the effectiveness of SFP, which can produce a more compressed model with comparable performance to the original model.

4.3 ResNet on ILSVRC-2012

Settings. For ILSVRC-2012 dataset, we test our SFP on ResNet-18, 34, 50 and 101; and we use the same pruning rate 30% for all the models. All the convolutional layer of ResNet are pruned with the same pruning rate at the same time. (We do not prune the projection shortcuts for simplification, which only need negligible time and do not affect the overall cost.)

Results. Tab. 2 shows that SFP outperforms other stateof-the-art methods. For ResNet-34, SFP without finetuning achieves more inference speedup to the hard pruning method [Luo et al., 2017], but the accuracy of our pruned model exceeds their model by 2.57%. Moreover, for pruning a pre-trained ResNet-101, SFP reduces more than 40% FLOPs of the model with even 0.2% top-5 accuracy increase, which is the state-of-the-art result. In contrast, the performance degradation is inevitable for hard filter pruning method. Maintained model capacity of SFP is the main reason for the superior performance. In addition, the non-greedy all-layer pruning method may have a better performance than the locally optimal solution obtained from previous greedy pruning method, which seems to be another reason. Occasionally, large performance degradation happens for the pretrained model (e.g., 14.01% top-1 accuracy drop for ResNet50). This will be explored in our future work.

To test the realistic speedup ratio, we measure the forward time of the pruned models on one GTX1080 GPU with a batch size of 64 (shown in Tab. 3). The gap between theoretical and realistic model may come from and the limitation of IO delay, buffer switch and efficiency of BLAS libraries.

4.4 Ablation Study

We conducted extensive ablation studies to further analyze each component of SFP.

Filter Selection Criteria. The magnitude based criteria such as ℓp-norm are widely used to filter selection because computational resources cost is small [Li et al., 2017]. We compare the ℓ2-norm and ℓ1-norm. For ℓ1-norm criteria, the accuracy of the model under pruning rate 10%, 20%, 30% are 93.68±0.60%, 93.68±0.76% and 93.34±0.12%, respectively. While for ℓ2-norm criteria, the accuracy are 93.89±0.19%, 93.93±0.41% and 93.38±0.30%, respectively. The performance of ℓ2-norm criteria is slightly better than that of ℓ1-norm criteria. The result of ℓ2-norm is dominated by the largest element, while the result of ℓ1-norm is also largely affected by other small elements. Therefore, filters with some large weights would be preserved by the ℓ2norm criteria. So the corresponding discriminative features are kept so the performance of the pruned model is better.

Varying pruning rates. To comprehensively understand SFP, we test the accuracy of different pruning rates for ResNet-110, shown in Fig. 3(a). As the pruning rate increases, the accuracy of the pruned model first rises above the baseline model and then drops approximately linearly. For the pruning rate between 0% and about 23%, the accuracy of the accelerated model is higher than the baseline model. This shows that our SFP has a regularization effect on the neural network because SFP reduces the over-fitting of the model.

Sensitivity of SFP interval. By default, we conduct our SFP operation at the end of every training epoch. However, different SFP intervals may lead to different performance; so we explore the sensitivity of SFP interval. We use the ResNet110 under pruning rate 30% as a baseline, and change the SFP interval from one epoch to ten epochs, as shown in Fig. 3(b). It is shown that the model accuracy has no large fluctuation along with the different SFP intervals. Moreover, the model accuracy of most (80%) intervals surpasses the accuracy of one epoch interval. Therefore, we can even achieve a better performance if we fine-tune this parameter.

Selection of pruned layers. Previous works always prune a portion of the layers of the network. Besides, different layers always have different pruning rates. For example, [Li et al., 2017] only prunes insensitive layers, [Luo et al., 2017] skips the last layer of every block of the ResNet, and [Luo et al., 2017] prunes more aggressive for shallower layers and prune less for deep layers. Similarly, we compare the performance of pruning first and second layer of all basic blocks of ResNet-110. We set the pruning rate as 30%. The model with all the first layers of blocks pruned has an accuracy of 93.96 ± 0.13%, while that with the second layers of blocks pruned has an accuracy of 93.38 ± 0.44%. Therefore, different layers have different sensitivity for SFP, and careful selection of pruned layers would potentially lead to performance improvement, although more hyper-parameters are needed.

5 Conclusion and FutureWork

In this paper, we propose a soft filter pruning (SFP) approach to accelerate the deep CNNs. During the training procedure, SFP allows the pruned filters to be updated. This soft manner can maintain the model capacity and thus achieve the superior performance. Remarkably, SFP can achieve the competitive performance compared to the state-of-the-art without the pre-trained model. Moreover, by leveraging the pre-trained model, SFP achieves a better result and advances the stateof-the-art. Furthermore, SFP can be combined with other acceleration algorithms, e.g., matrix decomposition and lowprecision weights, to further improve the performance.

Acknowledgments

Yi Yang is the recipient of a Google Faculty Research Award. We acknowledge the Data to Decisions CRC (D2D CRC), the Cooperative Research Centres Programme and ARC's DECRA (project DE170101415) for funding this research. We thank Amazon for the AWS Cloud Credits.