Improving Deep Neural Network Sparsity through Decorrelation Regularization

1 Introduction

In recent years, deep neural networks (DNNs) have become the dominant tool for most computer vision tasks. Most classic DNN models are designed for pursuing the best performance. However, for many of the potential application scenarios such as mobile phones and embedded systems, the energy consumption and computation resources are extremely limited. The model size and computation complexity thus become a bottleneck when deploying modern DNN models to such platforms [Sun et al., 2016; Zhou et al., 2014; Cai et al., 2016]. This difficulty motivates researchers to develop more compact and efficient DNN models without hurting the state-of-the-art performance.

The deep neural network compression is first explored by [Han et al., 2015]. They find that removing most of the DNN parameters according to their magnitude does not affect the network performance. Since then, various network compression methods are proposed [Li et al., 2017; Lin et al., 2016;

Luo et al., 2017; Zhou et al., 2018; Zhu et al., 2018]. Among these methods, training with group LASSO regularization draws much attention [Wen et al., 2016; Liu et al., 2017; Huang and Wang, 2017]. Compared with other compression methods which modify a pretrained large network and finetune the compressed model, group LASSO guides the network to learn the sparse representation during training. Another advantage of group LASSO is its structured sparsity inducing property. Although removing non-structured weights can also result in a sparse network, the computation algorithms involved with non-structured sparse weights are not trivial. Models with structured sparsity can be easily implemented with most off-the-shelf deep learning libraries. However, by investigating the weights of networks trained with group LASSO regularization, we observe that even for a well trained sparse network, there still exist significant correlation among the convolution filters. The existence of correlation indicates that there is still room for further compression.

In this paper, we propose the filter decorrelation regularization for reducing the filter correlation and improving the network sparsity. During the training process, a sparse mask is maintained for each layer, and we minimize the correlation of the filters which have not been masked. This regularization explicitly forces the network to learn a set of less correlated filters, and the sparse mask ensures that decorrelation regularization is only imposed to those indispensable filters. The overall structure of our proposed regularization is shown in Figure 1.

We evaluate our proposed method on several popular DNN architectures. Experimental results show that by using our decorrelation regularization, the filter correlation could be effectively weakened. We also empirically demonstrate that weakening the correlation among convolution filters will result in a more sparse network, which makes our compression results superior than using the sparsity regularization solely and other comparison network compression methods.

2 RelatedWorks

In this section, we briefly review the related works in network compression, which could be summarized into four aspects: network quantization, weight pruning and decomposition, sparsity regularization, and specially designed architectures.

2.1 Network Quantization

Observing that tiny perturbation on parameters makes little degradation on the network performance, many works try to compress a DNN by reducing the required number of bits for each parameter. In [Courbariaux et al., 2015], sign function is used for weight quantization and the weights are constrained in {+1,-1}. Binary weights dramatically reduce the model storage size, but the acceleration effect is not obvious. XNOR-Net makes further acceleration by using 1-bit for both network weights and layer activations. Thus the convolution computation could be replaced with more efficient XNOR operations [Rastegari et al., 2016].

2.2 Weight Pruning and Decomposition

Network pruning is a widely explored network compression method. In [Han et al., 2015], weight magnitude is used as pruning guidance to prune out weights with small absolute value. Due to the difficulty of unstructured sparsity computation, Hu et al. define the average percentage of zeros (APoZ) as the filter importance measurement, if a filter always outputs zero, then the whole filter could be safely removed [Hu et al., 2016]. Entropy based pruning uses the entropy of filter's output as filter importance measurement, and prunes the filter which has low output entropy [Luo and Wu, 2017]. Weight decomposition is another structural compression method. Lebedev et al. adopt CP decomposition to factorize a 4-D convolution kernel into four small 1-D convolution kernels [Lebedev et al., 2015]. Similarly, in [Kim et al., 2015], Tucker decomposition is used to transform a 4D convolution kernel into two small 1-D kernels and a 2D kernel.

2.3 Sparsity Regularization

As mentioned before, adding sparsity regularization to network weights could learn a sparse network directly and usually achieves better compression rate than pruning a pretrained network. In [Wen et al., 2016], group LASSO is applied to the filters, channels, filter shapes and residual blocks as a structured sparsity regularization. The group LASSO regularization [Yuan and Lin, 2006] makes the parameters in some groups all zero, while parameters in other groups are all non-zero. Some works add a gate variable to the filters, channels, or residual blocks, then apply the LASSO regularization to these gate variables [Huang and Wang, 2017; Liu et al., 2017]. After training, the filters, channels, or residual blocks which correspond to zero valued gates can be safely removed. In [Yoon and Hwang, 2017], it not only use group LASSO for network sparsity, but also introduces the exclusive LASSO on each layer's input channels. They claim that exclusive LASSO forces each filter to use different inputs, thus the hidden layers will learn more meaningful features.

2.4 Specially Designed Architectures

Many works try to design a more efficient network architecture directly. In [Howard et al., 2017] the standard convolution layer is replaced by a cheap layer-wise convolution and a point-wise convolution to reduce the convolution computation. Wu et al. introduce a zero FLOP shift layer, which shifts feature maps to different directions [Wu et al., 2017]. They claim that using this as an alternative to the spatial convolution can achieve state-of-the-art performance with much less computation.

3 Method

In this section, we investigate the DNN trained by group LASSO regularization, and show that strong filter correlation still exists in a sparse network. Then we describe our proposed decorrelation regularization algorithm in detail.

3.1 Motivation

For a convolutional neural network, the convolution kernel in layer l is a 4-dimensional tensorWl∈RDl×Dl×Cl×Nl , where Dl is the spatial size of the convolution kernel in layer l, Cl is the number of input channels, and Nl is the number of filters in layer l. The filter-wise structured sparsity in [Wen et al., 2016] divides each weight tensor into N groups W(g) , g = 1, · · · ,N, where each group contains the weights of one single filter, and applies group LASSO to these filter groups as a regularization termRS in the loss function:

where

Similarly, the channel-wise structured sparsity applies the group LASSO regularization to C groups of weights W(g) , g = 1, · · · ,C in each layer, where each group contains the weights that connected to a single input channel. The group LASSO forces some groups of weights all zero, while other groups are all non-zero, thus the network is regularized to have structured sparsity.

Group LASSO regularization effectively removes unimportant weights during training, but it remains uncertain if the resulting model does reach the limit of compression. To answer this question, we investigate the correlation between filters. A filter can be represented by other filters if it's correlated with these filters, thus the filter correlation indicates the redundancy in the network. To measure the filter correlation, we define the mean filter correlation Corrl of the filters in layer l as:

where C is the Pearson correlation coefficient matrix, and Ci is the ith row of matrix C. The mean filter correlation of the whole network is then defined as:

Intuitively, the mean filter correlation picks the most correlated filter pairs and measures the mean correlation value of these pairs. Corr close to 1 means that most filters in the network are correlated. For a set of fully uncorrelated filters, the corresponding mean filter correlation should be zero.

We train a VGG-16 model on CIFAR10 dataset with the coefficient η of group sparsity regularization picked from a wide range, and measure the mean filter correlation of the trained networks. Table 1 shows the test accuracy on CIFAR10 as well as compressed weight size and mean filter correlation. It could be observed that although the weight size keeps decreasing as the coefficient η increases, the mean filter correlation does not decrease significantly. The mean filter correlation of sparsified networks stays close to the original dense model until the test accuracy begin to drop, which indicates that the group LASSO regularization does not effectively reduce the correlation among filters.

We compute the Pearson correlation coefficient matrix of first layer filters in original VGG-16 and its sparse regularized counterpart. The absolute value of correlation matrix is shown in Figure 2. It could be seen that although many filters are removed by the group LASSO regularization, there still remains a noticeable correlation within the remaining filters. This phenomenon inspires us that removing filter correlation may be the key to the further network compression.

3.2 Filter Decorrelation

Motivated by removing filter correlation in a network, a straightforward solution is to perform low-rank tensor decomposition as in [Lebedev et al., 2015; Kim et al., 2015; Lin et al., 2016]. Although these decomposition methods can find a compact representation of the network parameters, their acceleration results are not satisfactory. Since each convolution layer in the original network needs to be replaced with several convolution layers, it leads to dramatically increased network depth and temporary data size.

Instead of decomposing pretrained weights, we propose to minimize the filter correlation during training, and learn a set of less correlated filters directly. To achieve this goal, it's natural to extract the non-diagonal elements of filter correlation matrix, and add its L2-norm to the loss function, which will act as another regularizer and reduce the filter correlation directly:

However, due to the structured sparsity induced by group LASSO regularization, the number of valid filters is changing dynamically during the training process. The filters that have values all close to zero will become uncorrelated to other filters. Thus the loss value will be drastically reduced by these "dying" filters. Another concern is that the dead filters will be removed right after training, reducing the correlation between these unnecessary filters does not benefits the final performance. Therefore we introduce a sparse mask M l to the filters in layer l. If filter k's maskM lk has value 1, then it will be sent to compute the correlation matrix, while the filters with mask value 0 will be skipped in this iteration. We calculate the sparse mask according to the mean absolute values of each filter:

where τ is a predefined threshold, and we use mean absolute values to prevent the undesired weighting effect caused by the different number of filters in different layers. With the sparse mask, the improved filter decorrelation regularization term is formulated as:

where ˆCl is the correlation matrix of the unmasked filters in layer l. We also divide the squared L2-norm by ∑k M lk , which is the number of unmasked filters in layer l. Otherwise, when the number of valid filters changed, the dimension of correlation matrix will change, which will cause the loss value fluctuate drastically.

In structured sparsity regularization, the filter-wise structural sparsity works together with the channel-wise structural sparsity, because removing the input channels in current layer is equivalent to removing the corresponding filters in last layer. Such connection could also be inherited to our proposed method. The channel decorrelation regularization has the same form with filter decorrelation regularization, except that the correlation matrix is computed from input channels, and the sparse mask is determined by the mean absolute values of each input channel. In the experiments, we combine both two types of decorrelation regularizations.

The total loss for network training is represented as:

where Lt is the task related loss (e.g., cross entropy for image classification), the second term is the L2 regularization on weights (i.e., weight decay), the third term in Eq. 8 is the structured sparsity regularization, and the last term is the proposed filter decorrelation regularization. λ, η and γ are the hyper parameters that control the magnitude of these three regularization terms, respectively.

Apart from the decorrelation regularization for training, the influence of weight initialization should also be taken into consideration. We adopt the SVD decomposition to generate orthogonal randommatrices for weight initialization. The weight tensorWl of layer l is first generated from the initialization method described in [He et al., 2015], and reshape the Dl×Dl×Cl×Nl tensorWl into a Nl×(Dl×Dl×Cl) matrix ˆWl . Therefore each row of this matrix is the weight of one filter. Then the SVD decomposition is applied to the weight matrix ˆWl :

If Nl < Dl×Dl×Cl, the matrix Vl will be a Nl×(Dl×Dl×Cl) unitary matrix, whose rows are all orthogonal to each other, and we use matrix Vl as the initialized weights in layer l. Notice that in some layers there are Nl > Dl×Dl×Cl, which means we can't find N orthogonal filters, in these layers we do not perform such orthogonal initialization.

4 Experiments

In this section, we report the experimental results of our proposed decorrelation regularization on several network architectures and datasets. The compression results are shown in terms of the weight size as well as FLOPs (floating-point operations). We also show the results of ablation studies in order to demonstrate the effectiveness of each individual component in our method.

4.1 Results on CIFAR10/100

CIFAR is a medium scale image classification dataset introduced in [Krizhevsky and Hinton, 2009]. The dataset has 50000 images for training and 10000 images for testing. The training images are mirror padded by 4 pixels, then random flipped and cropped for data augmentation.

We evaluate our proposed method on the widely used VGG-16 [Simonyan and Zisserman, 2014] and ResNet56 [He et al., 2016] architecture. For both architectures, the weight decay parameter λ is set to 0.0005, and the structured sparsity parameter η is 0.0015 for VGG-16, and 0.001 for ResNet-56. These are the values of η that result in highest sparsity without significant performance drop when only using structured sparsity regularization on VGG-16 and ResNet-56, respectively. We use the same η when combining structured sparsity with our decorrelation regularization to keep the comparison under the same condition. The decorrelation parameter γ is set to 5 and the sparsity threshold τ is set to 1e-4 according to grid search. We use stochastic gradient descent with momentum 0.9 for training. The initial learning rate is set to 0.1 and decays every 30 epochs with a factor of 0.5.

Evaluation on VGG-16 Table 2 shows the results of our proposed method compared with the naive group LASSO regularizaiton (SSL) [Wen et al., 2016], filter pruning [Li et al., 2017] and network slimming [Liu et al., 2017]. We can observe that with the comparable test accuracy, combing our filter decorrelation regularization with group LASSO always ends up in a more sparsified network than SSL. The model trained by our regularization has 6.8 MB of weights, which is only 11.7% of the baseline model, and the FLOPs is 31.3% of the baseline model. Our method also has a much higher compression rate than the state-of-the-art filter pruning method and the network slimming, which is another sparsity regularization based method. For CIFAR100 experiments, the advantage in compression rate of our method is also noticeable. The compression rate of our method is not as significant as it in the CIFAR10 experiments, we attribute this to the increased task complexity. The VGG-16 model is barely enough for the CIFAR100 dataset, thus there are not much redundancy in the original network.

To give a better illustration on the regularization effect, Figure 3 shows the correlation matrices of model regularized with and without decorrelation regularization, respectively. In contrast with SSL which only uses group LASSO, adding our decorrelation regularization causes most of the remaining filters less correlated after training. The number of remaining filters in each layer is shown in Figure 4. Most of the filters in last layers are removed during training, and our method reduce more filters than only using group LASSO in all layers. The rate of sparsity also indicates that the first several layers seem to be more important, where only few filters have been zeroed out.

Evaluation on ResNet-56 The baseline model of ResNet56 has a 93.39% accuracy on CIFAR10 and 71.59% accuracy on CIFAR100. The results are shown in Table 3. Compared with SSL [Wen et al., 2016], filter pruning [Li et al., 2017] and channel pruning [He et al., 2017], although the original ResNet model already has less redundancy than VGG-16, we still achieve a compression rate of 2.1× in weights and 1.9× in FLOPs. Again, our method outperforms the SSL and other comparison methods in compression rate of both weight size and computation.

4.2 Results on ILSVRC2012

ILSVRC2012 is a large scale image classification dataset with about 1.2 million images for training and 50000 images for validation [Deng et al., 2009]. In our experiments, the training images are randomly resized and cropped into 224×224, then transformed by random horizontal flipping and mean subtracting. Both training and validation images are subtracted by mean value and divided by standard deviation per channel.

We adopt the AlexNet [Krizhevsky et al., 2012] for the ILSVRC2012 experiments. Results of different ηs are listed to show the trade-off between compression rate and performance, γ and τ is also set to 5 and 1e-4, respectively. The network is trained by stochastic gradient descent with a momentum of 0.9. The initial learning rate is set to 0.01 and reduced by 0.1 at 60, 80 and 90 epochs.

Table 4 lists the experimental results on ILSVRC2012. On such a large scale dataset our method also outperforms SSL both in weights and FLOPs compression rate with an acceptable performance loss. It's worth noting that following by the settings in [Wen et al., 2016], we do not compress the fully connected layers. Since over 97% of the computations in AlexNet come from the convolution layers, and over 94%

of the weights in AlexNet are in the fully connected layers, the compression rate on weights is not as significant as it is on FLOPs. Combining our regularization with other fully connected layer compression methods will leads to better weight compression result.

4.3 Ablation Study

In Section 3 we state that sparse masks and orthogonal weight initialization are introduced for better decorrealtion results. Here we evaluate the impact of these components and the ablation experimental results are listed in Table 5. From the results of experiments without sparse mask, it could be seen that not only the compression rates are inferior to the models trained with sparse mask, but also the mean filter correlations are higher than the comparison models. This indicates that including the dead filters into correlation calculation introduces negative effect on other remaining filters. A lower mean filter correlation is achieved by adding the orthogonal weight initialization, which demonstrates that orthogonal initialization does benefit the following decorrealtion training.

To verify that whether using the decorrelation regularization alone is already enough for inducing network sparsity, we conduct experiments with only the decorrelation regularization on VGG-16. From the results in Table 6 we could observe that although the filter correlation has been suppressed, the resulting network does not exhibit a good compression rate. This observation indicates that only forcing the network filters to be less correlated does not necessarily make the filters sparse, thus the decorrelation regularization should be combined with group sparsity regularization to take effect.

5 Conclusion

In this paper we focus on the filter correlation in DNNs, which indicates that redundancy exists even in a sparse network model. To reduce the filter correlation and squeeze the redundancy out, we propose the decorrelation regularization, which explicitly minimizes the correlation among the filters. A sparse mask is introduced to remove the side effect of unimportant filters, and orthogonal weight initialization is adopted to give the filters a uncorrelated initial state. Experiments on CIFAR10/100 and ILSVRC2012 datasets demonstrate that our decorrelation regularization effectively weakens the filter correlation, and effectively compresses the corresponding DNN models.

Acknowledgements

This work was supported in part to Dr. Houqiang Li by 973 Program under contract No. 2015CB351803 and NSFC under contract No. 61390514, and in part to Dr. Wengang Zhou by NSFC under contract No. 61472378 and No. 61632019, the Fundamental Research Funds for the Central Universities, and Young Elite Scientists Sponsorship Program By CAST (2016QNRC001).