Self-boosting for Feature Distillation

1 Introduction

Knowledge distillation (KD) is a hot topic in deep learning. With the continuous development of portable devices, the demand of cost-efficient and well-behaved deep models is increasing, such as deep object-detection models, deep segmentation models and so on. Knowledge distillation is a useful tool for model compression, which exploits additional information of the well-trained large network (Teacher) to help the small network (Student) to train. It simply and effectively improves the performance of small deep models.

Though KD makes promising results in applications of computer vision, Mirzadeh et al. [Mirzadeh et al., 2020] found that Student cannot imitate Teacher perfectly when the model size of Teacher is considerable large. When the gap of Teacher and Student in model size is large, the performance of Student is far worse than that of Teacher. Thus, Teacher Assistant (TA) is proposed to alleviate the gap between Teacher and Student in [Mirzadeh et al., 2020]. However, TA needs to add an additional network to assist Teacher to guide Student, which is time consuming and high resource consumption.

*Corresponding Author.

Some latest methods focus on improving Teacher's guidance to Student without the TA network. In [Xu et al., 2020a], the feature from Teacher is normalized in order to appropriately simplify learning goals of Student. In [Yue et al., 2020], an additional feature matching optimization needs to be performed at each iteration. Obviously, such methods may cause part important information from Teacher to be lost. Besides, some methods [Kim et al., 2018] add convolution modules on Teacher, which requires additional pre-training steps.

Different from the abovementioned methods which focus on lowering the capacity of Teacher or exploring novel knowledge, we propose a novel distillation method named Self-boosting Feature Distillation (SFD) which enhances the ability of Student by self-boosting to bridge the gap of Teacher and Student. In other words, we aim to improve Student's learning ability by Student's self-boosting, rather than reducing the quality of Teacher's knowledge. SFD contains two aspects: feature boosting and model-parameter boosting. Concretely, as for feature boosting, we adopt feature integration strategy to enhance the discriminability of Student's feature by a carefully designed feature integration module. Student's integrated feature is encouraged to imitate Teacher's original feature, which builds a bridge between Student and Teacher and prompts Student to adaptively pay attention to Teacher's useful information. Note that our feature integration module is only used during the training phase, which is jointly optimized with Student and introduces a little computation. As for model-parameter boosting, we propose an easy-to-operate self-distillation method which does not require additional forward propagation or memory, to stabilize training process and promote Student's behavior. Compared with previous methods, our method does not require additional pre-training steps, while retaining Teacher's information to the greatest extent. Furthermore, SFD can be explained in theory while other methods are only explained empirically. As shown in Figure 1, SFD helps Student to learn more stable and converge faster.

Our contributions can be summarized as follows:

• We propose a novel distillation method called Selfboosting Feature Distillation (SFD), which bridges the gap between Teacher and Student by self-boosting in feature integration and self-distillation of Student. Unlike the existing methods which focus on lowering Teacher's capability, SFD improves the capability of Student.
• We design three feature integration modules to improve the discriminability of Student, in order to reduce the difference between Teacher and Student in model discrimination. Besides, self-distillation is proposed to further promote the convergency of Student, in which only the parameters of the previous model are used, so no additional forward propagation or memory is required.
• Unlike the existing methods which bridge the TeacherStudent gap experimentally, we explain SFD in theory of Richardson extrapolation: the feature integration increases the order of convergence.
• The proposed method is evaluated on multiple benchmarks and networks. Experimental results show that our method greatly enhances the performance of Student.

2 RelatedWork

2.1 Knowledge Distillation

Most researches focus on exploring diverse Teacher knowledge. In [Komodakis and Zagoruyko, 2017], the difference between the attention maps of Teacher and Student is minimized to optimize Student. In [Park et al., 2019], multiple outputs of Teacher is treated as a structural unit, and Student is encouraged to learn Teacher's structured information. Variational Information Distillation (VID) [Ahn et al., 2019] defines the optimal transfer performance of middle layers as maximizing the mutual information between Teacher and Student. Contrastive Representation Distillation (CRD) [Tian et al., 2020] captures the relevance of instances and higherorder output dependence through the transfer loss based on contrastive learning. We do not mine new Teacher knowledge, but simply utilize the network's features and weights.

Some methods try to appropriately simplify Teacher's knowledge. In [Kim et al., 2018], Teacher's information is showed to be difficult for Student to understand, so Teacher's middle-layer features are transformed into a simpler representation by a paraphraser. Xu et al. [Xu et al., 2020a] proposed to decompose features into direction and magnitude, and encourage Student to learn the direction of Teacher. Recently, Matching Guided Distillation (MGD) [Yue et al., 2020] argues to pose matching features of Teacher and Student as an assignment problem. These methods may cause missing of Teacher's knowledge to varying degrees, and some even require additional pre-training steps (such as FT [Kim et al., 2018]) to train additional modules. We only do some transformations on Student's feature, so the transformation module can be trained simultaneously with Student, and avoid the missing of Teacher's information to the greatest extent.

2.2 Feature Integration

Feature integration is mainly used in object detection and semantic segmentation. Feature Pyramid Networks (FPN) [Lin et al., 2017] achieve a comparable effect with the image pyramid algorithm by accumulating the shallow and deep features; In [Li and Zhou, 2017], resized features from different layers with different resolutions are concatenated, followed by some downsampling blocks, which forms the new feature pyramid; DeepLab [Chen et al., 2018] utilizes dilated convolutions for multi-scale feature extraction to obtain richer feature information. We perform feature integration at the middle layers of Student to generate more discriminative features.

2.3 Self-distillation

Self-distillation is a kind of distillation using the information of Student itself. In [Xu and Liu, 2019], the authors propose a method which transfers knowledge between different distorted versions of the same training data. The actual batch size is twice that of conventional training (there are two versions of an image in each batch), which significantly increases the memory. Snapshot Distillation [Yang et al., 2019] proposes to use the model of a previous time step as Teacher and its output as transferable knowledge. Xu et al. [Xu et al., 2020b] take the average of the parameters from

(a) Layer-wise Integration

(b) Attention Integration

(c) Receptive-field Integration

Figure 3: The architecture of three different modules based on feature integration.

Student's past K time steps as Teacher, which also needs to calculate Teacher's output. Most methods of self-distillation inevitably increase the forward propagation calculation and memory consumption.

3 Methodology

Figure 2 shows the framework of SFD. There are two streamlines: Teacher and Student. The feature integration module bridges Teacher and Student. In the training stage, Student fits the feature of Teacher via distillation loss. Simultaneously, Student updates its own model parameters by selfdistillation. In the testing stage, we only use Student to predict the class without the feature integration modules.

3.1 Feature Distillation

In our method, we perform feature distillation which uses Teacher's feature as the transferable knowledge for Student to learn, rather than using the output distribution of the classes. Considering that features from higher layers are more distinctive, we simply adopt the feature of the last stage of Teacher as knowledge to guide Student's learning.

Feature distillation can be treated as a multi-task learning (learning classification labels and Teacher's feature). There is a certain correlation between these two tasks. Student contains three components in the training phase: an encoder fθ (CNN backbone), an integrator gθ (feature integration module) and a predictor qθ (classifier), as shown in Figure 2. Suppose that images X are fed into Student, and the outputs are formulated as follows:

where Z is the backbone's output, Zg is the result of feature integration by the integrator, and Zq is the classification result.

In previous feature-distillation methods, gθ is usually a module consisting of 1 × 1 convolution layers for channel alignment. Obviously, 1 × 1 convolution can only make a linear combination of features from different channels, which cannot alleviate the semantic differences between features of Student and Teacher. In order to make full use of the feature integration module to bridge the Teacher-Student gap, we propose three solutions (as shown in Figure 3): Layerwise Integration, Attention Integration and Receptive-field Integration.

Given a set of intermediate features {Zl}3l=1 in different layers of Student, and the target feature Y ∈ Rc×h×w from Teacher's last stage, the three integration methods can be formulated as follows. Note that we simply choose features of the last three stages if Student has more than three stages.

Layer-wise Integration Inspired by Feature Pyramid Networks (FPN) [Lin et al., 2017], we perform feature integration at the middle layers of Student. Layer-wise Integration (LI) is different from FPN in operation mechanism: LI only generates the feature with a single scale, and the integrated feature is not applied to subsequent classification. LI can be formulated as:

where Rescalel(·) is a transform function to rescale features from different stages to the same scale (h ×w) as Teacher's feature, which is a simple downsampling (if larger) or upsampling (if smaller) operation. Convl(·) implements 1 × 1 convolution to reduce channels of features, and the numbers of output channels are c4 , c4 and c2 , respectively. After that, the multi-layer integrated feature Zg is obtained through concatenating, smoothing and batch normalization operations in sequence. Note that Smoothing(·) is a convolution with kernel size c × c ×3×3.

Attention Integration LImodule treats the features of each channel equally, but in fact they have different degrees of importance. To solve this problem, we design an attention module named Attention Integration (AI) which can be formulated as:

Z = Conv4(Z ⊙ W)

where Rescale(·) resizes features to size of h × w, GAP(·) denotes the global average pooling, and ⊙ denotes the element-wise multiplication. {Convi}4i=1 represents bottleneck (1 × 1 convolution) layers, and kernel sizes are (c1 + c2 + c3) × cmid × 1 × 1, cmid × cmid 16 × 1 × 1, cmid 16 × cmid × 1 × 1 and cmid × c × 1 × 1, respectively. Note that cmid =max( c1+c2+c3 16 , 96).

Receptive-field Integration Given the feature Z from the last stage of Student, and the target feature Y ∈ Rc×h×w from the last stage of Teacher, Receptive-field Integration (RI) can be expressed as:

where AAPi(·) is a transform function consisting of adaptive average pooling (AAP) and 1 × 1 convolution layers, where the kernel sizes of convolution layers are all c4 × c4 × 1 × 1. Upsample(·) is a bilinear interpolation to rescale features to the size h × w, and a rescaling operation is also conducted on Z if its scale is not equal to h × w. Then the integrated feature of different receptive fields can be obtained through concatenation followed by smoothing and batch normalization. Note that the scales of Z1, Z2 and Z3 are h4 × w4 , h2 × w2 and 3×h 4 × 3×w 4 , respectively.

3.2 Theoretical Analysis

In this part, we give a theoretical explanation that SFD can help to improve the performance of Student. According to Richardson extrapolation, we can get better numerical results by the linear combination of several numerical results at different inputs due to the increase of the convergence order of a model. Take the 1-D function as an example, according to Taylor expansion, a simple function a(t) w.r.t t is calculated as:

It can be seen that the linear combination of two results at a(t +∆t) and a(t -∆t) increase the order of convergence.

We implement Richardson extrapolation on the function matrix of Student. Given Student's function matrix A(·), then the feature maps (denoted as A(ti)) from Student's different layers can be regarded as values of the function with different parameters (ti)). Let t0 denote Student's optimal parameters to imitate Teacher, the function can be formulated as:

Feature integration provides a trainable operation B, so:

where ⊙ is the element-wise operation of matrices. Note that with feature integration, the term with one order of derivation can be eliminated by learning suitable B, namely, ∑i Bi ⊙ (ti - t0)T ∂A ∂t = 0, so the function has a higher order of convergence. Therefore, SFD can help Student learn better.

3.3 Self-distillation

We propose a novel self-distillation method, which updates the parameters only by using those obtained by the previous epoch, without any additional forward propagation or memory consumption.

Given a network with a set of weights θ, and θt denotes the network's parameters of the t-th epoch. After the t-th training epoch, we perform an additional weights updation:

Considering that the fluctuation range of parameters gets very small during the later stage of training, if Eq. 8 is used to update the parameters, the parameters hardly change. Therefore, we make a little modification:

where hflip(·) is a random horizontal flip with probability 0.5. Eq. 8 is named Self Distillation (SD), and Eq. 9 is named Random Self Distillation (RSD). Since the network is expected to be robust to the horizontal flip of the data, it is reasonable to randomly flip parameters of the previous epoch.

In fact, our self-distillation strategy conducts an exponential moving average of parameters from previous epochs. On one hand, this method can effectively use the information of the previous epoch model without introducing additional calculations and memory consumption; on the other hand, the exponential moving average can effectively improve the stability of the training process.

3.4 Distillation Loss

In SFD, the loss function is used to guide feature distillation. Inspired by Overhaul [Heo et al., 2019], we measure the distance between the feature before activation layer of Teacher and the integrated feature before ReLU of Student to conduct feature imitation. The distillation loss is formulated as:

where s is Student's integrated feature before ReLU, and t = max(Y, m) in which Y is Teacher's feature before ReLU and m < 0 is a margin value computed as expectation over all training samples.

4 Experiments

In this section, we evaluate SFD on several different benchmarks, including classification and fine-grained recognition. CIFAR-100 [Krizhevsky et al., 2009] is a commonly used small dataset for classification, which contains 60, 000 RGB color images within 100 classes (50, 000 training images and 10, 000 test images) with a resolution of 32 × 32. CUB200 [Wah et al., 2011] is a dataset for fine-grained recognition, which consists of 11, 788 images of different birds. ImageNet [Russakovsky et al., 2015] is a large-scale classification benchmark which has around 1.2 million images in 1, 000 classes.

4.1 Experimental Settings

In experiments, we compare our method with several state-ofthe-art methods 1 : standard KD [Hinton et al., 2015], AT [Komodakis and Zagoruyko, 2017], VID [Ahn et al., 2019], CRD [Tian et al., 2020], Overhaul [Heo et al., 2019] and MGD [Yue et al., 2020] under multiple network structures to verify the effectiveness of our method. For the fairness of comparison, we use the experimental settings of the compared methods presented publicly by the authors. As for SFD, the weight of distillation loss is 3.

For all networks, we use stochastic gradient descent (SGD) optimizer with momentum 0.9 and weight decay 5×10-4 . On CIFAR-100, models are trained for 240 epochs with an initial learning rate of 0.05 and divided by 10 at epoch 150, 180 and

1 We used a reference implementation: https://github.com/ HobbitLong/RepDistiller.git

210, and standard data augmentation schemes (padding 4 pixels, random cropping, random horizontal flipping) are carried out. On ImageNet and CUB-200, the number of total epochs is 100 and 120 respectively, the learning rate is dropped by 0.1 per 30 epochs, and we perform random cropping and horizontal flipping as data augmentation. Following [Tian et al., 2020], we use the model denotation: WRN-d-w is a Wide ResNet with depth d and width factor w; resnetd is a CIFARstyle resnet with depth d and basic blocks; resnetdxw is a w times wider network; ResNetd represents an ImageNet-style ResNet with depth d and bottleneck blocks.

4.2 Ablation Studies

Ablation studies are performed on CIFAR-100 to explore the influence of different solutions of feature integration and hyperparameters on the performance of models.

Different Components As shown in Table 1, all of the three feature integration methods we propose can greatly improve the performance of Student, whether Teacher and Student are homogeneous or heterogeneous. Considering the effects and complexity of the three feature integration solutions, we adopt RI as the final solution. Besides, both self-distillation methods are useful for improving Student's performance, and random self-distillation (RSD) behaves slightly better.

Hyperparameters of Self-distillation Different values of τ (in Eq. 9) have much influence on the performance of selfdistillation. We select WRN-16-2 and ShufflenetV2 for experiments separately, and comparison of model accuracy with different τ is shown in Table 2. Obviously, τ with different values can help to improve the performance of models to some extent, and models reach the highest accuracy when τ is linear growth from 0.995 to 0.999 with current epochs.

4.3 Comparison with SOTAs

CIFAR-100 We conduct experiments including homogeneous and heterogeneous Teacher-Student combination. Compared with previous methods, our method provides consistent gains in all Teacher-Student frameworks and is significantly better than other methods. For example, in Table 3, when the Teacher-Student combination is resnet110

and resnet20, our method achieves the gain in accuracy by 0.83% compared with CRD (ranked second). In Table 4, when the Teacher-Student combination is resnet32x4 and ShuffleNetV2, the accuracy of our method is 2.29% higher than CRD (ranked second).

ImageNet On ImageNet, we use ResNet34 as Teacher and ResNet18 as Student. The results are shown in Table 5. It can be seen that SFD achieves the best performance with the gain of 0.43% against the second place CRD+KD. Obviously, our method also works well in large-scale classification scenarios.

CUB-200 As shown in Table 6, for all configurations, our method achieves significant accuracy gains over other methods used for comparison. When the Teacher-Student combination is ResNet50 and ShuffleNetV2, the top-1 accuracy of SFD is 3.34% higher than MGD (ranked second). Besides, when Teacher is ResNet50 and Student is MobileNetV2, the accuracy of Student even surpasses Teacher with the gain of 1.45%. This may be because Student is sufficient to learn from Teacher, and SFD allows Student to better balance the relationship between self-study and imitation.

4.4 Analysis

It is recognized that integrated features are more discriminative. While our method does not use the feature integration module in the inference phase, but directly adopt the original feature extracted by Student for classification. To verify that Student learns under SFD framework, we visualize Student's (ShuffleNetV2, distilled by a well-trained ResNet50) training loss curves with the number of epochs. As shown in Figure 1, under the action of SFD, the training process is more stable and Student converges faster.

Besides, we make some comparison of several items on the test set: top-1 accuracy, classification loss, feature distance, and CKA similarity [Kornblith et al., 2019] between Student and Teacher. As shown in Table 7, our method achieves the best accuracy (1.60% higher than Overhaul), and has the lowest classification loss on the test set. Moreover, the KL divergence between outputs of Student and Teacher is the smallest, and the Teacher-Student similarity is the highest. Above all, our method makes Student imitate Teacher better.

5 Conclusion

In this paper, we tackle the Teacher-Student gap problem from a new perspective: self-boosting of Student rather than the previous methods lowering the level of Teacher. We propose the self-boosting feature distillation (SFD) method. To improve the learning ability of Student, self-boosting is conducted on Student, which contains two aspects: feature integration of its own feature and self-distillation on the parameters of Student, so Student adaptively learns from Teacher. SFD achieves state-of-the-art performance on multiple datasets with different Teacher-Student architectures. Theoretical analysis shows our method can improve the order of convergence. Extensive experiments shows that our method is significantly superior to other methods.

Acknowledgements

This work is supported by the National Key Research and Development Program of China No.2020AAA0108301, the National Natural Science Foundation of China under Grant 61876161, Shanghai Municipal Science and Technology Major Project (Grant No.2018SHZDZX01) and ZJLab.