Learning from Students: Online Contrastive Distillation Network for General Continual Learning

1 Introduction

Human beings have the gift of quickly learning new knowledge on the basis of learned knowledge without interfering the stability of learned knowledge, but it remains challenging in current machine learning technology. On the one hand, the new task will seriously interfere with the old task's performance when the neural network is trained directly on the new task, which is the notorious catastrophic forgetting problem [McCloskey and Cohen, 1989]. On the other hand, training a neural network on both old and new tasks obviously consumes more resources. Continual learning, also known as incremental learning and lifelong learning, has been developed

∗Corresponding Author

(a) Offline Distillation

(b) Online Distillation (Ours)

Figure 1: The illustration of utilizing knowledge distillation in GCL. (a) Diagram of the popular offline distillation approach. At time step t + 1, the teacher model's response is biased to the classes from the time step t when the reserved sample x(t) from time step t is input into the teacher model. (b) Diagram of the proposed online distillation approach. The teacher model is trained simultaneously by integrating the student model's parameters, which is beneficial for obtaining less biased responses to guide the training process of the student model. Note that there are responses for time step t + 1 since it is essential to initialize the model by providing ample class classification heads to avoid aligning outputs of different length.

for addressing these problems. It focuses on quickly learning new tasks from a non-stationary data stream and overcoming the catastrophic forgetting problem [Delange et al., 2021].

Currently, lots of studies simplify the continual learning problem by providing the additional prior knowledge, such as assuming that tasks don't overlap each other and applying task boundaries during training stage [Tao et al., 2020], [Liu et al., 2021], [Zhao et al., 2021], [Quang et al., 2021]. However, these prior knowledge don't always exist in the real-world scenarios where tasks are complex and diverse. Therefore, General Continual Learning (GCL) has been proposed to meet the requirements of real-world scenarios. It emphasizes that the training and testing processes of continual learning don't rely on the task boundaries and the memory size is bounded. There are mainly two lines of approaches to tackle this issue: Sample strategy-based approaches [Rahaf et al., 2019], [Ji et al., 2021b] and knowledge transfer-based approaches [Buzzega et al., 2020], [Cha et al., 2021]. The former selects vital samples by designing sample strategies while the latter utilizes knowledge distillation to transfer the learned knowledge. Specifically, the knowledge transfer-based approaches usually leverage the old model as the teacher model and distill its response to help the new model alleviate the catastrophic forgetting.

However, the response bias towards classes of current step is also inherited, which limits the positive effect of knowledge distillation, as shown in Fig. 1. This response bias is caused by the imbalanced data between the old and the new classes since the old classes are reserved in the buffer with a small amount of data while the new ones are input currently with abundant data. It leads to the weights of the classifier (i.e. the last fully connected layer) towards classes of current step [Wu et al., 2019], [Zhao et al., 2020], [Ahn et al., 2021]. The old model's response helps the student model to alleviate the catastrophic forgetting on some old classes as well as accumulating inherent response bias when distilling the output of the teacher model to the student model. Thus, taking the old model as the teacher model has severe limitation.

Apart from distilling the response of samples, maintaining the relation among classes is also crucial to preserve the learned knowledge. The response of a single sample is not robust to model changes, which may mislead the process of knowledge distillation. That is, it inevitably damages the model's ability to preserve the learned knowledge by only aligning the response of a single sample without considering the internal structure information among classes.

To address the above limitations, an Online Contrastive Distillation Network (OCD-Net) is proposed. It includes an Online Response Distillation Module (ORD-Module) and a Contrastive Relation Distillation module (CRD-Module), respectively. The ORD-Module is designed to align those less biased responses between the teacher and the student models. Particularly, the teacher model is trained by accumulating the student model's parameters since the student model at each time step is good at classifying different classes. The CRDModule utilizes an embedding space to maintain the consistency of class similarities among classes, which is more robust to model changes. Moreover, an adaptive perception approach is devised to enforce the ORD-Module to fix attention on distilling the high-quality response. With the combination of ORD-Module and CRD-Module, the teacher model enables the student model to learn richer information to preserve the learned knowledge, the student model helps the teacher model accumulate new knowledge with less forgetting at the same time. The main framework of the proposed approach is illustrated in Fig. 2. The highlights are summarized as follows:

• We propose an Online Contrastive Distillation Networks (OCD-Net) for GCL, which leverages the merit of the student model by integrating their parameters to consolidate the learned knowledge and accumulate the new knowledge.
• To exploit internal structure information among classes, we further introduce a contrastive relation distillation objective function to maintain the consistency of class similarities between the teacher and the student models.
• We propose an adaptive distillation approach to perceive and strengthen the high-quality teacher model's response in online distillation, which is beneficial for the student model to consolidate the learned knowledge.

2 RelatedWork

To alleviate the catastrophic forgetting problem, a bounded replay buffer is usually utilized to store the old data in GCL. Typical works such as [David and Akansel, 2018], [Chaudhry et al., 2019], [Rahaf et al., 2019], and [Ji et al., 2021b] can be classified as sample strategy-based approaches, which focus on devising effective criteria for selecting the appropriate replay samples. For example, [David and Akansel, 2018] utilized reservoir sample strategy [Vitter, 1985] to ensure each sample to be selected in the buffer with an equal probability. Based on this approach, [Chaudhry et al., 2019] introduced the average episodic memory loss over the previous tasks to constrain the gradient of the model's update. [Rahaf et al., 2019] further proposed an approximated greedy strategy to increase the sample diversity between the gradient of individual samples. [Ji et al., 2021b] leveraged the training loss value as a metric to discard those less vital samples.

Another line of studies [Buzzega et al., 2020], [Simon et al., 2021], [Cha et al., 2021] can be summarized as knowledge transfer based approaches, which exploits knowledge distillation [Zhang et al., 2020], [Chen et al., 2021], [Zhu et al., 2021b] to mitigate the catastrophic forgetting via transfer the learned knowledge (e.g. logits and features) to the new model. For example, [Buzzega et al., 2020] utilized knowledge distillation between the old and the new models' response. [Cha et al., 2021] stored the old model and regulated the instance-wise changes in feature relation.

Our proposed OCD-Net falls into the category of the knowledge transfer-based approaches and is related to [Buzzega et al., 2020] and [Cha et al., 2021]. Different from them, we replace the offline teacher model with the online one, which effectively accumulates the new knowledge and consolidates the learned knowledge via integrating the student model's parameters.

3 Method

GCL aims at sequentially learning a model to classify all seen classes from a non-stationary data stream D without utilizing task boundaries. Specifically, a GCL model is trained by the labeled sample Dt = {(x, y)} at time step t. Additionally, we introduce a bounded replay buffer M = {(xi, yi)Bi=1 } with fixed size B to store few previous samples.

The architecture of our OCD-Net mainly consists of an Online Response Distillation Module (ORD-Module), and a Contrastive Relation Distillation Module (CRD-Module), as shown in Fig. 2. In the ORD-Module, we leverage the online distillation scheme to transfer less biased response from the teacher model. Meanwhile, the CRD-Module aims at maintaining the consistency of class similarities between the teacher and the student models, which reduces the intra-class variance. Moreover, an adaptive perception idea is developed to ensure that the student model learns high-quality response by adjusting the weights of each sample.

3.1 Online Response DistillationModule

To alleviate the inherent response bias of the teacher model in the offline knowledge distillation, we propose an Online Response Distillation Module (ORD-Module), which explores Student Model the merit of the student model to obtain a model with less forgetting on the learned knowledge. It aims at learning the reserved samples' response of the teacher model to preserve the learned knowledge. Obviously, the teacher model is crucial to preserve the learned knowledge in the online distillation scheme, in which a key challenge is how to train the teacher model. Inspired by the idea of momentum update [He et al., 2020] in contrastive learning, we simply train the teacher model by random momentum update, which randomly integrates the student model's parameters to accumulate the new knowledge. In this way, the teacher model is capable of achieving knowledge accumulation.

Figure 2: The main architecture of the proposed OCD-Net. We adopt the online knowledge distillation framework where the student model is trained to align the relation and response of the teacher model, and the teacher model is updated via a random momentum update of the student's parameters.

Formally, we employ two feature extractors fΘt and fΘs to encode feature representations from sample x. Then, two classifiers fΦt and fΦs map the corresponding feature representations into the label space. The student model is trained by the online response distillation loss, which is written as:

where the ∥·∥2 operator refers to the ℓ2 norm.

Meanwhile, the teacher model is trained by the random momentum update approach, which is formulated as:

where m is a momentum coefficient and X obeys Bernoulli distribution, which is denoted as:

where Bernoulli probability p is in range (0, 1).

By doing so, we avoid adjusting the particularly sensitive momentum coefficient m. Besides, to help the teacher model better accumulate new knowledge, the value of momentum coefficient m is designed to increase gradually in the early stage:

where n is the number of model iterations and constant η is set to 0.999.

3.2 Contrastive Relation DistillationModule

Apart from distilling the response from the teacher model, maintaining the consistency of class similarities is also essential to alleviate the catastrophic forgetting. Since samples from the same class usually have similar embeddings, the embeddings from the student model should be consistent with those in the same class in the teacher model. To this end, we introduce a contrastive relation distillation loss to encourage the same class embeddings to be pulled closer and the other samples be pushed away between the teacher and the student models. Concretely, we introduce two learnable projectors fΨt and fΨs to map the samples' features to an embedding space where the contrastive relation distillation loss is applied. The projector fΨt is also updated by randommomentum update:

Given a batch of data with samples x, the embedding can be expressed by zt = fΘt ,Ψt(x) and zs = fΘt ,Ψs(x). To maintain class similarities between the teacher and the student models within a batch, we fix the teacher's embedding zt as anchor and enumerate the student's embedding zs to approach the embeddings of the same class in the teacher model. The contrastive relation distillation loss is:

where zt+ represents the set of all teacher embeddings of the same label with zsi , and critic function h : {zi, zj} → [0, 1]

indicates whether the embedding tuple (zi, zj) is drawn from the joint distribution p(zi, zj), which is defined as:

where τ is a temperature hyper-parameter and the (·)⊤ operator means transpose.

In addition to maintaining the internal structure information with the teacher model, the student model also encodes the data's inherent co-occurrence relationships for better generalization. Following CoCa [Ji et al., 2021a], we also employ the idea of supervised contrastive learning [Khosla et al., 2020] to raise the critic values from the same class and reduce them from different classes, which is formulated as:

where zs+ represents the set of positive embeddings form zs .

Noticeably, the contrastive relation distillation loss Lcrd and supervised contrastive learning loss Lscl are cooperative. The former is designed to match the marginal feature distribution p(zs ) with p(zt ) for consolidating the learned knowledge while the latter learns marginal distribution p(zs ) for better generalization. By minimizing the Lcrd and Lscl simultaneously, the student model better consolidates the learned knowledge and learns new knowledge. Therefore, learning from the student model is capable of accumulating less biased knowledge.

3.3 Adaptive Perception

To further mitigate the catastrophic forgetting in our proposed OCD-Net, we deploy an adaptive perception approach to evaluate the quality of the teacher model's response, which enforces the student model to pay more attention to learn high-quality response. Specifically, the adaptive perception gets as input a sample response rt = fΘt ,Φt(xi) with its label yi, and yields as output a quality score ω(xi):

where ρ is the temperature parameter and C refers to the number of possible classes. By constructing the quality score, we dynamically adjust the contribution of each teacher model's response, in which a high score response is emphasized and a low score response is weakened in knowledge distillation. To this end, the online response distillation loss is substituted by the adaptive online distillation loss, which is denoted as:

3.4 Overall Objective

In summary, the total training objective for the student model in our proposed OCD-Net can be expressed by:

Algorithm 1 The training algorithm of the OCD-Net

Input: Stream data D, Buffer M, Learning rate γ, Bernoulli Probability p, Temperature hyper-parameters τ and ρ, Weighting factors α1, α2 and α3

Parameter: Teacher parameters Θt , Φt and Ψt , Student parameters Θs , Φs and Ψs

Output: Learned teacher parameters Θt and Φt

• 1: Initialize Θt = Θs ,Φt = Φs ,Ψt = Ψs ,M← {}.
• 2: for (X, Y ) inD do
• 3: M←Reservoir(M, (X, Y )).
• 4: (Xm, Ym) ←Sample(M).
• 5: Calculate Laod loss by Eq. (11).
• 6: zt ← fΘt ,Ψt(Xm), zs ← fΘs ,Ψs(Xm).
• 7: Calculate Lcrd loss by Eq. (7).
• 8: Calculate overall loss L by Eq. (12).
• 9: Update Θs , Φs , Ψs by gradient descent: ϑ ← ϑ - γ∇ϑL, ϑ ∈ {Θs , Φs , Ψs }.
• 10: Update Θt , Φt , Ψt by Eq. (2), Eq. (3) and Eq. (6).
• 11: end for
• 12: return Θt , Φt .

where α1, α2 and α3 are weighting factors, and the classification loss Lce is calculated by:

We elaborate on the overall training procedure of our proposed OCD-Net in Alg. 1. When new data come, we first utilize the reservoir sample strategy [Vitter, 1985] to update the buffer. Then, we optimize the student model by the overall objective L. Finally, we modify the teacher model by the randommomentum update approach. At the test stage, we leverage the teacher model for testing. The reason lies in that the student models at different time steps are good at classifying different classes, the teacher model learned from the student models could accumulatively learn their merits. Therefore, the teacher model has a stronger ability in classifying all seen classes than a student model.

4 Experiments

4.1 Datasets

We evaluate our OCD-Net on both spilt and smooth benchmarks. Split benchmarks mean that the classes of each task are disjoint and the number of classes of each task is equal. Following [Buzzega et al., 2020], [Chaudhry et al., 2019], we adopt three spilt benchmarks in our experiments: Split CIFAR-10 (S-CIFAR-10) [Krizhevsky and Hinton, 2009], Split CIFAR-100 (S-CIFAR-100) [Krizhevsky and Hinton, 2009] and Split Tiny-ImageNet (S-Tiny-ImageNet) [Hadi and Saman, 2015]. They consist of 5, 20 and 10 tasks, each including 2, 5 and 20 classes respectively. Smooth benchmarks refer to the class of each task emerges irregularly and the distribution of class shifts gradually, which are closer to real-world scenarios. For this type of datasets, we choose MNIST-360 [Buzzega et al., 2020] and generalized CIFAR-100 [Mi et al., 2020] datasets. Specifically, MNIST-360 dataset offers a stream of data in which each task constitutes by two continuous rotating digits. Generalized CIFAR-100 has two variants: Uniform Generalized CIFAR100 (UG-CIFAR-100) and Longtail Generalized CIFAR-100 (LG-CIFAR-100). Both of them have 20 tasks, the difference is whether the class distribution is uniform.

4.2 Implementation Details

Following [Buzzega et al., 2020], [Ji et al., 2021a], we adopt a fully-connected network with two layers for MNIST-360 dataset and ResNet-18 [He et al., 2016] for the other datasets as feature extractors. A linear layer and an MLP with one hidden layer are instantiated as classifier and projector, respectively. The stochastic gradient descent optimizer is employed for optimization. The model is trained on each task with 50 epochs for all datasets except for MNIST-360 dataset, which is trained with a single epoch.

4.3 Comparison with the State-of-the-Art

We compare our OCD-Net against a series of GCL approaches, including ER [David and Akansel, 2018], GSS [Rahaf et al., 2019], DER [Buzzega et al., 2020], DER++ [Buzzega et al., 2020], and CER [Ji et al., 2021b]. Meanwhile, we also modify A-GEM [Chaudhry et al., 2019], GeoDL [Simon et al., 2021], and CO2L [Cha et al., 2021] in line with the GCL setting. Besides, we also provide approximate upper and lower bounds on all datasets, in which the former is trained on all shuffled samples (JOINT) and the latter trains the model without any strategy (SGD). For the same dataset, all approaches utilize the same architecture to ensure a fair comparison.

Comparison on Split Benchmarks. Table 1 summarizes the average classification accuracy of OCD-Net and competitors on three split benchmarks with three different buffer sizes. We could observe that OCD-Net clearly outperforms the competitors in all cases. Particularly, it obtains at least 4.29% improvements among these datasets with different buffer sizes. For example, OCD-Net achieves 81.28%, 36.80% and 28.42% accuracies on S-CIFAR-10, S-CIFAR100 and S-Tiny-ImageNet datasets with 500 buffer sizes, outperforming the second-best approaches by 4.65%, 6.30% and 8.30%, respectively. Similar to other approaches, it could also be observed that increasing the buffer size is beneficial to improve the performance. Moreover, it can be observed that the performance on S-CIFAR-10 dataset is higher than S-CIFAR100 and S-Tiny-ImageNet datasets since it has fewer classes and tasks than the other datasets.

Comparison on Smooth Benchmarks. Table 2 reports the results of the comparison with the same eight competitors on three smooth benchmarks. It demonstrates that our proposed OCD-Net improves steadily on all datasets under all buffer sizes. Concretely, our OCD-Net achieves an 8.71% improvement over the sub-optimal approach DER on UG-CIFAR-100 dataset with 200 buffer sizes, which is quite notable. It can also be observed that the results on LG-CIFAR-100 dataset perform worse than those on UG-CIFAR-100 dataset, which illustrates the imbalance of classes aggravate the catastrophic forgetting problem.

4.4 Ablation Studies

Table 3 shows the ablation studies for each component in OCD-Net on S-CIFAR-10 and UG-CIFAR-100 datasets. We first validate the impact of the buffer by removing the replay buffer (OCD w/o BUF). Then, we train OCD-Net by removing the CRD-Module and the ORD-Module (OCD w/ ON). Finally, we train OCD-Net by removing the CRD-Module (OCD w/o CRD), the ORD-Module (OCD w/o ORD) and the adaptive perception (OCD w/o AP), respectively. From the results, we have the following observations: (1) OCD w/o BUF brings a sharp decline on both datasets. The results of OCD w/o BUF and OCD w/ ON indicate that it is essential to help the student model to preserve the learned knowledge in our OCD-Net. (2) The proposed ORD-Module, CRDModule and adaptive perception approach are all proved to be positive in OCD-Net. Moreover, they are mutually complementary in mitigating the catastrophic forgetting of the student model.

4.5 Quantitative Analysis

The Impact of Bernoulli Probability p. An experiment is conducted to further discuss the influence on the teacher and the student models of the Bernoulli probability p in Eq. (4), as shown in Fig. 3. This Bernoulli probability is designed to control the updating frequency of the teacher model. From the results, we could observe similar trends in both the teacher and the student models, indicating that the student model is crucial for the teacher model in online distillation schemes. Additionally, the teacher model achieves the apex when probability p = 0.2. We speculate that higher probability accumulates more inherent bias, while lower probability accumulates less learned knowledge.

Norms of the Weight Vectors in the Classifier. Figure 4 shows the norms of the weight vectors in the classifier (i.e. the last fully connected layer) on LG-CIFAR-100 dataset. We take the popular offline distillation method DER++ [Buzzega et al., 2020] for comparison. As presented in Fig. 4(a), the norms of the weight vectors show a large variance, which illustrates the classifier is biased towards some classes. By contrast, the weight learned by our proposed OCD-Net is obviously less biased, as shown in Fig. 4(b). It proves that the proposed online distillation scheme is capable of alleviating the inherent classifier bias widely exists in the offline distillation scheme.

5 Conclusion

In this paper, we have proposed the Online Contrastive Distillation Networks (OCD-Net) for GCL, which exploits a strong teacher model by accumulatively learning the merit of the student model via an online distillation scheme. The teacher model is trained via integrating the student model's parameters to accumulate the new knowledge while the student model is trained via maintaining the relation and adaptive response to consolidate the learned knowledge. Therefore, the teacher model integrated from the student models not only quickly learns new knowledge but also effectively alleviates the catastrophic forgetting. The experimental results on six benchmarks demonstrate that the proposed OCD-Net outperforms state-of-the-art approaches by a large margin. Future works include extensions of the OCD-Net on more challenging scenarios like semi-supervised GCL [Brahma et al., 2021] and non-exempler GCL [Zhu et al., 2021a].

Acknowledgments

This work was supported by the National Natural Science Foundation of China (NSFC) under Grants 62176178.