Joint Input and Output Coordination for Class-Incremental Learning

1 Introduction

In recent years, incremental learning has attracted much attention since it can play an important role in a wide variety of fields, including unmanned driving [Santoso and Finn, 2022] and human-computer interaction [Tschandl et al., 2020]. Incremental learning is nontrivial since the parameters of deep models in the old tasks are often destroyed in the process of learning new tasks. This leads to the occurrence of catastrophic forgetting [French and Chater, 2002]. How to well preserve past information and fully explore new knowledge has become a major challenge of incremental learning.

Existing incremental learning approaches mainly focus on memory storage replay [Ahn et al., 2021; Li and Hoiem, 2017; Wu et al., 2019; Rebuffi et al., 2017; Yan et al., 2021], model dynamic expansion [Serra et al., 2018; Mallya and Lazebnik, 2018], and regularization constraints design [Aljundi et al., 2019]. Memory store replay has been demonstrated to be very effective, and it alleviates the destruction of old task weights by storing past data or simulating human memory. However, due to the privacy restriction and limited memory, the data to be accessed from old tasks are often quite scarce. This makes incremental learning models suffer from severe inter-task class bias, or known as the class imbalance issue between old and new tasks.

∗Corresponding authors: Yong Luo, Wei Yu.

There exist some recent approaches [Ahn et al., 2021; Rebuffi et al., 2017; Yan et al., 2021] that alleviate the problem of class imbalance between old and new tasks by utilizing rescaling, balanced scoring, or softmax separating. Although these approaches can improve the performance to some extent, the problem of category imbalance still exists, since during the incremental learning progresses, the category imbalance becomes more severe as the number of sample categories continuously increase. Moreover, the mutual interference between old and new tasks has not been well addressed. That is, only the predictions in old tasks are tried to be maintained, and the output scores of old task data on the classification heads of new tasks are not well suppressed. The output consistency of new task data on old classification heads before and after updating the new task model is also not considered. Besides, none of the existing approaches deal with the class bias within tasks. An illustration is shown in Figure 1.

In order to address these issues, we propose a joint input and output coordination (JIOC) mechanism, which enables incremental learning models to simultaneously alleviate the class imbalance and reduce the interference between the predictions of old and new tasks. Specifically, different weights are adaptively assigned to different input data according to their gradients for the output scores during the training of the new task and updating of the old task models. Then the outputs of old task data on new classification heads are explicitly suppressed and knowledge distillation (KD) [Menon et al., 2021] is utilized for harmonization of the output scores based on the principle of human inductive memory [Williams, 1999; Redondo and Morris, 2011].

The main contributions are summarized as follows:

• We propose a joint input and output coordination mechanism for incremental learning. As far as we are concerned, this is the first work that simultaneously adjusts input data and output layer for incremental learning;
• We design an adaptive input weighting strategy. The samples of different classes are weighted according to their gradients of the output scores. This alleviates the class bias problem both in and between tasks.
• We develop an output coordination strategy, which maintains the outputs of new task data on the old task classification heads before and after training, and suppresses the outputs of old task data on the new task classification heads.

The proposed method is general and flexible, and can be utilized as a plug-and-play tool for existing incremental learning approaches that use memory storage. To demonstrate the effectiveness of our mechanism, we incorporate it into some recent or competitive incremental learning approaches on multiple popular datasets (CIFAR10-LT, CIFAR100-LT, CIFAR100 [Krizhevsky et al., 2009], MiniImagNet [Vinyals et al., 2016], TinyImageNet [Le and Yang, 2015] and Cub200-2011 [Wah et al., 2011]). The results show that we can consistently improve the existing approaches, and the relative improvement is more than 10% sometimes.

2 RelatedWork

2.1 Incremental Learning

Incremental learning [De Lange et al., 2021] has received extensive attention in recent decades. In incremental learning, input data in new tasks are continuously used to extend the knowledge of existing models. This makes incremental learning manifest as a dynamic learning technique. An incremental learning model can be defined as one that meets the following conditions: (1) The model can learn useful knowledge from new task data; (2) The old task data that has been used to train the model does not need to be accessed or has a small amount of access; (3) It has a memory function for the knowledge that has been learned. The current study on incremental learning mainly focuses on domain incremental learning [Mirza et al., 2022; Garg et al., 2022; Mallya et al., 2018], class-incremental learning [Ahn et al., 2021; Rebuffi et al., 2017; Yan et al., 2021; Zhang et al., 2020;

Liu et al., 2021], and small sample incremental learning [Tao et al., 2020; Cheraghian et al., 2021].

There are many works on class-incremental learning (CIL), and most of these works overcome catastrophic forgetting by using knowledge distillation (KD) together with a small amount of old task data accessed. For example, DMC [Zhang et al., 2020] utilizes separate models for the new and old classes and trains the two models by combining double distillation. SPB [Liu et al., 2021] utilizes cosine classifier and reciprocal adaptive weights, and a new method of learning class-independent knowledge and multi-view knowledge is designed to balance the stability-plasticity dilemma of incremental learning.

Although the above approaches can achieve promising performance sometimes, none of them address class bias within tasks, nor adequately address class bias between old and new tasks. Therefore, we propose joint input and output coordination (JIOC) mechanism that enables incremental learning models to alleviate class imbalance and reduce interference between the predictions of old and new tasks.

2.2 Human Inductive Memory

The inductive memory method is a unique ability of human beings. It causes the memorized content to be induced according to different attributes or categories; Subsequently, these contents are memorized by different categories or attributes. As early as 1999, Williams et al. [Williams, 1999] investigated the relationship between memory for input and inductive learning of morphological rules relating to functional categories in a semi-artificial form of Italian. The ability to perform induction appears in the early age of human, while the underlying mechanisms remain unclear. Therefore, Fisher et al. [Fisher and Sloutsky, 2005] demonstrated that category- and similarity-based induction should result in different memory traces and thus different memory accuracy. Hayes et al. [Hayes et al., 2013] examined the development of the relationship between inductive reasoning and visual recognition memory, and demonstrated it through two studies. Inspired by human inductive memory, Geng et al. [Geng et al., 2020] proposed a Dynamic Memory Induction Network (DMIN) to further address the small-sample challenge. These examples of inductive memory inspire us to propose an output distribution coordination mechanism.

3 Method

3.1 Notations and Problem Setup

In CIL, data for new tasks are arriving constantly, which are represented as D = {D1 ,D2 , · · · ,Dt , · · · ,DT }. The data in the t-th new task is Dt = {(xti,j , yt i,j )i=1,2,··· ,m;j=1,2,··· ,nm }, where m is the number of classes, nm is the number of samples for the m-th category, x is the input data, and y is the corresponding data label. The number of samples may vary for different categories in the new task. When learning the tth new task, we assume that there are a small amount of data stored for the old tasks, i.e.,

where i = 1, 2, · · · ,m and j = 1, 2, · · · , nold, nold ≪ n. That is, the number of old data Dtold in the repository is much smaller than that of Dt . In CIL, a feature extractor f(·) (such as ResNet [He et al., 2016]) and a fully connected layer (FCL) together with a softmax classifier is generally adopted, i.e.,

where τ = {1, 2, · · · , t}, Θ is the parameter of the feature extractor, W is the parameter of the classifier, and ˆpτi,j is a vector of output scores. When incremental learning proceeds to the t-th task, all the data in Dt ∪ Dtold = {(xτi,j , yτ i,j ), (τ = 1, 2, · · · , t)} are utilized for training, and the following cross-entropy loss is usually adopted:

where Nold is total number of stored data for old tasks, nnew is the total number of samples for the new task, and ˆ pτi,j,k is the output score at the k-th neuron.

3.2 Overview

According to the above problem setup, it can be seen that when performing incremental learning, only a limited number of samples from the old tasks will be retained. Due to the large number of samples in the new task, incremental learning suffers from the class imbalance issue between the old and new tasks. The class imbalance issue also exists within the new task, but this is ignored by existing CIL approaches [Ahn et al., 2021; Rebuffi et al., 2017; Yan et al., 2021].

Therefore, we propose the joint input and output coordination (JIOC) mechanism, as shown in Figure 2, where we assign different weights to different input data according to their the absolute value of the gradient for output scores. In addition, in order to prevent the mutual interference of output distributions between old and new tasks, we split the softmax layer inspired by the principle of human inductive memory. This is similar to the SSIL [Ahn et al., 2021] approach, but has several significant differences, as shown in Figure 3: 1) for each of the old tasks, we utilize KD to maintain the output distribution of each task. To make the output scores of new task data on the classification heads of old tasks consistent, we also employ KD to enforce the outputs after updating the old task models agree with the scores before the update; 2) to suppress the outputs of old task data on the classification heads of new tasks, their ground-truth target values are directly set to be zero for training.

3.3 Input Coordination

As we know, the class imbalance issue may lead to significant bias in the learned weights of the fully connected layers [Li et al., 2020]. Therefore, ˆ pτi,j,k may deviate greatly from its corresponding true value pτi,j,k , and hence it is necessary to balance the weight of fully connected layers between tasks and within tasks.

Due to the severe bias in the weights of the fully connected layer, we propose to utilize the outputs of fully connected layer's previous layer to adjust the weights. Suppose that ˆqτi,j is the vector of the previous layer that outputs scores ˆpτi,j . The derivative of Lce,t w.r.t. ˆqτi,j (we refer to the supplementary material for the detailed calculation) can be given by:

Then the absolute value of the gradient of the output score for the input data when k = i is:

When the number of data is large for a certain category, the model tends to bias to this category and thus the absolute value of the gradient in Eq. (6) tends to be small in the learning process. To alleviate the bias issue, we propose to regard the absolute value as the weight for the corresponding input sample and add it into the loss during the training. That is, smaller weights will be adaptively assigned to the samples of the category that has more input data, and hence the model would focus more on the category that has fewer samples.

Based on the above analysis, we utilize the absolute values δτ i,j of the gradient to induce a weight for each input data during the training. First of all, we incorporate the absolute value of the gradient of the input data into the traditional cross-entropy loss (Eq. (4)), i.e.,

Then, we can use Eq. (7) to balance the loss of each category. In this way, the category weights of the fully connected layer can be balanced according to the absolute value δi,j of the gradient of each input data. It not only alleviates the category bias between old and new tasks in incremental learning, but also greatly reduces the within-task bias. The main procedure is summarized in Algorithm 1 1 .

3.4 Output Coordination

According to the above analysis, it is necessary to keep the output distribution of the new task data Dt on the old task classification heads consistently before and after updating the old task models 2 . Also, it is necessary to suppress the output scores of the old task data Dtold on the classification heads of Algorithm 1 Main procedure of input coordination.

1 In the entire algorithm pipeline, the outer loop and inner loop iterate T otal and m ∗ nm + Nold batchsize times, respectively. We neglect the time complexity of Eq. (6), Eq. (7), as well as the parameter updates for Θ and W. The overall time complexity of the algorithm pipeline is O(T otal ∗ m ∗ nm + Nold batchsize ).

2 During the updating of the (t - 1) old tasks, there are only m · (t - 1) classification heads. This does not contain the classification heads for the t-th task.

Input: The data of the incremental learning model {Dtold ,Dt }; the feature extractor of the current model is f (·,Θ); the parameter of the current fully-connected layer is W ;

Output: The updated parameters Θ and W;

• 1: for epoch = 1; epoch < Total; epoch++ do
• 2: while batchsize loads {Dtold ,Dt } data do
• 3: (1) δτ i,j ← {ˆqτi,j }, by using Eq. (6);
• 4: (2)LIC,t ←Lce,t and δτ i,j , by using Eq. (7);
• 5: (3) According to the loss value LIC,t obtained in the previous step, the parameters Θ and W of the incremental learning model are updated.
• 6: end while
• 7: Return the updated Θ andW.
• 8: end for

the new task (In Figure 1, this is to keep the blue solid line consistent with the dotted line, and make the green solid line approach to 0).

When the model trains the t-th task, we suppose that the output score of the data Dt ∪ Dtold without going through softmax layer is given by ˆ zτ i,j,k . Before updating the old tasks models and training the t-th task, the output score of the data Dt ∪ Dtold is ˜ zτ i,j,k . By considering the principle of human inductive memory, KD is used to enforce the output consistency of the new task data on the classification heads of each old task before and after updating the corresponding model, i.e.,

where ρϵKL (·) is the distillation loss, and ϵ is a temperature scaling parameter.

The output of the old task data Dtold on the classification head of the new task can be adjusted according to:

where i ∈ {1, · · · ,m(t -1)}.

Although the principle of Eq. (8) is similar to the SSIL [Ahn et al., 2021] approach, the output coordination mechanism proposed in this paper is different from the SSIL approach, as shown in Figure 3. Combining the output coordination loss LIC,t of Eq. (7), the overall loss function LJIOC,t of the method proposed can be obtained, i.e.,

4 Experiment

4.1 Datasets and Evaluation Criteria

Datasets. In this paper, we not only validate the effectiveness of our method on unbalanced CIFAR10-LT and CIFAR100-LT datasets but also conduct corresponding validation on balanced CIFAR100 [Krizhevsky et al., 2009], MiniImageNet [Vinyals et al., 2016], TinyImageNet [Le and Yang, 2015], and Cub-200-2011 [Wah et al., 2011] datasets. The CIFAR10 and CIFAR100 datasets both consist of 50, 000 training images and 10, 000 test images, with 10 and 100 categories respectively. To create unbalanced settings for the balanced datasets, we reduce the number of training samples for some classes. To ensure that our method is applicable to various settings 3 , we consider long-tail imbalances [Cui et al., 2019], and a summarization of the dataset is reported in Table 1. The MiniImageNet dataset was excerpted from the ImageNet [Russakovsky et al., 2015] dataset, and it contains 100 classes, each with 600 images of size 84 × 84.

Typically, the training and test split of this dataset is 80 : 20. The TinyImageNet dataset is also a subset of the ImageNet [Russakovsky et al., 2015] dataset and contains 200 classes, with each class containing 500 training images, 50 validation images, and 50 testing images. The Cub-200-2011 dataset is a bird dataset used for image classification. It covers 200 categories with a total of 11,788 images.

Evaluation Criteria. Following [Shi et al., 2022], the average accuracy is used to measure the performance of the incremental learning algorithm, i.e.,

where Aτ is the accuracy of the τ-th task.

Baseline Protocol. The training sets of CIFAR10-LT is divided into T = 5 tasks, and the number of categories for each task is 2. The number of samples in the memory is fixed to be Nold = {1000, 2000} during the incremental training. In the CIFAR100-LT, CIFAR100, MiniImageNet, TinyImageNet, and Cub-200-2011 datasets, the training tasks are divided into T = 10. The memory sizes of CIFAR100, MiniImageNet, and TinyImageNet dataset are also fixed as Nold = {1000, 2000}, and the memory size is chosen from Nold = {1000, 1500} In addition, for the Cub-200-2011 dataset, the memory storage size fixed as Nold = {500}. The numbers of categories for each task of CIFAR10-LT, CIFAR100, MiniImageNet, TinyImageNet and Cub-200-2011 dataset are 10, 10, 10, 20 and 20, respectively. Regarding the memory storage samples related to our fusion algorithm, we all follow the existing algorithm [Rebuffi et al., 2017; Yan et al., 2021; Wang et al., 2022].

3 Since we use a small amount of old task data, the setup is slightly different from that of [Cui et al., 2019].

Implementation details. Our method and all the compared approaches (BiC [Wu et al., 2019], PODNet [Douillard et al., 2020], COIL [Zhou et al., 2021], SSIL [Ahn et al., 2021], ICARL [Rebuffi et al., 2017], DER [Yan et al., 2021] and FOSTER [Wang et al., 2022]) are implemented using PyCIL [Zhou et al., 2023] and Pytorch [Paszke et al., 2017].

On the experimental dataset, we used ResNet18 [He et al., 2016] and ResNet32 as feature extractors respectively. ResNet32 is just used to further demonstrate that this mechanism can have some effectiveness in other network frameworks. In terms of parameter settings, we align with the original methods on PyCIL [Zhou et al., 2023] to facilitate a fair comparison. Among these, the batch size is set to 128. Additionally, the SGD optimizer is used to gradually update the weights during incremental learning model training. The learning rate is initially set to be 0.1 and gradually decays. We run the training on two NVIDIA 3090RTX GPUs.

4.2 Results and Analysis

We incorporates the proposed JIOC strategy into existing class-incremental learning algorithms (ICARL [Rebuffi et al., 2017], DER [Yan et al., 2021], and FOSTER [Wang et al., 2022]). The experimental results on different datasets are shown in Table 2 and Table 3.

Results on CIFAR10-LT and CIFAR100-LT. From the overall performance analysis of the Table 2, it can be seen that the ICARL, DER, and FOSTER algorithms on our created imbalanced datasets have been significantly improved. The relative improvements of ICARL JIOC are 1.70%, 1.24%, 20.90%, 18.29%, 10.11% and 6.84% compared with the original ICARL algorithm. For the DER algorithm, our DER JIOC improves it by 1.03%, 1.80%, 1.88%, 0.55%, 3.55% and 3.07%. In regard to the FOSTER algorithm, the relative improvements are the significant 3.51%, 3.66%, 3.02%, 10.10%, 2.20%, and 3.37% respectively. Compared with all the counterparts, the best performance is usually achieved by the proposed FOSTER JIOC method. This not only demonstrates the effectiveness of our method on imbalanced datasets, but also further confirms its ability to alleviate catastrophic forgetting in other network frameworks.

Results on MiniImageNet, TinyImageNet, Cub-200-2011 and CIFAR100. We can observe from Table 3 that the mechanism proposed in this paper also has significant improvements on the MiniImageNet, TinyImageNet, Cub-2002011 and CIFAR100 datasets. For example, our FOSTER JIOC outperforms the original FOSTER algorithm by 4.28%, 6.24%, 4.07%, 2.06%, 38.60%, 6.91%, 14.90%, 1.67% and 2.82%, respectively. The best performance is also achieved by the proposed DER JIOC and ICARL JIOC, which are comparable. This further demonstrates that the proposed method not only alleviates catastrophic forgetting in class-imbalanced datasets but also has a forgetting-mitigation effect on normal data.

4.3 Ablation Studies

In this section, we first examine the effectiveness of the input and output coordination strategies separately, and then analyze how the output coordination strategy shows a more significant effect in mitigating forgetting compared to the SSIL.

Figure 4: A Compare the experimental results of SSIL with those of SSIL OC (SSIL OC is formed by integrating the proposed OC strategy into the SSIL algorithm. ∗ means our impementation).

Study on the effectiveness of Input and Output Coordina-

tion. To validate the effectiveness of input and output coordination strategies, we exclusively employed ResNet18 as the feature extractor on the CIFAR100-LT dataset, , as shown in Tables 4. Furthermore, on the CIFAR100 dataset, we conducted experiments separately using ResNet18 and ResNet32 as feature extractors, as shown in Table 5 and Table 6 (we refer to the supplementary material for Table 5 and Table 6).

(1) The results are reported in Table 4, where we can see that the average accuracy of the ICARL algorithm with the proposed input coordination is 32.34 on the old tasks, 76.22 on the new task, and 43.16 overall. Compared with the original ICARL approach, the improvements are 27.22%, 3.50%, and 12.19%, respectively. The results from the old task, new task, and overall task in Table 5 and Table 6 also illustrate the competitiveness of the input coordination strategy. This demonstrates that the input coordination strategy can alleviate class imbalance in incremental learning. Besides, the ICARL algorithm only uses KD to maintain the output distribution on the old task classification heads. It does not take into account the human inductive memory mechanism for coordinating output distribution across different tasks. In Table 4, the ICARL algorithm achieved average performances of 34.45, 74.83, and 45.13 on the old tasks, new tasks, and overall, respectively. Our proposed output coordination mechanisms improved these performances by 35.52%, 1.62%, and 17.31%, respectively. It can be concluded that the input and output coordination strategy proposed in this paper yields significant improvements, whether applied to the CIAFR100 dataset, the CIAFR100-LT dataset, or different network architectures with varying depths.

(2) In the CIART100-LT dataset, the input coordination strategy demonstrates notable enhancements in the outcomes for old tasks (27.22%), new tasks (3.50%), and overall task (12.19%) performance when compared to the original ICARL algorithm, as shown in Table 4. Similarly, In the CIART100 dataset, the input coordination strategy further improves the original ICARL algorithm by 42.05%, 3.61%, and 19.51% on the old tasks, new tasks, and overall tasks, as shown in Table 5. According to the description and corresponding improvement effects of CIART100LT and CIART100, it can be seen that the input coordination strategy has a good regulating effect on the imbalance of old and new categories.

Experiment Comparison between Output Coordination Strategy and the SSIL. To quantitatively analyze the differences between the output coordination strategy and the SSIL, we conducted corresponding experimental results based on different feature extractors and datasets, including class-imbalanced and balanced datasets, as shown in Figure 4. From the results in Figure 4, it is evident that using the output distribution coordination strategy leads to significant improvements in each stage task on class-imbalanced datasets and in deep feature networks (ResNet32). This also indicates that the output distribution coordination strategy enables new task data to maintain consistent output distributions on the old task classification head and suppresses old task data on the new task classification head during the incremental learning process. This avoidance of interference between new and old tasks is achieved. However, SSIL does not avoid interference from old task outputs, which results in its performance being inferior to the output distribution coordination strategy.

5 Conclusion

Although the existing approaches address the class bias issue in class-incremental learning (CIL) to a certain extent by scaling and dividing the softmax layer, they all ignore the bias within the task. In addition, the mutual interference between old and new tasks has not been well resolved. Therefore, we propose a joint input and output coordination (JIOC) mechanism to enable incremental learning models to simultaneously reduce the interference between predictions for these tasks and alleviate the class imbalance issue between and within tasks. From the extensive experiments on multiple popular datasets, we observe significant improvements when incorporating the proposed mechanism into the existing CIL approaches that utilize memory storage. In the future, we intend to design more sophisticated strategies to reweight the inputs, and develop a general end-to-end framework for CIL.

Acknowledgements

This work is supported in part by the National Natural Science Foundation of China (Grant No. U23A20318), the Special Fund of Hubei Luojia Laboratory under Grant 220100014, the Fundamental Research Funds for the Central Universities (No. 2042024kf0039), the National Research Foundation Singapore and DSO National Laboratories under the AI Singapore Programme (AISG Award No: AISG2GC-2023-006), and the CCF-Zhipu AI Large Model Fund OF 202224.