Delve into Base-Novel Confusion: Redundancy Exploration for Few-Shot Class-Incremental Learning

1 Introduction

Deep Neural Networks have achieved remarkable success in many domains [Liu et al., 2020; Li et al., 2022; Daihong et al., 2022]. However, real-life scenarios often involve streaming data, posing challenges to obtaining sufficient data for continuous learning of new knowledge, especially with limited samples. To address this, Few-Shot Class-Incremental Learning (FSCIL) [Tao et al., 2020] was introduced. FSCIL entails learning base classes with an adequate number of samples in the base session, followed by learning novel classes using only a few samples in incremental sessions (Fig.1a).

∗ denotes the corresponding author.

The primary challenges in Few-Shot Class-Incremental Learning (FSCIL) center around the catastrophic forgetting of learned knowledge and the overfitting [Hu et al., 2019] to novel classes due to limited data. To address these challenges, current approaches typically initiate training on base classes and then freeze the backbone while learning novel classes [Snell et al., 2017]. However, although the fixed-backbone method maintains the model's performance on base classes, it impedes effective generalization to novel classes.

To investigate this issue, we briefly plot the performance of classifying base-class, novel-class, and all-class samples into all encountered classes, denoted as BA-Acc, NA-Acc, and AA-Acc, separately in Fig. 1b(left). We can see the yellow dashed line notably surpasses the red dashed line, signifying a considerable decrease in performance on novel-class samples compared to base-class ones. Then, we divide the novel-class performance into classifying novel-class samples against only novel classes and against all (i.e., base + novel) classes, denoted as NN-Acc and NA-Acc, respectively in Fig. 1b(right). We can observe the prediction accuracy significantly drops by 26.23% when base classes are introduced, which is a major cause of the low performance on novel-class samples. This phenomenon indicates the model tends to misclassify novelclass samples into base classes, and we call it the confusion between base and novel classes. Although this phenomenon widely exists in current FSCIL methods, only a few works [Wang et al., 2023] attempted to explore its cause.

In this study, we delve into the cause of the confusion between base and novel classes, focusing on the redundancy in the feature space and the pixel space. We interpret the confusion as the collision between the novel-class region and the base-class region in the feature space, which is caused by the redundant area of base-class regions (Fig. 1c). Through qualitative and quantitative experiments, we find such redundancy reflects shortcuts of the model training, which can be decoupled to alleviate the collision between base and novel classes.

Based on the above analysis, to alleviate the confusion between base and novel classes, we propose an approach named Redundancy Decoupling and Integration (RDI) to address the redundancy within the base-class feature space. We first decouple the redundancy in the feature space by finding the label-irrelevant regions, and then map these regions to the pixel space for image masking, which would shrink the intrabase-class space. Then, we integrate these label-irrelevant regions by classifying masked images into a dummy class, which further compresses the base-class feature space and reserves a buffer space for novel classes. Consequently, the reserved space would reduce the collision between base and novel-class regions in the feature space, therefore alleviating the confusion between base and novel classes.

The contributions of this work are summarized as follows:

• Our experimental analysis reveals that the model's generalization on novel classes is majorly limited by the confusion between base and novel classes, predominantly attributed to label-irrelevant redundancies in the base-class feature space.
• Based on the analysis, we propose an innovative approach to mitigate the confusion between novel and base
• classes, which firstly decouples the label-irrelevant redundancies in both the feature space and the pixel space, and then integrates these redundancies to reserve a butter space for novel classes.
• Experimental results not only showcase the efficacy of our approach in enhancing generalization capacity on novel classes but also demonstrate its achievement of state-of-the-art performance on benchmark datasets, including CIFAR-100, CUB200, and miniImageNet.

2 Interpretation of Base-Novel Confusion

In this section, we initially outline the Few-Shot ClassIncremental Learning (FSCIL) task and the baseline method. Then, we conduct experiments to interpret the cause of the confusion between base and novel classes.

2.1 Problem Set-up

Few-shot Class Incremental Learning (FSCIL) is a learning paradigm that involves mastering base classes with ample examples before tackling novel classes with limited samples. Typically, FSCIL employs a sequence of T training sessions denoted as {D0 train , . . . ,DT-1 train }, each associated with a distinct label space {C 0 , . . . , C T-1 }, where there is no overlapping between them. During each session, only the sessionspecific training set Dt train is available, comprising examples {(xi, yi)}nt i=1 with yi ∈ C t , and nt denotes the number of examples. The testing set Dt test , however, includes samples from all previously encountered and current classes, denoted as Dt test = {(xi, yi)}mt i=1 , where yi ∈ ⋃ti=0 C i . The initial session (t = 0) involves a training set with ample examples per class. The model is built as f(x) =W T ϕ(x), where ϕ(x) represents feature extraction and W denotes the classifier for base class categorization. Subsequent incremental sessions (0 < t < T) consisting of limited samples are organized as N-wayK-shot tasks. In these sessions, themodel learns novel classes while maintaining performance on base classes.

2.2 Baseline Description

The baseline model follows a structured approach for incremental learning. In the base session, the model undergoes training on D0 train utilizing the cross-entropy loss:

Moving to the incremental session, each base class's classifier is replaced with the class's average embedding, known as the prototype. The prototype calculation is denoted as

where I represents the indicator function, ϕ(·) is the feature extractor, and n0c denotes the count of examples in class c. During the incremental session, the classifier for base classes remains fixed, while for novel classes, the classifier can be refined through fine-tuning or by employing the average embedding. During evaluation in session T, an input x is predicted using its output feature ϕ(x) and the classifier W, by:

2.3 Interpretation of Confusion Between Novel and Base Classes

In Fig. 1b, we conclude that the major limitation of the novelclass generalization: the model tends to classify novel-class test samples into base classes, termed "Base-Novel Confusion". In this section, we delve into this confusion from the redundancy in the feature space.

Confusion represented as collisions in the feature space

To explore the confusion between base and novel classes, we first plot the feature distribution for both base and novel classes in Fig. 2. Initially, we randomly select sixty samples from each of six randomly sampled base classes in CIFAR100 and visualize the distribution of feature points (circles) and decision boundaries using t-SNE [Van der Maaten and Hinton, 2008], as shown in Fig. 2a. To simulate the baseline model's learning process of novel classes, we then visualize the feature distribution of six newly introduced (randomly sampled) novel classes together with base classes, as depicted in Fig. 2b. Notably, these novel classes, represented as triangles, fall within the space allocated for base classes, with some even closely approaching the feature points (circles) representing base classes. Although we can see clear clusters even for novel classes, novel samples are located within the decision boundary for each base class, leading to collisions in feature spaces and confusing the model to classify novel-class samples into base classes. In essence, the collision exists because the model learns to occupy the entire feature space during the base-class training. Therefore, a natural question arises: do base classes really need to occupy so much space to make the model discriminative on them?

Redundancies caused by the ordinary classification loss

To answer the above question, we look back into the classification loss utilized to train the baseline model. The ordinary cross-entropy loss in Eq.1 can be rewritten as illustrated in Fig.3. To minimize this loss, the model needs to increase eτcos(xi,wc) and decrease ∑m̸=c eτcos(xi,wm) . In other words, the model learns to (1) push features in the class c close to class c's centroid while (2) pulling away these features from other classes. However, these two targets may not be aligned, because features away from other classes may not be close to the class c, which could lead to redundancies in the feature space. For example, in Fig.3, although red hexagons (could be features of an instance or an image patch) are pulled away from other classes (the gray circle) by decreasing ∑m̸=c eτcos(xi,wm) , it may not be close to class c's centroid (the yellow circle). This means the features represented by red hexagons, although may help the classification of the class c, may not really capture the central characteristics of the class c, which could be understood as a kind of shortcut that overfits the class c.

To validate our hypothesis, we compute the similarity of each patch f(a, b) ∈ Rd of the feature map F ∈ Rh×w×d using a pre-trained model from its ground truth class, and categorize the patches into central and redundant patches by thresholding the similarity. The central patch is close to its ground truth class, while the redundant patch is far from that class. Subsequently, we utilize eτcos(xi,wc) and ∑m̸=c eτcos(xi,ym) to compute the average similarity between patches and classes, as depicted in Tab.1. Redundant patches are not only distant from both the ground truth class but also far from other classes. This quantitatively indicates that the model tends to learn label-irrelevant but discriminatory redundant patches to minimize ∑m̸=c eτcos(xi,ym) , thereby reducing the overall loss.

Redundancies as label-irrelevant regions in the heatmap

To ascertain whether redundancies are genuinely labelirrelevant, we examine them in the pixel space. Initially, we scrutinize the feature map of base classes under the pretrained baseline model, visualizing the output from the last convolutional layer as a heat map using ProtoNet [Snell et al., 2017] (Fig.4, 'Feature Map'). Although the pre-trained model can identify semantic objects with higher activations, not all activated regions necessarily align with the label. For instance, in the visualization (third row of Fig.4), activated regions corresponding to both a fish and a part of stones were identified, despite the image being labeled as a fish. The parts of the stones are considered irrelevant to the label, while the fish is relevant. We subsequently isolate central and redundant patches from the original feature map by their similarities with the ground truth class following Tab.1. We can see the redundant patches correspond to label-irrelevant but discriminatory regions. Based on the above experiments, we conclude that the ordinary cross-entropy loss leads the model to focus not only on label-relevant features (central patches) but also on label-irrelevant redundancies for discrimination.

Since redundant features also occupy spaces within the decision boundary of each base class, it could lead to the collision between base and novel classes in the feature space. This inspires us to revisit the feature space to explore the relationship between redundancies and collision.

Redundancies lead to collisions in the feature space

We first decouple the Activated Label-Irrelevant (ALI) features and Activated Label-Relevant (ALR) features from the original feature map, where ALI corresponds to the redundant patches and ALR corresponds to the central patches. Then, we visualize the feature distribution and decision boundaries of ALI features and ALR features from the six previously selected classes, as shown in Fig.5. We can see that ALI features (gray fork) are distributed around ALR features, and ALR features exhibit a more compact distribution compared to the original feature distribution. In other words, these ALI features, acting as redundancies in feature space, expand the base-class space. Without these redundancies, the base-class space would be more compact.

Therefore, we hypothesize that these redundancies in the base-class feature space are the primary cause of the model's confusion between base and novel classes. To substantiate this, we integrate all ALI features as a dummy class and visualize the decision boundaries of base classes (using ALR features) and the dummy class (grey region). Then, we plot the feature distribution of novel classes (Fig.5 right). We observe that novel classes are concentrated in the grey region where redundant ALI features are distributed, which confirms our speculation that the redundancies lead to the collision.

Conclusion and Discussion

We conclude that the confusion between novel base classes can be interpreted as the collisions in the feature space. Based on the analysis of the regular classification loss, we find that it will encourage the model to learn label-irrelevant but discriminatory patches, which could be redundancies. To investigate the relationship between redundancies and collision in feature space, we visualize the redundancies in feature space and find that it is the redundancies in base-class space that mainly cause the collision between base and novel classes.

3 ProposedMethod

While the baseline approach addresses catastrophic forgetting to some extent by employing a fixed feature extractor, it compromises the model's capacity to generalize over novel classes. This raises a natural question:

How can we improve novel-classes generalization while maintaining base-classes performance?

Based on the above analysis, we pinpoint the key factors leading to inadequate generalization of existing methods in novel classes-the confusion between novel and base classes. This confusion predominantly stems from redundancy within the base-class feature space. In response, we introduce our method, Redundancy Decoupling and Integration (RDI), for Few-Shot Class Incremental Learning (FSCIL). RDI comprises two modules. The first module decouples redundancy-Activated Label-Irrelevant (ALI) features-from the base-class feature space, utilizing Activated Label-Relevant (ALR) features for base-class classification. In the second module, we integrate redundancies as a dummy class to contribute to ALR-based recognition. An overview of our method is illustrated in Fig.6.

3.1 Shrink Intra-Base-Class Space by Decoupling Redundancy from Base-Class Space

For a more precise decoupling of the base-class space, we propose a two-step approach. Firstly, we pre-train the model using the cross-entropy loss, obtaining a classifier Wbase = [w1, . . . ,wn] ∈ Rd×n and a feature extractor Φ(·). Here, n is the number of base classes. Next, we initiate the decoupling process by segmenting the feature map output from the pre-trained model into ALR (Activated LabelRelevant) and ALI (Activated Label-Irrelevant) regions.

Based on the analysis in Section 2, we aim to decouple redundancies by computing the distance between each patch f(a, b) ∈ Rd of the feature map F ∈ Rh×w×d and its ground truth class and generating an ALR mask M ALR ∈ Rh×w :

where σ(·) is an activation function with a threshold ϱ as a hyperparameter, yp is the predicted label, and M ALR ∈ Rh×w is a 0-1 mask, Considering the possibility of misclassified samples and incorrectly activated feature maps after the pretraining stage, we suggest using the predicted label instead of the real label for all samples to ensure the validity of the feature map. The predicted label yp can be obtained by:

Subsequently, the ALR feature f ALR is obtained as:

We label the ALR feature with its real label, i.e., yALR = y. Consequently, the cross-entropy loss of using the ALR feature for classification can be represented as:

The classification loss using the original feature is denoted as Lbase. The total loss is then expressed as a weighted sum of the original loss and the ALR loss: L = Lbase + λLALR base . Through LALR base , the model guides central patches to approach the ground truth class and distance themselves from other classes. This strategy mitigates shortcuts in the base-class training, effectively decoupling redundancies from the baseclass space.

3.2 Reduce Inter-Base-Class Space by Integrating Redundancies as a Dummy Class

In contrast to previous methods, our innovative approach integrates redundancies as a dummy class to contribute to ALRbased recognition. This process compresses intra-base-class space and reserves a space for novel classes, effectively mitigating confusion between novel and base classes.

Building on the insights from Section 2, we categorize patches in the feature map into two groups: central patches (ALR) and redundant patches (ALI). Subsequently, we obtain the ALI mask from the ALR mask, denoted as M ALI (a, b) = 1-M ALR(a, b). Utilizing this ALI mask, we derive the ALI feature f ALI ∈ Rd .

Afterward, the ALI feature is labeled as n + 1, indicated as yALI = n + 1. To set aside space for novel classes, we adjust the classifier by incorporating the dummy class, yielding Wdummy = [Wbase,wn+1] ∈ Rd×(n+1) . Subsequent to this modification, the loss for the ALR feature can be updated:

Then, we can calculate the loss for the ALI feature:

The total is expressed as:

In this equation, the first term ensures base-class performance. The second term facilitates the separation of redundancies from the base-class space, leading to a reduction in intra-base-class space. The third term introduces a dummy class to compress inter-base-class space, thereby creating room for novel classes. The combination of the last two terms alleviates the model's confusion between novel and base classes. λ and β are hyper-parameres.

Before the incremental session, we extract base-class average embedding (prototypes) as classifiers for base classes using Eq.2. During the incremental session, we maintain the feature extractor Φ(·) fixed and employ it to extract features from novel-class examples for inference.

4 Experiment

In this section, we introduce benchmark datasets and implementation details, validate our method's effectiveness through qualitative and quantitative analyses, perform ablation experiments to analyze component contributions, and compare our method's performance with state-of-the-art approaches on benchmark datasets.

4.1 Implementation Details

Datasets & Split: Following [Yang et al., 2023], Our method is evaluated on CIFAR-100 [Krizhevsky et al., 2009], miniImageNet [Russakovsky et al., 2015], and CUB200 [Wah et al., 2011]. CIFAR-100 has 60,000 images (64×64) across 100 classes, miniImageNet has 60,000 images (84×84) representing 100 classes from ImageNet [Deng et al., 2009], and CUB200 comprises 11,788 fine-grained images (224×224). Both miniImageNet and CIFAR-100 have 60 base classes initially and 40 novel classes distributed across 8 incremental sessions as 5-way 5-shot tasks. For CUB200, the initial session has 100 base classes, and 100 novel classes are distributed over 10 incremental sessions as 10-way 5-shot tasks. Training Details: Our training protocol follows [Yang et al., 2023], utilizing the ResNet-12 for CIFAR-100 and miniImageNet, and pretrained ResNet-18 for CUB200. Batch sizes of 128 and 100 are employed for the base session and incremental sessions, respectively.

4.2 Qualitative and Quantitative Analysis

This section aims to substantiate key claims regarding our method's efficacy through a series of experiments:

• Decoupling Redundancy
• Compressing Base-Class Space
• Alleviating the Collision in Feature Space
• Mitigating Base-Novel Confusion

Decoupling Redundancy from Base-class Space

We randomly choose images from the miniImageNet dataset to visualize the feature map. The heat maps in Fig.7 show changes in the feature map. The top row represents the baseline, and the bottom row showcases our method. Significantly, our method directs the model's attention to the labelrelevant region, effectively decoupling label-irrelevant redundancies from the base-class space.

Compressing Base-class Space

To quantitatively showcase our method's efficacy in reducing base-class space, we measure intra-base-class and inter-baseclass cosine distances on CIFAR-100 (Fig.8). Our approach (solid red line) adeptly diminishes both intra-base-class and inter-base-class space, consistently placing cumulative distribution functions to the left of the baseline (gray dashed line).

Alleviating Collisions in Feature Space

To visually highlight our method's efficacy, we randomly selected twelve base and ten novel classes from CIFAR-100, each with twenty samples. Visualizing the feature distribution after inserting novel classes using both baseline and our methods (Fig.9), the left side shows novel classes (green squares) are mixed with base classes (orange dots) in the baseline. In contrast, on the right side, with our method, novel classes are located within the dummy class (* marked in gray) region and are separated with base classes, showcasing our method's ability to alleviate feature space collisions.

Mitigating the Model's Confusion

We quantitatively demonstrate our method's effectiveness in reducing the model's confusion between novel and base classes on miniImageNet. Using NN-acc and NA-Acc metrics in Section 2, the larger difference between them indicates increased confusion between base and novel classes. This evaluation, depicted in Fig. 10, spans both the baseline and our model. Results unequivocally emphasize our approach's success in mitigating the model's confusion.

4.3 Ablation Study

We conducted ablation studies on three benchmark datasets, summarizing key findings in Table 3. Results highlight our approach's pivotal role in enhancing final session accuracy and improving novel-class generalization. Consistently across benchmark datasets, introducing both LALR dummy and LALI dummy enhances the model's ability to learn novel classes. The combination of two loss terms proves highly effective in reducing confusion, leading to substantial improvements across all classes. These ablation experiments provide insights into contributions of each component, emphasizing the overall effectiveness of our proposed method.

4.4 Comparison with the State-of-the-Art Methods

We comprehensively compare our proposed method with existing approaches across three FSCIL benchmarks. Performance curves for miniImageNet and CUB200 are shown in Figure 11, and detailed performance values for CIFAR100 are provided in Table 2. Our method consistently outperforms previous approaches in average accuracy across the benchmarks, including a notable improvement over NCFSCIL [Yang et al., 2023], a recent top-performing method. Specifically, our method achieves marked improvements of 1.37%, 0.94%, and 0.76% in the last session on CIFAR100, miniImageNet, and CUB200, respectively. Moreover, we substantiate our superiority by attaining an average accuracy enhancement of 1.74% on miniImageNet.

5 RelatedWork

Few-Shot Class-Incremental Learning (FSCIL) [2019; 2017; 2021] seamlessly combines Class Incremental Learning (CIL) [2018; 2019; 2022] and Few-Shot Learning (FSL) [2019; 2020; 2018]. FSCIL prioritizes mastering base classes with abundant examples initially and then adeptly addressing novel classes with few instances in each incremental session.

FACT [Zhou et al., 2022a] addresses feature distribution discrepancies between novel and base classes by introducing forward compatibility. It allocates space for novel classes through virtual instances from instance fusion. However, concerns may arise regarding the accuracy of synthesized samples in representing real-world instances.

N-FSCIL [Yang et al., 2023], the leading approach, preassigns classifier prototypes for the entire label spectrum to rectify feature-classifier misalignment for old classes. It reserves space for novel classes by referencing class prototypes to accommodate both base and novel categories. However, this method requires prior knowledge of novel class counts, limiting its practical applicability.

TEEN [Wang et al., 2023], closely related to our work, addresses the lower performance of novel classes in FSCIL, highlighting the model's tendency to misclassify them into base classes. It introduces a training-free prototype calibration method based on semantic similarity between base and novel classes. In contrast, our study thoroughly analyzes the root cause of low novel class performance from a feature distribution perspective, proposing a more effective and widely applicable solution to this issue.

6 Conclusion

In this paper, we address the challenge of confusion between novel and base classes within the Few-Shot ClassIncremental Learning framework, particularly when using a fixed backbone trained on base classes. Through a thorough analysis of the root cause behind the model's confusion in existing methods - redundancy within the feature space of the base class leading to collisions between novel and base classes in feature space - we introduce the Redundancy Decoupling and Integration (RDI) method. Extensive evaluations on benchmark datasets affirm RDI's effectiveness, significantly enhancing novel-class generalization and achieving state-of-the-art performance across all classes.

Acknowledgments

This work is supported by National Natural Science Foundation of China under grants 62206102, 62376103, 62302184, 62377009, U1936108 and Science and Technology Support Program of Hubei Province under grant 2022BAA046.