CD^2: Constrained Dataset Distillation for Few-Shot Class-Incremental Learning

1 Introduction

In real-world applications such as robotics, healthcare, and remote sensing, dealing with sequential data streams and fewshot data arise frequently and often simultaneously [Sun et al., 2024; Li et al., 2024]. While deep neural models demonstrate impressive performance in static and data-abundant settings [Shen et al., 2024; Wei et al., 2024], it is still significantly challenging for them to learn new concepts continually with a very restricted quantity of labeled samples. In this situation, few-shot class-incremental learning (FSCIL) has emerged as a promising solution and has attracted much research attention [Tao et al., 2020]. In FSCIL, the model acquires extensive knowledge with sufficient labeled samples in the base session, and continually learns new knowledge from a few samples while retaining previously learned concepts in subsequent incremental sessions. Within FSCIL, the primary challenge lies in preventing catastrophic forgetting [Kang et al., 2023b; Babakniya et al., 2023], a phenomenon where models tend to overwrite previously acquired concepts when confronted with new data due to the unavailability of data from previous sessions, especially learning with limited data.

Recently, mainstream methods tend to freeze the backbone and adopt an external memory as an auxiliary tool during incremental sessions to preserve previously learned knowledge [Kukleva et al., 2021; Yang et al., 2023a; Ahmed et al., 2024]. The memory stores a small amount of data from previous sessions by replaying real samples [Kukleva et al., 2021; Zhu et al., 2022], generating samples [Liu et al., 2022; Agarwal et al., 2022] or calculating pseudo features [Hersche et al., 2022; Ji et al., 2023; Yang et al., 2023b; Liu et al., 2023a], enabling quick recall of old knowledge in subsequent sessions. Compared to only freezing the backbone, using a memory significantly boosts performance in incremental sessions, which illustrates that the memory enhances the ability of the model to revisit old knowledge and improves its information discrimination capacity. By leveraging prior experience effectively, using the memory offers a robust solution to catastrophic forgetting in an FSCIL setting.

However, we find that existing methods may still learn redundant features and cannot preserve the previous distribution in incremental sessions. As shown by red lines in Figure 1 (a), when building the memory, previous methods dilute critical class-related knowledge due to mixing discriminative and redundant knowledge, which hinders the recall of old knowledge in subsequent sessions. Recently, Dataset Distillation (DD) is proposed to obtain a few highly condensed and informative samples from a dataset [Zhao et al., 2021; Zhang et al., 2024], which aligns with our goal of learning essential class-related knowledge. Therefore, we introduce DD for building a memory to obtain critical knowledge and synthesize highly informative samples as shown in Figure 1 (b). Further, there is a distribution gap between data in the memory and the new dataset in incremental sessions [Yang et al., 2024]. Treating them equally prompts the model to focus on distribution differences, leading to the covariate shift of old classes as illustrated by green arrows in Figure 1 (a) and (b). Facing the gap, we incorporate the core idea of knowledge distillation (KD) [Hinton et al., 2015] during training. KD transfers knowledge from a teacher model (usually a welltrained, high-performing model) to a student model. And we constrain the change of distribution across continual sessions meticulously by putting the previously trained model as the teacher for the current model, which can mitigate the covariate shift and catastrophic forgetting. Combined with the core idea of distillation, our method can extract and curate critical knowledge as shown in Figure 1 (c).

Specifically, we propose a Constrained Dataset Distillation framework CD2 to support FSCIL, which contains a more refined memory strategy and a more effective retention method. Firstly, we analyze existing popular memory strategies, including sample replay (reusing past training samples during incremental learning) and prototype computing (calculating the mean of intermediate features for each class) Then, to retain more essential knowledge, we propose a dataset distillation module (DDM) inspired by DD [Zhao et al., 2021]. DDM synthesizes samples for each class to condense critical class-related knowledge, which retains knowledge stably and prevents performance degradation. Further, we employ a distillation constraint module (DCM) to transfer knowledge in a stable and flexible manner. The DCM promotes features and structures of old classes in the memory to align seamlessly between previous and current sessions, thereby enabling the model to constrain the distribution gap and utilize the memory more precisely and efficiently.

Our main contributions can be summarized as follows:

• We propose a framework termed Constrained Dataset Distillation (CD2 ) to facilitate the FSCIL task. To the best of our knowledge, we are the first to introduce dataset distillation to FSCIL, which could effectively captures critical knowledge.
• We propose a dataset distillation module (DDM) to build a memory with more critical knowledge and a distillation constraint module (DCM) to reduce the covariate shift and maintain the stability of knowledge transmission, ensuring effective leveraging of dataset distillation.
• Extensive and comprehensive experiments on three benchmark datasets are conducted, where all results demonstrate the effectiveness of our framework.

2 RelatedWorks

2.1 Few-Shot Class-Incremental Learning

Few-shot class-incremental learning (FSCIL) enables the model to incrementally learn new knowledge while effectively retaining old knowledge with a few new samples [Yang et al., 2023a; Ahmed et al., 2024]. Some early methods update the whole model using the topology [Tao et al., 2020] and knowledge distillation [Dong et al., 2021] in incremental sessions, which suffer from catastrophic forgetting and overfitting due to finetuning a large number of parameters.

Based on this, mainstream methods finetune part of the parameters with the assistance of a memory during incremental learning, which can reduce overfitting while retaining certain old knowledge. Some methods select random [Kukleva et al., 2021] or adaptive [Zhu et al., 2022] samples from old classes as the memory. However, replaying a few samples cannot obtain a representative and high generalization representation, which is also prone to exposing privacy. Some generative methods effectively discourage forgetting and protect privacy by generating synthetic data with a GAN-like model [Liu et al., 2022; Agarwal et al., 2022]. However, there are still deviations between generated and real samples. And some methods store pseudo-features to mitigate catastrophic forgetting under privacy protection, calculating vectors from real features [Hersche et al., 2022], introducing virtual prototypes [Zhou et al., 2022a; Yang et al., 2023b; Liu et al., 2023a], and compressing knowledge into a small number of quantized reference vectors [Chen and Lee, 2021; Ji et al., 2023], which can retain knowledge quickly. However, these methods compress knowledge crudely and treat them equally with real data, affecting the model performance. Based on this, we explore a more refined memory strategy and a higher-utilization method to retain more old knowledge.

2.2 Dataset Distillation

Dataset Distillation (DD) aims to learning compressed data to enhance learning efficiency [Wang et al., 2018; Cazenavette et al., 2022], which is widely used in federated learning [Liu et al., 2023b] and neural architecture search [Such et al., 2020]. DD retains the critical information needed to train a model effectively, which can reduce the amount of data without significantly reducing the performance of the model. Some optimization methods incorporate meta-learning into the surrogate image updating [Zhou et al., 2022c]. And other methods optimize the synthetic images by matching the training gradients [Kim et al., 2022], feature distribution [Zhao et al., 2023a], or training trajectories [Du et al., 2023]. Different from traditional data compression, DD preserves adequate task-useful information so that the model trained on it can generalize well to other unseen data in subsequent tasks.

3 Preliminaries

3.1 Formulation of FSCIL

Few-shot class-incremental learning (FSCIL) [Tao et al., 2020] consists of a base session and multiple incremental sessions, which trains a model incrementally on a time sequence of training sets D = {D(t) }Tt=0 . The base (0-th) session dataset D(0) consists of a large label space C (0) with sufficient samples per class. And the t-th incremental session dataset D(t) (t > 0) has limited data with its label space C (t) (t > 0), which has N classes and K training examples per class (N-way K-shot). Any two label spaces are disjoint, meaning C (i) ∩ C (j) = ∅ for all i, j(i ̸= j). Among them, D(t) = {(xi, yi)}|D(t) | i=0 , where xi is an example (e.g, image), yi ∈ C (t) denotes its target. While evaluating the performance in session t, the model is assessed on the current and all previous validation datasets {T (i) }ti=0 .

During training, we train a model ϕ(x) in all sessions, which consists of a backbone ϕb(xi) with the intermediate feature fi = ϕb(xi), and a classifier ϕc(fi) with the final output vi = ϕc(fi). The classifier contains an MLP block to project intermediate features and a fully connected layer for classification. We train the model on dataset D(0) in the base session, then finetune the classifier with dataset D(t) (t > 0) and an extra memoryM(t) in t-th incremental session. Thus, the FSCIL task is formulated into two steps as follows:

where Eq. (1a) and Eq. (1b) minimizes the empirical risk EB in the base session (t = 0) and EI in incremental sessions (t > 0). In incremental sessions, the model only has access to the dataset of the current session and not the training set of previous sessions, causing catastrophic forgetting. In this context, the memory plays an important role in mitigating this issue. Previous methods always build and use memory crudely, which cannot properly preserve critical old knowledge and further affects the distinguishing ability of the model. Therefore, we are devoted to exploring a more refined memory strategy and a higher-utilization method, which preserves more old knowledge during incremental learning and improves the continuous learning ability of the model.

3.2 Explore the Previous Memory Strategies

Sample replay. To achieve high performance in the joint space of old and new classes, the sample replay technique is used in FSCIL. Some methods store some real samples from previous sessions in a memory. In t-th session (t > 0), few samples from dataset D(t-1) are sent to the memory M(t) , which alongside new added samples to finetune the model as

These methods re-expose previous samples to the model during training, which is like a reminder to help the model retain previous knowledge when it learns new knowledge. Prototype computing. Although using sample replay can recall old knowledge, it may develop biases for the model due to limited knowledge and is prone to exposing privacy, which restricts the ability to mitigate catastrophic forgetting. Inspired by ProtoNet [Snell et al., 2017], some methods store prototypes as a memory for subsequent tasks as:

Each prototype is the mean of intermediate features, condensing the common pattern of each class within an embedding vector. The prototypes remain constant across incremental sessions to alleviate catastrophic forgetting.

Analyse. Figure 2 shows the performance of building a memory by sample replay and prototype computing. When using sample replay, the performance gradually increases with the number of samples, which indicates that more samples cover more knowledge. However, the selection problem of samples, resource consumption, and privacy issues cannot be ignored. When outlier samples are chosen for replay, it affects the model to generalize and retain old knowledge properly. Prototype computing preserves privacy and integrates knowledge efficiently. However, averaging features may cause the dilution of critical class-related information, which fails to capture the knowledge best suited for model training. Besides, they may be influenced by a few outlier samples with extreme values, especially when facing scarce data in incremental sessions. These outliers introduce biases and noise that can compromise the model performance. To address that, we further introduce a more available strategy to build the memory and a more efficient method to use the memory.

4 Methodology

As shown in Figure 3, our CD2 framework for FSCIL begins with training a model with sufficient data in the base session. In incremental sessions, we finetune the classifier with the assistance of a memory. Before each incremental session, we generate a set by a dataset distillation module (DDM) as close as possible to critical knowledge and store it in the memory, which can be trained together with the current dataset to achieve the purpose of learning new concepts and preserving old concepts. In addition to the cross-entropy loss, we also employ a distillation constraint module (DCM) to mitigate catastrophic forgetting while balancing stability and flexibility to make better use of the generated data.

4.1 Dataset DistillationModule

Although previous memory strategies mitigate catastrophic forgetting effectively, they compress knowledge crudely.

These strategies lump together critical knowledge that enhances the discriminative ability of the model and redundant knowledge that contributes less to the model performance. Due to a lack of guidance in obtaining critical knowledge, these methods inadvertently downplay critical class-related knowledge and even overlook essential details. The dilution of critical knowledge not only impairs the model performance and discrimination ability, but also limits its potential for continuous learning and self-optimization from experience.

Based on this, we are oriented to effectively retain more critical old knowledge and propose a dataset distillation module (DDM). The core of the strategy is to choose critical class-related knowledge that is valid for incremental sessions, enhancing the adaptability and recognition ability of the model under FSCIL settings. In this way, the model can keep and update knowledge that is critical to performance, while effectively weakening less important knowledge.

Inspired by dataset distillation [Zhao et al., 2021; Liu and Wang, 2023], we synthesize samples for each class to compress critical knowledge. We randomly select K samples from D(t) for each class and initialize the synthetic set D(t) S = {(xm, ym)|ym ∈ C (t) }|C(t) |×K m=0 . While synthesizing, we freeze the whole model and update the synthetic set by class. The maximum mean discrepancy (MMD(·)) [Gretton et al., 2012] is used to estimate the distance between the real and synthetic data distribution as:

where sup is taking an upper bound and E is the expectation. To synthesize samples more finely, we convert Eq. (4) to estimate the distance of synthetic set D(t) S and real dataset D(t) and optimize the loss for each class as follows

where c ∈ C (t) . The learned synthetic set can gain complementary information and summarize important knowledge from different samples of the same class, which helps improve the power of statistical analyses and ignores sampling errors. Meanwhile, it reduces the possibility of privacy leakage because it is difficult to recover the source data. Further, to reduce training consumption while gaining critical knowledge, we replace the dataset D(t) with dataset D(t) R sampled from dataset D(t) to perform the synthesis and calculation.

4.2 Distillation Constraint Module

Facing catastrophic forgetting, we involve a memory generated by DDM during training to recall old knowledge. Generally, a synthetic sample in the memory has a different distribution and contains more abundant knowledge than a real sample. The same treatment brings the covariate shift of old classes when adding new classes, and the classification loss cannot take full advantage of rich knowledge, impacting the model performance and the discrimination ability.

Based on this, we incorporate a distillation constraint module (DCM) in incremental sessions to limit the distribution change across continual sessions, thereby bridging the distribution gap between data in the memory and the dataset, which facilitates the discriminative ability between old and new classes. The DCM consists of feature retention loss and structure retention loss, which ensure that old knowledge is as accurately retained as possible.

Feature retention loss (FR loss): The feature retention (FR) loss guarantees the consistency of features for old classes between the previous and the current session, thereby maintaining the integrity and coherence of processing data across different sessions. While calculating the FR loss, we get pairs of vectors {v(t-1) m , v(t) m }|M(t) | m=0 from the memoryM(t) , where v(t-1) m = ϕ(t-1) c (fm) and v(t) m = ϕ(t) c (fm). And we constrain the first C = | ∑t-1 i=0 C (i) | elements of each pair of vectors to be consistent, which is formulated as

With the FR loss, the pairs of vectors in the previous model ϕ(t-1) c and the current model ϕ(t) c exhibit a tendency to convergence. And the FR loss constrains the location information, which enables the gap between real data and synthetic data as constant as possible to maintain the coherence and stability of knowledge transmission.

Structure retention loss (SR loss): Further, we hope to retain old knowledge more flexibly and elastically. Inspired by relational knowledge distillation (RKD) [Park et al., 2019], we employ the structure retention (SR) loss to ensure the consistency of the structure distribution of synthetic samples in the previous session and the current session. RKD transfers structural information among a set of samples from the teacher model to the student model, endowing the student model with more flexibility to learn new knowledge. It constrains structural information by constraining the angle between triplets of samples {(xa, xb, xc)}, which is as follows

where ti = ψ(xi) and si = ζ(xi). Here, ψ(xi) is the teacher model and ζ(xi) is the student model.

We incorporate the core idea of RKD into our CD2 , transferring the structural information of the memory (synthetic samples) in the feature space from the old model ϕ(t-1) c to the current model ϕ(t) c . To constrain the feature structure, we formulate the SR loss based on a linear transformation between the features, which can be expressed as:

where P(t) = vmVT is a vector of linear transformation and V = {ϕ(t) c (fm)|(xm, ym) ∈ M(t) } is a matrix of output vectors in the memory M(t) . By minimizing this loss, we not only ensure the stability of the model when dealing with previous knowledge but also maintain its capacity for exploration and adaptation to new data and knowledge.

Distillation constraint module (DCM). The above two components form the distillation constraint module (DCM), maintaining a stable identification of old classes while being more flexible. The total loss can be summarized as follows:

where α and β are factors. The FR loss functions to constrain location information, while the SR loss aims to preserve structural information. And these dual constraints help to stabilize the distribution changes across continuous learning sessions when incorporating new classes. The DCM is designed to subtly account for the two core requirements of stability and flexibility, which reduces the covariate shift.

4.3 Model Training &Memory

Fig. 3 illustrates the training process of our CD2 in t-th session. According to different inputs, we divide the whole training into two stages: base learning and incremental learning. Base learning: During model training, we train a model with dataset D(0) with a sufficient amount of samples. For each training example xi, the model ϕ(xi) represents it into a final vector vi. In the base session, both the backbone ϕb(xi) and the classifier ϕ(0) c (fi) are jointly trained on D(0) by minimizing the empirical risk loss as

After model training, we build a small buffer as a memory to participate in subsequent training. We select the initial samples from dataset D(0) for each class and generate samples as Eq. (5) to gain a set D(0) S . And we send the set D(0) S to the memory M(1) .

Incremental learning: During model training, we freeze the backbone and finetune the classifier with dataset D(t) and the memoryM(t) . In t-th incremental session, we minimize the learning objective function as:

where Lgce is the global classification loss as

As the number of sessions increases, the ratio of previous classes to new classes gets larger, gradually making it more difficult to retain previous knowledge. The model should allocate more energy to the old knowledge as the session grows. Meanwhile, considering that the excessive proportion of the FR loss will affect the flexibility of the model, we only constrain the SR loss adaptively, which factors are set as α = ln((-50/| ∑(t) i=0 C (i) |)3 + 2).

After model training, we generate a synthetic set D(t) S from dataset D(t) in a similar way as in base learning and add it to the memoryM(t+1) for subsequent fine-tuning.

5 Experiments

5.1 Experimental Setup

Datasets. We evaluate our CD2 on three FSCIL benchmark datasets: CIFAR100 [Krizhevsky and Hinton, 2009], miniImageNet [Russakovsky et al., 2015], and CUB200 [Wah et al., 2011] following previous settings [Liu et al., 2022; Yang et al., 2023b]. On CIFAR100 and mini-ImageNet, the 100 classes are organized into 60 base classes and 40 incremental classes. 60 base classes contain 500 training images for the training model. And 40 incremental classes are further structured in 8 different sets with a 5-way 5-shot setting. And 200 classes of CUB200 are organized into 100 base classes and 100 incremental classes in a 10-way 5-shot FSCIL setting for 10 incremental sessions.

Network. Following previous work [Liu et al., 2022; Yang et al., 2023b; Ahmed et al., 2024], we employ ResNet [He et al., 2016] as a backbone. We use ResNet12 without pretraining for CIFAR100 and MiniImageNet, and ResNet18 pre-trained on ImageNet for CUB200. We adopt a three-layer MLP block and a fully connected layer as a classifier.

Implementation details. Our model is optimized using SGD with momentum and adopts a cosine annealing strategy for the learning rate during training. In the base session, we train for 100 to 200 epochs while initializing a learning rate of 0.25 for CIFAR100 and mini-ImageNet, and 0.01 for CUB200. In each incremental session, we train for 100 to 300 iterations initializing a learning rate of 0.25 for CIFAR100 and MiniImageNet, and 0.001 for CUB200. Augmentations include random resizing, random flipping, Mixup [Zhang et al., 2018], and CutMix [Yun et al., 2019]. And we set β = 0.1 for model training. When using DDM, we train 1000 iterations initializing a learning rate of 0.2.

5.2 State-of-the-art Comparison

We evaluate our CD2 on three public datasets and conduct a comparative analysis with some class-incremental learning methods with FSCIL setting and some state-of-the-art FSCIL methods using different memory strategies. Table 1 presents the results obtained on the CIFAR100, while Figure 4 illustrates the evaluation results on the mini-ImageNet and CUB200. The results demonstrate our method has superior performance across all three datasets, surpassing previous state-of-the-art methods.

Notably, our method achieves the highest average accuracy in three datasets. Firstly, our method outperforms classincremental learning (CIL) methods(such as iCaRL [Rebuffi et al., 2017], EEIL [Castro et al., 2018]), which primarily focus on mitigating catastrophic forgetting in the context of CIL rather than FSCIL. In contrast to these methods, our approach is tailored specifically to tackle the challenges of catastrophic forgetting in FSCIL. And secondly, compared with some FSCIL methods, we obtain the best accuracy in all sessions on CIFAR100 and MiniImageNet and the best average accuracy on three datasets, maintaining the accuracy advantage. Our accuracy curve declines more slowly than other methods on CIFAR100 and MiniImageNet, particularly achieving improvements of 0.80% in the base session and 2.60 - 0.39% in incremental sessions than NC-FSCIL [Yang et al., 2023b] on CIFAR100. Although the accuracy curve appears dented on several sessions on CUB200, we still achieve the highest average accuracy. The excellent performance indicates the efficacy of our CD2 , and the memory facilitates the preservation of previous knowledge for subsequent tasks effectively.

5.3 Ablation Study

To validate the effectiveness of each component, we further study the effect of different memory strategies and the DCM based on CIFAR100. More studies are in the appendix.

The effect of different memory strategies. We compare the impact of different memory strategies on accuracy as shown in Table 2. Compared with prototype computing, CD2 achieves higher performance in incremental sessions, which improves more than 0.44% in the first incremental session and more than 1.33% in the last incremental session. Different memory strategies give different knowledge to the model, which directly affects the update direction of the parameters and the final accuracy. DDM can capture the key knowledge and is different from real data, which gives the model a good foundation to learn knowledge continually.

The effect of the DCM. The DCM plays an important role in the whole training. As shown in Table 2, the model achieves higher accuracy when both SR loss and FR loss are applied. When using the SR loss alone, the model retains old knowledge stably, but may lead to displacement. When only using the FR loss, the model can retain old knowledge more accurately, but strict requirements limit the adaptation of the model to new classes. When SR and FR losses are used together, the model can subtly pursue both flexibility and accuracy, resulting in optimal overall performance.

5.4 Further Discussion

For a detailed analysis, we observe the number of synthetic sets and the visualization. More studies are in the appendix. The effect of K. As shown in Table 3, the model gets an improvement in all sessions as K increases, which means the number of generated samples affects the accuracy of the model. As the sample size increases, the model can be exposed to more diverse data and situations, thus accumulating richer knowledge, which is directly reflected in the improved accuracy in all sessions. However, when K ⩾ 2, the number of samples has a negligible impact on the model performance. Considering the resource consumption and the performance, we set K = 2.

Representation visualization. Fig. 5 clearly shows the aggregation of classes by t-SNE visualization [Laurens and Hinton, 2008]. Firstly, the samples of the same class can be aggregated well except for some special cases, where the model can effectively capture common patterns of the same class and distinguish the intrinsic properties of the different classes. Secondly, novel classes have closer inter-class distances than base classes, which means learning new data is more difficult than learning base data, resulting in the model being less sensitive to the differences between novel classes. Finally, synthetic samples are well aggregated in their classes, where the model is capable of learning from real data and effectively transferring this learning to the synthetic set. Meanwhile, this also indicates that the model successfully captures the key characteristics of the real data and saves them in simulated samples for subsequent tasks.

6 Conclusion

In this paper, we propose a framework CD2 to support FSCIL, which contains a dataset distillation module (DDM) and a distillation constraint module (DCM). Inspired by dataset distillation, DDM synthesizes highly condensed samples with class-related clues, which can capture essential information. Furthermore, we introduce DCM to constrain distribution in incremental sessions, enabling more precise memory utilization by constraining feature alignment and structure preservation flexibly and stably, which mitigates catastrophic forgetting. Extensive experiments show that our approach achieves and even surpasses state-of-the-art performances.

Acknowledgements

This work was supported by the Pioneer R&D Program of Zhejiang Province (2024C01024) and the Open Research Project of National Key Laboratory of Science and Technology on Space-Born Intelligent Information Processing (TJ02-22-01).

Contribution Statement

Daichi Zhang is the project lead of this work. Shiming Ge is the corresponding author of this work.