FineFMPL: Fine-grained Feature Mining Prompt Learning for Few-Shot Class Incremental Learning

1 Introduction

Recently, deep networks have achieved remarkable performance in numerous computer vision tasks owing to massive data and computational resources. In practice, their performance is severely limited when dealing with a continual data stream from unseen new classes [Ji et al., 2023]. To handle this problem, Class Incremental Learning (CIL) is investigated to continually learn new classes with abundant labeled data while alleviating the catastrophic forgetting of old classes. However, the strict requirement of sufficient training samples of new classes is impractical in many scenarios when annotated data are hard to obtain [Tao et al., 2020]. Therefore, the Few-Shot Class Incremental Learning (FSCIL) in Figure 1, i.e., the model is trained on abundant labeled samples in the base session and learns new classes from few labeled samples in incremental sessions without seeing old classes, has attracted more and more attention recently [Song et al., 2023]. FSCIL encompasses the catastrophic forgetting problem of inaccessible old class data and the overfitting problem caused by scarce new class samples.

∗Corresponding author.

A popular FSCIL paradigm is to well-train the backbone network in the base session which is frozen for fine-tuning the classifiers to learn new classes via knowledge distillation [Dong et al., 2021; Cheraghian et al., 2021; Zhao et al., 2023], attempting to achieve a balance between suppressing forgetting old classes and learning new classes. Though promising progress has been achieved, the above methods are still struggling due to the limited learning ability of the backbone network on new data. As the feature extractor, the backbone network is trained on abundant data in base sessions and frozen in incremental sessions, which causes a lower learning ability for new classes compared with the base classes [Tao et al., 2020; Zhang et al., 2021]. In addition, the classifiers are fine-tuned by very few training samples from the entire images, which generally lack attention to the discriminant parts of distinguishing different classes.

Recently, the large-scale Vision-Language (VL) pretraining models, such as the well-known CLIP [Radford et al., 2021], have shown powerful feature extraction ability, which can be adapted to various vision tasks by prompt learning [Zhou et al., 2022d; Sun et al., 2023b]. Inspired by the above observations and analyses, we propose the Finegrained Feature Mining Prompt Learning (FineFMPL) of VL model to adapt it to the FSCIL task, which learns and memorizes fine-grained discriminative information of emerging classes to achieve promising performance, as shown in Figure 1. Concretely, a visual probe prompt is proposed to induce the image encoder of VL model to scale and gather discriminative image patch information from visual objects. Then, visual prototypes of classes are constructed based on the image patch significance analyses and weighted features aggregation to memorize the class from the visual side. Next, the textual context prompt is proposed conditioned on crossmodal mapping of visual prototypes, which contains implicit object attribute information. It is then input into the text encoder of VL model to generate textual prototypes to depict and memorize the class information from the textual side. Finally, classes' visual and textual prototypes are comprehensively utilized for the few-shot class incremental learning. To sum up, the main contributions can be summarized as follows:

• We propose a Fine-grained Feature Mining Prompt Learning (FineFMPL) method to guide the visionlanguage model to learn and memorize discriminative information of classes as visual and textual prototypes for few-shot class incremental learning.
• The visual probe prompt is proposed to scale and gather object-level information, and visual prototypes of emerging classes are preserved. Based on the crossmodal mapping of visual prototypes that contain implicit object attribute information, the textual context prompt is constructed to depict and memorize classes as textual prototypes.
• Extensive experiments are conducted on three widely used FSCIL benchmark datasets to demonstrate that our proposed FineFMPL approach surpasses existing stateof-the-art FSCIL methods significantly.

2 RelatedWork

In this section, we briefly review related works about class incremental learning, few-shot class incremental learning, and prompt learning.

2.1 Class Incremental Learning

In the Class Incremental Learning (CIL) task, the critical challenge is to learn new classes without forgetting the old knowledge. Existing CIL works can be roughly classified into three kinds to address this problem. The first kind of CIL works regularize the model's predictions [Hinton et al., 2015; Li and Hoiem, 2017; Liu et al., 2023b] between the old and current models, where knowledge distillation is commonly used. The second kind of CIL works [Castro et al., 2018; Hou et al., 2019] select a few representative samples of old classes, which are utilized for rehearsal in learning new classes. The third kind of CIL works [Abati et al., 2020; Han et al., 2023] attempt to expand the network for learning new classes. Some prompt learning-based CIL methods [Wang et al., 2022; Smith et al., 2023] recently have been proposed to achieve promising performance. However, it is noted that all the above CIL methods need abundant training samples of both the old and new classes, whose performance drops sharply when there are only few training samples of new classes in realistic scenarios. Therefore, it spurred the research of few-shot class incremental learning.

2.2 Few-shot Class Incremental Learning

Few-Shot Class Incremental Learning (FSCIL) continually recognizes new classes of a few training samples, which face both the catastrophic forgetting problem in CIL and the overfitting problem in few-shot learning [An et al., 2023]. FSCIL was first proposed in [Tao et al., 2020], which utilized a neural gas network to preserve topologies of classes. In the following works, CEC [Zhang et al., 2021] proposed to utilize an independent classifier for each class, where the graph model conducted the information interaction between classifiers. F2M [Shi et al., 2021] proposed finding flat minima to alleviate the catastrophic forgetting problem. Recently, SAVC [Song et al., 2023] proposed introducing semantic knowledge by imaging virtual classes to help divide the classification space during training. CABD [Zhao et al., 2023] proposed the class-aware bilateral distillation to transfer the knowledge from base classes to new classes for alleviating the forgetting and overfitting problem. However, existing FSCIL works generally fix the model's parameter in incremental sessions, which attempt to balance the stability for base classes and plasticity for new classes. Due to the data volume disparity, they are prone to base classes of abundant data, which limits the model's ability to recognize new classes.

2.3 Prompt Learning

Prompt learning is proposed to adapt the large-scale pretrained models to downstream tasks with limited data, which was first proposed in natural language processing. For example, general knowledge is extracted from GPT [Radford et al., 2019] and BERT [Devlin et al., 2018] for various downstream language tasks with prompt designs, such as utilizing learnable vectors as prompts [Li and Liang, 2021]. Recently, prompt learning has been introduced into the computer vision area, which attempts to adapt the Vision-Language (VL) pre-training model to downstream vision tasks, such as image classification. As the pioneering work, Zhou et al. [Zhou et al., 2022d] proposed CoOp to introduce learnable vectors into the text prompt as the context, which obtained more adaptive classification weights with the text encoder of CLIP. As an extension work of CoOp, Zhou et al. [Zhou et al., 2022c] further proposed an input-conditional text prompt with a mapping neural network, which injected the visual information into the text prompt. Besides, some adapter methods have also been researched for adapting the VL model. Zhang et al. [Zhang et al., 2022] proposed Tip-Adapter-F to utilize the few-sample training set to construct the cache model, which was fine-tuned to adapt VL models to image classification.

In summary, VL models own powerful feature extraction and generalization ability. However, the above methods cannot achieve satisfactory results in FSCIL because of the catastrophic forgetting problem of old classes and the adaptation problem to new classes with only a few samples. Inspired by the importance of discriminative features in classification [He et al., 2022; Sun et al., 2022; Sun et al., 2023a], we propose a fine-grained feature mining prompt learning method to induce the VL model to sufficiently learn and memorize discriminative information of objects as visual and textual prototypes, which benefits the continual learning of new classes with limited data while not forgetting old classes in FSCIL.

3 Approach

The overview of our FineFMPL model is shown in Figure 2. There are two branches to extract coarse-grained global features and fine-grained object features where the proposed visual probe prompting is utilized to scale and gather discriminative information. Global-level and object-level visual prototypes are constructed based on the patch significance calculation for memorizing the class's visual knowledge. Textual context prompt is built by cross-modal mapping of the visual prototypes that contain implicit object attribute information, which generates textual prototypes to depict classes from the textual side for utilizing the cross-modal knowledge in VL models. Finally, the image classification is conducted in a two-pathway recognition way, i.e., visual-visual matching calculation and visual-textual matching calculation.

3.1 Visual Probe Prompt

Discriminative information learning is essential for learning differences among different classes, which plays an important role in image classification. Thus, we propose introducing a Visual Probe Prompt (VPP) into the image encoder of the VL model, which induces the model to scale and gather significant image patch information of visual objects for feature extraction, as shown in Figure 2.

Concretely, the visual probe prompt (VPP) is inserted into the input sequence of the image encoder of the VL model. Then, two branches are constructed to capture global information and visual object information, respectively. The original branch of the image encoder is utilized for extracting the global-level feature in the class token (abbreviated as CLS in Figure 2), denoted as f ′ (glo). A new branch for scaling and gathering discriminative image patch information is constructed with the proposed VPP prompt. Specifically, we denote the output of the L - 1 layer of the image encoder as zL-1 = [CLSL-1 , ...,FL-1 (xip), ...,FL-1 (VPP)]. Through the learnable VPP token, we scale the image patch token features adaptively as follows:

where ⊙ demotes the element-wise multiplication. Then, znew L-1 is input into the last transformer layer, which conducts the information interaction to obtain [f ′ (CLS), ..., f ′ (xip), ..., f ′ (VPP)]. Then, the f ′ (VPP) is utilized as a probe to gather discriminative information from image patch tokens based on the probe attention as the complement of f ′ (CLS):

where N denotes the number of image patches. Thus, the proposed visual probe prompt aggregates the information from significant image patches adaptively to obtain the discriminative object-level feature f ′ (obj).

3.2 Visual Prototypes Construction

In FSCIL, there are only a few training samples when learning new classes in incremental sessions. It needs to memorize discriminative information of new classes to distinguish them from old classes. Thus, we extract global-level and objectlevel visual prototypes from the training samples, as shown in Figure 3. The visual object in the image generally contains distinct image patch information crucial to the final classification. Thus, the patch significance calculation is utilized to induce the model to extract and memorize discriminative visual object information.

The image is first split into image patches as the input of the image encoder of the VL model. Concretely, the image X ∈ RH×W×3 with height H and width W is split with sliding stride S. Thus, the number of image patches is N = ⌊ HS ⌋×⌊ WS ⌋. The image patch is then projected by linear mapping F(·) and combined with the class token CLS. Next, the position embeddings are added to introduce the position information. The input image patch sequence is denoted as z0 = [CLS,F(x1p),F(x2p), ...,F(xNp )]. It is noted that the CLS token represents the whole image for interacting with all the image patches in the transformer layers, which is utilized for the final classification. Specifically, a multi-head self-attention (MSA) module and a feed-forward neural network (FFN) in the transformer layer propagate information among image patches and the CLS token. We denote the input of kth transformer layer as zk-1, and its output can be obtained as follows:

where LN(·) denotes the layer normalization. The above calculation in the transformer layer enhances the global representation ability of CLS token to cover comprehensive context information, which is then saved as the global-level visual prototypeCGj of the jth class by the averaging operation on all the training samples of the jth class.

In the self-attention calculation of the transformer, the higher the impact of the image patch on the CLS token, the higher its significance on the final classification, which can reveal the visual object in the image. Assume H selfattention heads exist in the lth transformer layer. We utilize the Q and K to denote the query vectors and key vectors of the image patches and CLS token. d denotes the dimension of the above vectors. The self-attention weights are calculated as follows:

where Alh ∈ R(N+1)×(N+1) (h = 1, 2, ...,H) depicts the mutual significance of image patches and CLS token. There are L transformer layers in total. We adopt the recursive matrix multiplication to calculate the total attention for hth attention head, following [He et al., 2022].

The attention weight between the CLS token and other image patch tokens is extracted from Ah and denoted as AMcls h ∈ RN×1 . Taking all the H attention heads into account, and then the final significance of each image patch token (Figure 3) can be calculated as follows:

For extracting the object-level visual features, we adopt the following weighted sum way:

Then, we can get the object-level prototype COj for the jth class by utilizing the averaging calculation on all the training samples of the jth class.

In summary, based on the image patch significance analysis, we get the global-level prototype CGj to memorize the comprehensive context information of classes. The objectlevel prototype COj is obtained to save the discriminative information in visual objects of classes.

3.3 Textual Context Prompt

The textual feature of class information extracted by the VL model can benefit the recognition of images by the text prompt design [Zhou et al., 2022d]. Intuitively, the more comprehensive context information the prompt covers, the more detailed class information that the textual feature describes. Thus, we propose textual context prompts conditioned on cross-modal mapping of visual prototypes, which contains implicit object attribute information, such as bird wing texture, based on the image patch significance analyses, as shown in Figure 2 and Figure 4.

Concretely, We can get the global-level visual prototype CGj and object-level visual prototype COj of the jth class in Section 3.2. To comprehensively use the visual features, we first learn the mutual feature by concatenating the features and mapping as follows:

where MLP conducts the cross-modal mapping. tj is extracted from the two kinds of visual prototypes, which contain different context information. CGj contains the global context, including the situation, and COj contains the object attribute information, including the object's color and texture. For obtaining stronger context information, the attentionbased feature enhancement is conducted as follows:

where ⊙ demotes the element-wise multiplication. tGj and tOj are added to learnable prompt tokens to form textual context prompts, which are input into the text encoder of the VL model to get the global-level textual prototype TGj and object-level textual prototype TOj to memorize the information of classes from the textual side. It benefits the model's final classification performance by utilizing the cross-modal alignment knowledge in VL models.

3.4 Inference

As shown in Figure 2, we construct two pathways for the model's recognition based on the dual-modality prompting. The first pathway utilizes the visual information of the training samples memorized by visual prototypes, and the second pathway utilizes the cross-modal knowledge contained in the textual prototypes.

In the first pathway, we utilize global-level visual prototype CGj and object-level visual prototype COj for cosine similarity matching calculation with corresponding extracted features of f ′ (glo) and f ′ (obj). A simple MLP network is added before the similarity calculation to narrow the gap between the pre-trained data of the VL model and downstream data. Finally, we get the prediction logit for the global level pI1 and object level pO1 and the total prediction logit for the first pathway is calculated:

where α is a tunable hyper-parameter. We obtain the prediction logit for the second pathway by calculating the crossmodal similarity directly, which utilizes global-level textual prototype TGj and object-level textual prototype TOj for matching calculation with f ′ (glo) and f ′ (obj). The prediction logit are denoted as pI2 and pO2 , respectively. The total prediction logit for the second pathway is p2 = αpO2 + pI2 . Thus, we can get the final prediction logit p of the image:

where β is a tunable hyper-parameter. Our FineFMPL approach mines and memorizes discriminative information of classes as visual prototypes and textual prototypes, which learns the classes sufficiently and alleviates the forgetting to benefit FSCIL.

4 Experiments

We conduct extensive comparison experiments and ablation studies on three standard few-shot class incremental learning (FSCIL) benchmark datasets, i.e., CUB-200-2011 [Wah et al., 2011], CIFAR 100 [Krizhevsky et al., 2009], and miniImageNet [Russakovsky et al., 2015], which shows the effectiveness of our proposed FineFMPL approach.

4.1 Dataset andMetric

For fair comparisons with state-of-the-art (SOTA) FSCIL methods, the same benchmark datasets and FSCIL setting [Tao et al., 2020] are adopted, as shown in Table 1.

CUB-200-2011. It is a fine-grained image classification dataset comprising 11,788 images from 200 bird classes. Subtle differences among different bird classes make this dataset very challenging. In FSCIL, 100 classes are selected as base classes, and the remaining classes are split into 10 sessions, where each session learns 10 classes with 5 examples for each class (10-way-5-shot).

CIFAR100. This is composed of 60,000 images from 100 classes. In the few-shot continual learning process, 60 classes are selected as the base class set, and the remaining 40 classes are incremental classes. There are 8 continual sessions, learning 5 new classes with 5 examples each class in each session, i.e., the 5-way-5-shot setting.

miniImageNet. It covers 60,000 images from 100 classes. 60 classes are set as base classes, and the remaining 40 classes are divided into 8 sessions, which learns 5 new classes with 5 examples each class, also in a 5-way-5-shot manner.

The classification accuracy is adopted as the evaluation metric, which is calculated after each session.

4.2 Implementation Details

Our FineFMPL approach adopts the widely-used public VL model, i.e., CLIP [Radford et al., 2021], as the backbone, where the ViT-B 16 is selected as the image encoder, and the transformer network is utilized as the text encoder. The original weights of CLIP are frozen during the whole training stage. We follow the mainstream experimental setting in [Tao et al., 2020] for a fair comparison. In the training process, we set 50 training epochs for CUB-200-2011 in the base session and 10 epochs for each incremental session. For the CIFAR 100 and miniImageNet datasets, we set 30 training epochs in the base session and 10 epochs in each incremental session. α is set to 0.5, 2, 0.5, and β is set to 1.5, 1, and 0.5 for the three datasets, respectively. The batch size is set as 256. The learning rate is initialized as 1e-3 in the base session and 1e-4 in each incremental session, which all adopt the cosine annealing schedule. AdamW [Kingma and Ba, 2014] is utilized as the model optimizer. All the experiments are conducted on one NVIDIA A40 GPU with Pytorch.

4.3 Comparison with State-Of-The-Art Methods

We conduct extensive comparison experiments with state-ofthe-art (SOTA) methods on three standard FSCIL benchmark datasets. The comparison results are shown in Table 2 and Figure 5. We can observe that:

• On the challenging CUB-200-2011 dataset, which has subtle differences among different classes, our proposed FineFMPL approach outperforms the compared methods by significant margins in each session, as shown in Table 2. Compared with existing SOTA FSCIL methods using pure vision backbone networks such as ResNet [He et al., 2016], our FineFMPL surpasses the typical F2M, SAVC, CABD, and TEEN by 10.2%, 10.3%, 12.4%, and 13.1% on the average classification accuracy of all sessions, respectively. Besides, our FineFMPL approach can keep the performance better in the incremental sessions compared with other SOTA FSCIL methods. SAVE proposed to imagine virtual classes to introduce semantic knowledge for enhancing the separation of different classes. By contrast, we propose to transfer the general knowledge of the VL model and mine the discriminative information of classes with the fine-grained dual-modality prompts design. Thus, our FineFMPL has a stronger generalization ability to learn new classes with limited data, which alleviates the overfitting problem in FSCIL to a large extent. Compared with CABD, which utilizes the class-aware bilateral dis-

tillation from base classes, our FineFMPL model directly memorizes discriminative information of classes as the visual and textual prototypes, which helps alleviate the catastrophic forgetting problem of old classes.

• We compare our FineFMPL method with the promptingbased methods, such as CoOp and the recent FSCIL method IOS, which all utilize CLIP as the backbone. Our FineFMPL model achieves 9.0% and 8.7% average performance gains, respectively. Compared with the textual prompt pool construction of IOS in different sessions, our FineFMPL method utilizes both the visual prompt and textual prompt for inducing the VL model to gather significant image patch information and depict class information, which enhances the model's learning ability of classes in FSCIL.
• Figure 5 shows the classification accuracy trend on the CUB-200-2011, CIFAR100, and miniImageNet datasets. Our FineFMPL approach can achieve consistent performance gains, bringing 8.7%, 6.7%, and 1.2% average improvements over the sub-optimal method. Our FineFMPL approach declines more slowly in incremental stages. We attribute it to discriminative information memorizing from both the visual and textual sides.

4.4 Ablation Experiments

Experimental results of ablation studies on the three FSCIL datasets are shown in Table 3. We can observe that:

• Compared with CLIP (our baseline), i.e., the first row, our FineFMPL brings significant average accuracy gains of 21.5%, 15.1%, and 5.6% on the CUB-200-2011, CIFAR 100, and miniImageNet datasets, respectively. We attribute the gains to our FineFMPL's inducing CLIP to learn and memorize the discriminative information of classes to alleviate old classes' forgetting and new classes' learning simultaneously.
• Compared with the baseline, our visual probe prompt brings 19.7%, 10.7%, and 3.6% performance gains when using both the global-level and object-level prototypes (as shown in row 4). The proposed visual probe prompt scales and gathers discriminative visual features, which help learn the classes sufficiently to improve the FSCIL performance.
• Compared with only utilizing the fine-grained visual prompt, the recognition accuracy further improves by 1.8%, 4.4%, and 2.0% through introducing the textual context prompt, as shown in the last two rows of the table. The memorization of classes in the textual side brings an important complement to visual information kept in visual prototypes, which benefits alleviating the forgetting problem in FSCIL.

4.5 Hyper-parameter Experiments

Hyper-parameter experiments about α in Eq. 13 and β in Eq. 14 are conducted on the CUB-200-2011 dataset under the FSCIL setting. The experimental results are presented in Figure 6. The best performance is obtained when α is set as 0.5. The trend of the curve indicates the importance of the object branch, which captures crucial discriminative traits for assisting classification. Our proposed FineFMPL approach can perform best when β is set as 1.5. It verifies the effectiveness of discriminative visual information memorized by the global-level and object-level visual prototypes, which alleviates the forgetting problem when learning new classes.

5 Conclusion

In this paper, we propose Fine-grained Feature Mining Prompt Learning (FineFMPL) of the Vision-Language (VL) pre-training model for Few-Shot Class Incremental Learning (FSCIL), which learns and memorizes discriminative information of classes as visual and textual prototypes. We propose the visual probe prompt for inducing the image encoder of the VL model to extract significant image patch information of visual objects, and the visual prototypes of classes are preserved to save visual knowledge of classes. The textual context prompt is then constructed and conditioned on the cross-modal mapping of visual prototypes that contain object attribute information implicitly, which help depict the class information as textual prototypes. Sufficient discriminative information learning and memorizing benefits the understanding of new classes with few training samples while not forgetting old classes, which achieves promising performance in FSCIL.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grants 61925201, 62132001, and 62373043.