Masked Feature Generation Network for Few-Shot Learning

1 Introduction

Few-Shot Learning (FSL) is a challenging task that aims at recognizing the novel target classes from a few instances with the assistance of a set of disjoint data-rich base classes. As a promising direction of alleviating the data-hungry issue in the deep models, FSL has attracted a lot of attention in recent years.

Typically, FSL is tackled by learning a general model from the base data and then generalizing to the target classes. Thus, most of the existing FSL approaches are fallen into the transfer learning branch. The metric-learning-based approaches [Snell et al., 2017; Sung et al., 2018] dominate the state-ofthe-art methods and have made significant advances. Though promising results have been achieved, the metric-learningbased approaches do not explore more information for the novel target classes, which may restrict their generalization ability when predicting the target classes.

A straightforward way for FSL is to augment more data for the data-insufficient target classes. A typical strategy is to generate new samples or visual features based on the adversarial framework. For example, [Schwartz et al., 2018] and [Li et al., 2020] assume that the intra-class variations are shared across categories and attempt to capture the class distribution in either a pair-wise manner or specific regularizers. However, the performances of the most existing dataaugmented approaches are slightly worse than the state-ofthe-art methods due to the mode collapse issue, resulting in that the generated instances or visual features hardly capture the intra-class variations.

As demonstrated in [Tian et al., 2020], the good feature embedding is a key to addressing FSL tasks. We argue that the properties of good feature embedding for FSL should be both discriminant and diverse. In this paper, we formulate this goal with an encoder-decoder framework, where the encoder learns a discriminative feature extractor via training a classification task and the decoder disturbs the feature embedding space to produce diverse feature embeddings.

Inspired by the success of MAE [He et al., 2021] that reconstructs the masked patches with an asymmetric encoderdecoder architecture and leads to an efficient pre-trained vision learner, we follow and extend the idea to take a set of samples from the same class as the input and regard each sample as a patch; and the model is used to reconstruct the masked samples, as illustrated in Fig. 1. However, considering that the implementation of FSL is achieved in the visual embedding space, thus we relax the architecture and design the decoder to reconstruct the feature embeddings instead of visual instances. Specifically, the encoder is performed as a backbone to extract the visual features and the decoder is performed as a generator to reconstruct the feature embeddings of the masked visual instances. The designs of both the encoder and decoder are flexible. In this paper, the encoder is a CNN backbone and the decoder is a transformer. Selecting the transformer as the decoder is due to that the transformer takes different position embeddings as input that is beneficial in reconstructing diverse visual features. In a nutshell, the highlights of this work are summarized:

1. We propose a novel framework called masked feature generation network (MFGN) for FSL. MFGN addresses the FSL via reconstructing both discriminative and diverse visual features.
2. We apply the transformer to reconstruct diverse visual features and verify that the transformer is capable to transfer the high-level visual features.
3. We conduct extensive experiments on both coarsegrained and fine-grained benchmarks. The experimental results show that the proposed MFGN could augment some meaningful visual features, which facilitates the approach to beat most of the competitors with a significant margin.

2 RelatedWork

As a promising way to alleviate the data demands of the deep models, FSL has been attracting a lot attention in recent years. Based on the way of training the base data, the existing FSL approaches can be roughly divided into three categories, i.e., meta-learning-based approaches, metric-based approaches, and the data-hallucination-based approaches.

The meta-learning-based approaches address FSL via training a meta-learner with the meta-learning paradigm, which attempts to adjust the optimization algorithms so that the models quickly adapt to the FSL tasks, such as modelagnostic meta-learner (MAML) [Finn et al., 2017], closedform solvers [Bertinetto et al., 2018], convex learners [Lee et al., 2019], and contextual learner [Cai et al., 2018].

The metric-learning-based approaches address the FSL tasks via learning a discriminative visual embedding space with the base data and expect that the learned embedding space also works for the novel classes. The existing metriclearning-based approaches differ in learning the visual embedding space, such as ProtoNet [Snell et al., 2017], MatchingNet [Vinyals et al., 2016], RelationNet [Sung et al., 2018], and DN4 [Li et al., 2019a]. Some nontrivial techniques are further combined with the pipeline, such as the attention [Zhu et al., 2020; Dong et al., 2021; Xu et al., 2021; Hou et al., 2019] and semantic information guidance [Xing et al., 2019]. Recently, some approaches [Chen et al., 2018; Tian et al., 2020] reveal that the visual embedding space learned with a simple classification task could achieve superior performance and the fine-tuning techniques on the pretrained model could further improve the performances [Sun et al., 2019]. The proposed MFGN is trained end-to-end from the scratch without fine-tuning, thus has the potential to further improve if the fine-tuning techniques are applied.

The data-hallucination-based approaches address the FSL via synthesizing the visual features [Schwartz et al., 2018;

Li et al., 2020] to capture the class-agnostic patterns in either an adversarial method or a pair-wise manner. However, most of the hallucination-based approaches are inferior to the stateof-the-art competitors. The proposed MFGN also falls into this branch. Differently, MFGN inherits the discriminative property of the metric-learning-based approaches and focuses on capturing intra-class variations in a one-to-many manner instead of revealing the real-like feature distribution.

The most related work to ours is the masked autoencoder (MAE) [He et al., 2021], a scalable self-supervised learner for visual feature extraction. Our MFGN is inspired by MAE and also follows an encoder-decoder framework but differs in both purpose and implementation. First, our MFGN is designed for the data-limited tasks and attempt to capture the intra-class variations while MAE is designed for the datahungry models to obtain a scalable visual learner. Second, MAE masks the patches of an image and enforces the model to reconstruct the masked patches; while MFGN masks the visual features of the input instances and learns to reconstruct the visual features with both the available visual features and the mask tokens. Finally, MFGN is trained in a supervised way to extract more discriminant visual features while MAE is an unsupervised approach.

3 Methodology

Our masked feature generation network (MFGN) follows the traditional autoencoder framework that consists of an encoder and a decoder. Different from the traditional autoencoder that reconstructs the original signal with the input observations, our MFGN attempts to reconstruct the variation of the input (i.e., the feature embeddings of the input visual images) given some partial observations. MFGN shares a similar asymmetric encoder and decoder framework with MAE [He et al., 2021], but has a specific design. In MFGN, the encoder only extracts the feature embeddings of the available visual instances and the decoder reconstructs the feature embeddings of both available and unavailable visual instances from the available visual embeddings and the mask tokens. The basic framework is illustrated in Fig. 2.

3.1 MFGN Encoder

We take some instances from the same classes as the input that is analogous to the input image in the ViT [Dosovitskiy et al., 2020], and here each instance is taken as a patch. The encoder performs as a feature extractor that maps the input instances to their corresponding feature embeddings. In this paper, we take the convolutional neural networks (e.g., ResNet12 and ConvNet4) as the encoder for fair comparison. We randomly sample a few of instances from one class and then feed them to the encoder to obtain their corresponding visual features. Then, the visual features of the several randomly sampled instances are masked and replaced with mask tokens. The remaining visual features and the mask tokens are fed into the decoder. In order to learn an efficient decoder, we follow the high masking ratio strategy applied in MAE [He et al., 2021] and only feed one visible feature representation into the decoder to reconstruct the visual features of the masked instances. Such a high masking radio makes the reconstruction task difficult, thus enforcing the decoder to learn more meaningful patterns.

3.2 MFGNDecoder

The decoder in our framework is composed of a series of transformer blocks, which takes the feature embeddings of the unmasked visual instances, the sample number embedding, and the mask token as the input and produces the feature embeddings of both the masked and unmasked instances. The mask token is a shared, learned vector that indicates the presence of masked visual instances. We also add the sample number embeddings for all the instances. Each number embedding is a vector that indicates the item number of the instance.

In this way, the framework has endowed the capability of generating more visual features from the seed feature embedding, thus could be used for addressing FSL tasks, as introduced in 3.5.

3.3 Loss Functions

Instead of reconstructing the original pixel values of the input instances, our MFGN predicts the feature embeddings of the masked visual instances. Suppose that a seed (unmasked) instance and its corresponding masked instances from the same class are provided, we apply the popular mean squared error (MSE) as the loss function to measure the differences between the predicted feature embeddings and the real feature embeddings extracted from the encoder. The reconstruction loss function is formulated as:

where xi denotes the seed instance; xji denotes the j-th masked instance from the same class as xi; t denotes the mask token, sj is the sample number embedding of j; f and g denote the encoder and decoder, respectively.

In order to extract more discriminant feature representations, the whole framework is trained under the supervision of the ground-truth labels of the input instances. Thus, we also add the classification loss function to enforce the encoder

Algorithm 1 Pseudocode of MFGN in a PyTorch-like style.

# f: CNN Encoder, g: Transformer Decoder # h: Classifier # lamb, gamma: hyperparameter # t: masked token, s: sample number embedding for x_i, x_j, y in loader: # x_i, x_j are from class y # feature embedding f_i = f.forward(x_i) f_j = f.forward(x_j) [g_i, g_j] = g.forward([f_i+s_i],[t+s_j]) # logit h_i, h_j = h.forward(f_i), h.forward(f_j) z_i, z_j = h.forward(g_i), h.forward(g_j) # loss l_ce = CE(h_i, y) l_mse = MSE(g_i-f_i)+MSE(g_j-f_j.detach()) l_reg = MSE(z_i-h_i)+MSE(z_j-h_j.detach()) # objective loss = l_ce + lamb*l_mse + gamma*l_reg loss.backward() # SGD update update f, g, h, t, s

to produce discriminant feature representations. Specifically, the classification loss function is formulated as:

where yi is the class label of xi; h denotes the classifier, which is a fully connected layer.

Furthermore, to ensure the reconstructed visual features to be discriminant, we also regularize the logits of the reconstructed visual features produced by the classifier to be close to those of the corresponding masked real feature embeddings, which is formulated as:

Note that the logits of the masked real feature embeddings are detached, which means no gradients are back propagated. Using this regularizer improves the performances in our experiments, please see 4.4.

Overall, the objective of our approach is:

where λ and γ are two hyperparameters. The pseudo code is provided in Alg. 1.

3.4 Simple Implementation

Our MFGN is trained end-to-end from scratch and does not require fine-tuning during the inference, thus is efficient. During training, we randomly sample a batch of instances, each of which is seen as a seed (unmasked) instance. Meanwhile, we also sample a few instances for each seed instance from the same class as the masked visual instances. As noted, no gradients are back propagated through the masked visual instances. In other words, the masked visual instances are only used for supervision and are detached during training.

3.5 MFGN for FSL

In an FSL task, a support set consisting of N classes is provided, each class has K instances, the task is to predict the query instances from the same N classes with the classifiers derived from the support set. Once the proposed MFGN is trained, we obtain both a feature extractor (encoder) and a feature generator (decoder) and the combination of them can be used for augmenting more feature embeddins of the support classes. Specifically, the implementation has the following steps. First, each support sample is taken as the seed instance to feed the encoder to obtain the feature embedding (which can be directly used for classification, see 4.4). Second, both the seed feature embedding and the learned mask token, along with some random number embeddings are fed into the decoder to obtain a series of augmented feature embeddins. Finally, both the feature embedding of the seed instance and the augmented feature embeddins are used for predicting the query instances. In the experiments, each support class is represented as a prototype that is obtained by averaging the seed and the augmented feature embeddins from the same class. The nearest neighbor classifier is applied and the cosine distance is used for measuring the similarity between the query instances and the prototypes.

4 Experiments

In this section, we conduct a set of experiments to evaluate the proposed approach. We first document the benchmarks and the implementation details and then compare the proposed approach with the existing competitors. Finally, further analysis are provided.

4.1 Datasets

We have conducted experiments on four datasets that are Caltech UCSD Birds (CUB) [Wah et al., 2011], StanfordDogs [Khosla et al., 2011], miniImageNet [Vinyals et al., 2016], and CIFAR-FS [Bertinetto et al., 2018]. These datasets consist of collections of images comprising a varied set of categories in different scopes: birds, dogs, and objects respectively. Both CUB and StanfordDogs are fine-grained datasets, and both miniImageNet and CIFAR-FS are two coarse-grained datasets. The CUB dataset contains 11,788 images from 200 bird species. Following the split proposed in [Chen et al., 2018], we divide the data into three disjoint sets, 100, 50, and 50 categories, respectively for training, validation, and test. The StanfordDogs consists of 120 dog breeds with a total number of 20,580 images. We follow [Li et al., 2019a] and split the dataset into 70, 20, and 30 classes for training, validation, and test, respectively. The miniImageNet dataset is a subset of ImageNet, which consists of 100 classes with 600 images per class. Following the split proposed in [Ravi and Larochelle, 2016], we divide the dataset with 64, 16, and 20 classes for training, validation, and test, respectively. The CIFAR-FS is a few-shot classification dataset built on the CIFAR100. Following [Bertinetto et al., 2018], we use 64/16/20 classes for training/validation/test, respectively.

4.2 Implementation Details

For fair comparisons with the other competitors, we conduct experiments with three popular architectures as the feature extractor, i.e., ConvNet4, ResNet12, and ResNet18. Specifically, ConvNet4 consists of four convolutional blocks, each of which consists of a convolutional layer, a batch normalization layer and a ReLU activation function, and a 2 × 2 maxpooling operation. The dimensionality of the visual features extracted with ConvNet4 is 64. ResNet12 consists of four residual blocks, each of which is composed of three convolutional layers with 3 × 3 kernels; a 2 × 2 max-pooling layer is applied after each of the first three blocks; a global averagepooling layer is on the top of the final block. The architecture of the ResNet18 is similar with that of the ResNet12 and consists of four residual blocks. Differently, the last two blocks are composed of four convolutional layers. The dimensionality of the feature embeddings extracted with both ResNet12 and ResNet18 is 640.

To train the model, we apply the SGD optimizer with a momentum of 0.9 and a weight decay of 5e-4 . The batch size is set to 32. For the fine-grained datasets, we train the models with 80 epochs. The learning rate is initialized as 0.05 and decayed with a factor of 0.1 at 60 and 70 epochs, respectively. For the coarse-grained datasets, we train the models with 60 epochs. The learning rate is initialized as 0.05 and decayed with a factor of 0.1 at 40 and 50 epochs, respectively. In the training stage, we adopt the standard data augmentation strategy as [Lee et al., 2019], including random crop, color jittering, and random horizontal flip. The images from CUB200-2011, StandfordDogs, and miniImageNet are resized as 84×84 and the size of the images from CIFAR-FS is 32×32.

To evaluate the model, we conduct 600 random few-shot classification tasks and report the average classification accuracy and 95% confidence interval. If not specific, both λ and γ are set to 1; the number of synthesized visual features is set to 5 for all datasets.

4.3 Results

Results on fine-grained datasets. Table 1 reports the 5Way 1-Shot and 5-Way 5-Shot classification results on both CUB and StanfordDogs datasets with the three different architectures. From the results, we reach the following conclusions. First, the proposed MFGN performs competitively on the fine-grained datasets under different few-shot classification tasks. When comparing with the models trained with the ConvNet4 architecture, our MFGN performs the best on both datasets under different few-shot tasks. Specifically, MFGN achieves 69.36% and 60.22% 5-Way 1-Shot classification accuracy on CUB and StanfordDogs, respectively beats the second-best results with 3.07% and 5.38%. MFGN also obtains 0.41% and 2.83% improvements on the CUB and StanfordDogs under the 5-Way 5-Shot task, respectively. When comparing the models trained with the ResNet18 architecture, our approach improves significant margins than the competitors that are both feature augmented approaches. When comparing the models trained with the ResNet12 architecture, our MFGN performs much better than the competitors on both datasets under different tasks. Specifically, MFGN obtains respectively 5.43% and 6.22% improvements to the second-best result under the 5-Way 1-Shot classification task on CUB and StanfordDogs, which is significant. Second, the results of both ResNet12 and ResNet18 are much better than the low capacity ConvNet4 architecture, which indicates the important role of the feature embeddings in the FSL tasks.

Results on coarse-grained datasets. Table 2 demonstrates the results of both miniImageNet and CIFAR-FS datasets with three different architectures under the 5-Way 1-Shot classification task. We select twelve competitors for comparison. They are ProtoNet [Snell et al., 2017], RelationNet

[Sung et al., 2018], R2D2 [Bertinetto et al., 2018], DC [Yang et al., 2020], ATLNet [Dong et al., 2021], RFS [Tian et al., 2020], MetaOptNet [Lee et al., 2019], Shot-Free [Ravichandran et al., 2019], MTL [Sun et al., 2019], MetaOptNet [Lee et al., 2019], AFHN [Li et al., 2020], and HGNN [Yu et al., 2022].

From the results, we observe that the proposed MFGN also performs very competitively on the coarse-grained datasets. When comparing with the models trained with the ConvNet4, MFGN achieves 57.48% and 66.1% on miniImageNet and CIFAR-FS datasets, respectively improves the second best competitors with 1.16% and 0.8%. When comparing with the models trained with the ResNet18, MFGN beats the second best competitors with a large margin on both miniImageNet and CIFAR-FS datasets. When comparing with the models trained with the ResNet12, MFGN also achieves the best on both datasets, which has 0.23% and 3.2% accuracy boost over the second best competitors on miniImageNet and CIFAR-FS, respectively. The experimental results on both fine-grained and coarse-grained datasets reflect that the generalization ability of MFGN is better than most of the competitors in dealing with the classification tasks with extremely limited data.

4.4 Further Analysis

Impacts of training class number. To evaluate the impacts of training class number to the few-shot classification performances, we train the model with the train classes and the combination of both train and validation classes. We report the comparison results in Table 3. From the results, we conclude that training with more classes is benefit in improving the FSL performances. This is reasonable due to that more training classes would provide more information and transferable patterns for the novel classes.

Impacts of loss functions. In this subsection, we evaluate the impacts of different loss functions. Specifically, we fix γ as 1 and range λ from 0 to 1 with an interval of 0.2 to evaluate the impacts of Lmse. Similarly, we fix λ as 1 and range γ from 0 to 1 with an interval of 0.2 to evaluate the impacts of Lreg. Fig. 3 shows the FSL results of both CUB and StanforDogs datasets with different λ and γ with ConvNet4. From the results, we observe that the performances are the best when λ is 0.4 and γ is 0.8 for both datasets and conclude that both Lmse and Lref bring significant improvements to the classification performances for both datasets.

Impacts of augmented visual features. In this subsection, we evaluate the impacts of the augmented visual features to the few-shot classification during test. Table 4 reports the 5-Way classification results on CUB and Dogs datasets varying the number of augmented visual features with ConvNet4. From the results, we observe that the results obtained with the augmented visual features are better than those without the augmented visual features, which indicates the augmented visual features indeed provide nontrivial information for the few-shot classification. Besides, we observe that more synthesized visual features would bring little benefits in the performance improvement. In the experiments, we report the results with 5 augmented visual features.

Fig. 4 shows the visualization of the feature embeddings of a random test task from the CUB dataset obtained with the models trained with and without the decoder. From the results, we observe that the decision boundaries of the model with the decoder are sharper than that without the decoder, which indicates the decoder also boosts the discriminative of the model.

(b) Results on the StanfordDogs with different λ and γ

Figure 4: tSNE visualization of a FSL test task from CUB dataset. For Baseline (without decoder), the decision boundaries are produced with the feature embeddings of original 5 samples from each class. For MFGN, the decision boundaries are produced with both original and generated features.

5 Conclusion

In this paper, we have proposed a feature-augmentation approach called MFGN for FSL. MFGN addresses FSL via learning both discriminant and diverse feature embeddings with an encoder-decoder framework. Though straightforward it is, MFGN achieves quite promising FSL results on both fine-grained and coarse-grained datasets. Besides, based on the observations from the experiments, we conclude that intra-class feature variation is important for FSL and our method implemented with a transformer could effectively capture the intra-class variations.

Acknowledgments

This work is supported in part by the Key R&D Program of Zhejiang Province, China (2021C01119), the National Natural Science Foundation of China under Grant (62002320, U19B2043, 61672456), the Fundamental Research Funds for the Central Universities (2020QNA5010).