Conditional Self-Supervised Learning for Few-Shot Classification

1 Introduction

Different from previous deep learning based methods that require a large number of manually labeled training data, fewshot learning methods can mimic human to recognize new classes with very few examples. The recent work of visual few-shot learning can learn a transferable feature representation from training a collection of tasks on base classes and generalize the representation to novel (unseen) classes with very few examples [Chen et al., 2019b]. However, due to the data scarcity, the obtained supervised information mainly focuses on the differences of the base class samples and ignores the semantic information within samples valuable for novel classes. Therefore, more semantic information for good representation from limited samples should be extracted for fewshot classification problems.

∗Contact Author

Self-supervised learning has emerged as an important training paradigm for exploring a strong visual representation, without relying on pre-defined annotations [He et al., 2020; Chen et al., 2020]. Usually, the self-supervised representation is learned from a pretext task by constructing correlated views with augmentation operations (e.g., rotation or exemplar) on original data and making a distinction between augmented views and the original one. Another method of self-supervision adopts contrastive loss, which brings representations of different views of the same data closer ("positive pairs"), and spreads representations of views from different data ("negative pairs") apart [He et al., 2020].

Recently, self-supervised learning considered to offer additional semantic information is applied to the few-shot classification [Gidaris et al., 2019; Su et al., 2020; Lee et al., 2020]. These self-supervision based few-shot learning methods use self-supervised tasks as auxiliary tasks and original few-shot classification tasks as main tasks to jointly learn a same representation. However, self-supervision is usually dependent on the availability of large training samples and unsuitable for few-shot scenarios. The direct application of previous selfsupervision learning tasks for few-shot scenarios might easily fall into the learning of undesirable shortcuts [Noroozi and Favaro, 2016a], e.g., color histograms and edge continuity instead of key semantic information. Therefore, the learning of self-supervision might be biased, which might lead to the wrong guidance for the main classification tasks and finally cause the performance degradation.

In this paper, we propose conditional self-supervised learning (CSS) that can be more resilient to few-shot classification. CSS treats supervised few-shot and self-supervised tasks equally and learns two feature representations by these two paradigms respectively. For self-supervised part, CSS leverages supervised information as a teacher to guide the learning of self-supervision. Finally, all the information is integrated into a unified distribution which can further enrich and broaden the original representation. Therefore, the knowledge learned by CSS can infer other things from one fact and further improve the generalization performance. It is worth noting that our approach is fundamentally different from semi-supervised learning methods, and does not require any auxiliary unlabeled data.

(c) Meta-training stage

Figure 1: Architecture of conditional self-supervised learning for few-shot classification.

The contributions of our work are:

• We explore a new pattern of self-supervision for fewshot learning. CSS learns two different representations by supervised and self-supervised learning respectively instead of the previous shared representation pattern, and fuses these representations into a new and augmented one to further enrich and broaden the semantic representation capacity for few-shot learning.
• We design a novel conditional module applied to selfsupervised learning. The module can shape the learning feature manifold of self-supervision and reduce representation bias. To the best of our knowledge, it is the first time to leverage supervised information to guide the training of self-supervision in few-shot learning.
• We propose CSS with a 3-stage training pipeline: supervised pre-training stage, self-supervised training stage and meta-training stage. In the last stage, we design a novel knowledge distillation (FD) to reinforce the fusion of representations learning from the first two stages.
• We perform adequate experiments to demonstrate that our approach is totally superior to original and selfsupervision-based few-shot classification methods. Systematic studies by varying the combinations of different stages are conducted, which verifies the effectiveness of

the proposed three-stage training method.

2 RelatedWork

Few-shot classification. Few-shot classification aims to acquire transferable visual representation by learning to recognize unseen novel classes from few samples with abundant training on base classes [Chen et al., 2019b]. Many efforts have been devoted to overcoming the data efficiency issue and can be mainly divided into two categories: the gradient-based model and metric-based model. The gradientbased model [Andrychowicz et al., 2016; Finn et al., 2017; Rusu et al., 2019] can rapidly adapt a model to a given task via a small number of gradient update steps. MAML [Finn et al., 2017] is a representative as the gradient-based model. This method advocates learning a suitable initialization of model parameters by learning from base classes, and transfers these parameters to novel classes in a few gradient steps. Metric-based model that leverages similarity information between samples to identify novel classes with few samples [Koch et al., 2015; Vinyals et al., 2016; Snell et al., 2017; Sung et al., 2018]. Prototypical Network [Snell et al., 2017] is a representative of metric-based model. It takes the mean of support samples i.e., training samples of a class as its class prototype and classifies a query, i.e., test sample according to the distance to each class prototype. In our work, we mainly consider the classification model combined with Prototypical Networks and cosine classifiers.

Self-supervised learning. Without the expensive manual annotations, self-supervision learns a feature representation by constructing pseudo-labels for annotation-free pretext tasks with only input signals and learning to predict them.

The recent advances of self-supervised learning usually consist of two components: image generation and contrastive learning. The former dedicates to design the pretext tasks to exploit compact semantic visual representation such as relative rotation prediction [Chen et al., 2019a], jigsaw puzzles [Noroozi and Favaro, 2016b], relative patch location prediction [Doersch et al., 2015] and colorization [Zhang et al., 2016]. However, these pretext tasks rely on specific transformations and heuristics and harm the generalization of learned representations [Chen et al., 2020]. Contrastive learning [Chen et al., 2020; Grill et al., 2020] currently achieves stateof-the-art performance in self-supervised learning. Contrastive learning aims to learn the feature representation of samples by bringing the representations of positive pairs closer, and spreading representations of negative pairs apart. Wu et al. [Wu et al., 2018] adopts a memory bank to store the instance class representation vector. Momentum Contrast (MoCo) [He et al., 2020] trains a visual representation encoder by matching an encoded query q to a dictionary of encoded keys with contrastive loss. SimCLR [Chen et al., 2020] uses the normalized temperature-scaled cross-entropy loss as contrast loss. BYOL [Grill et al., 2020] only relies on positive pairs rather than negative pairs to learn the feature representation. SimSiam [Chen and He, 2020] removes the momentum encoder and designs a simpler one based on BYOL. In this paper, we use the state-of-the-art SimSiam to extract semantic information of samples for simplicity and flexibility, which is more resilient to few-shot classification.

Some researchers are devoted to using self-supervision to squeeze out generalizable features and structural information from low data regimes. [Gidaris et al., 2019] proposes a multi-task method combining self-supervised pretext as an auxiliary loss with the main few-shot loss. [Su et al., 2020] uses self-supervised learning as a regularizer and uses importance weights to select data for self-supervision. SLA [Lee et al., 2020] treats the pretext task as augmented labels and avoids the optimization of the weighted summation loss between supervised and self-supervised tasks. These methods directly apply self-supervision as an auxiliary task to the main few-shot task. However, due to limited samples, undesirable shortcuts might be learned by self-supervision, which inevitably hurts the learning of key semantic information.

Different from the previous work, we use different feature representations for different tasks. Firstly, a supervised representation is trained by the few-shot classification task. Then, we use the supervised representation as a teacher to guide the self-supervision to avoid exploiting shortcuts to solve selfsupervised tasks. Finally, we learn a reinforced representation from the above representations with a novel fusing distillation method (FD). In that case, our method can benefit from these different paradigms respectively and learn a better semantic representation.

3 Method

3.1 Preliminary

Few-shot learning uses the episode training process and each episode samples a task. The meta train dataset and test dataset are represented by Dtr = {Ti}Ii=1 and Dte = {T ∗ i }I ∗ i=1 respectively, where T denotes the task. Each task instance contains a support set and a query set. The goal of each task is to estimate the class of samples in the query set with support set. For the meta train dataset Dtr = {Ti}Ii=1 = {Si, Qi}Ii=1 , the support set Si is composed of N×K samples that N classes are randomly selected from all classes of Dtr and K samples are extracted for each class. The query set Qi is composed of N×M samples that M samples for each class are unseen in Si. The model is trained by randomly selecting tasks from the task set {Ti}Ii=1 . Similar to Dtr, the test dataset Dte = {T ∗ i }I ∗ i=1 is considered as a task set, from which a new task T ∗ i = {S∗ i , Q∗i } is sampled and the classes of S∗ i and Q∗i are from Dte but unseen in Dtr.

3.2 Methodology

As illustrated in Figure 1, the proposed CSS uses a 3-stage training pipeline. Firstly, in the pre-training stage, CSS learns an initial feature extractor fθ (·) through the original supervised learning method. In the next self-supervised training stage, CSS uses the learned fθ as a prior condition to optimize the learning of self-supervision model gξ (·). In the final meta-training stage, CSS distills the knowledge of fθ (·) and gξ (·) learned in the first two stages into the final feature embedding network hϕ (·) through a novel fusion distillation method. It is worth noting that in all the three training stages, only Dtr is needed, without introducing any auxiliary datasets. In the remaining part of the section, we will describe these training stages in detail.

3.3 The Pre-Training Stage

In the pre-training stage as shown in Figure 1(a), feature extractor fθ (·) is learned with original few-shot classification tasks. For N-way K-shot problem, in each training episode, a few-shot task Ti = {Si,Qi} is sampled. Given the feature extractor fθ (·), the prototype for each class k can be computed as

where ∣∣Sk i ∣∣ denotes the number of support samples of the kth class. Then, given a new sample xq fromQi, the classifier outputs the normalized classification score for each class k

where simω (·, ·) is a similarity function parameterized by ω. In CSS, we consider the following similarity function:

where Fω (·) is a single layer neural network parameterized by ω and λ is the inverse temperature parameter. Then CSS can update θ and ω according to the following classification loss:

3.4 The Self-Supervised Training Stage

The core idea of the self-supervised training stage is to leverage the knowledge learned from the supervised pre-training stage to reduce representation bias and improve the learning of self-supervised tasks. For the above consideration, we propose the conditional self-supervision model, which consists of a self-supervised module and a conditional module. More details are illustrated in Figure 1(b).

In this stage, CSS trains the conditional self-supervision model by using all the data (removing labels) in Dtr without auxiliary unlabeled data. CSS trains a self-supervised feature extractor from scratch, rather than fine-tuning the feature extractor trained in the pre-training stage, which enables self-supervision to learn a diverse feature representation for few-shot learning.

Self-supervised module. We consider SimSiam [Chen and He, 2020] as a self-supervised task for simplicity and flexibility, other self-supervised methods are equally acceptable. Given a input image x, two augmented views x1 and x2 are generated according to random augment methods. The two views are processed by an encoder network consisting of the self-supervised feature extractor gξ (·) and a projection MLP head σ. A prediction MLP head, denoted as δ, encode the output of one view and matches it to the other view. Representing the two output vectors as p1 ≜ δ (σ (gξ (x1))) and z2 ≜ σ (gξ (x2)), the negative cosine similarity of two embedding vectors is obtained:

where ‖·‖2 is l2-norm. Besides, a stop-gradient [Grill et al., 2020] operation is implemented on z2. Similarly, switching these two views and obtaining the distance between the two vectors, the symmetrized loss can be defined as:

Conditional module. CSS takes the features learned in the pre-training stage as a prior condition to optimize the feature manifold learned in the self-supervised module. CSS minimizes the negative cosine similarity between fθ(x) and gξ(x) as the condition of self-supervised training:

Then, the final loss function combining condition loss with self-supervised loss can be obtained by:

where γ is a positive constant trading off the importance of the self-supervised and the conditional terms of the loss.

3.5 The Meta-Training Stage

In the meta-training stage, shown in Figure 1(c), CSS distills the knowledge learned in the first two stages with a novel fusion distillation (FD) to improve the quality of the representation. Specifically, the knowledge extracted from fθ (·) and gξ (·) is adopted to augment the representations of samples with graph convolution network (GCN) [Kipf and Welling, 2017]. In particular, for the sample xi, CSS first calculates its two corresponding embedding vectors fθ (xi) and gξ (xi), and then uses an augmentation operation zi = C (fθ (xi) , gξ (xi)) (e.g. concatenation) to connect them. By calculating zi of different samples, we can obtain the corresponding feature matrix Z = [z1, · · · , zn]T of these samples, where n is the number of samples. After that, we can calculate the cosine similarity between two samples and generate a graph S, where each vertex represents feature of a sample

where Si,j represents feature similarity between the i-th and j-th sample. Then CSS normalizes the graph matrix to obtain an adjacency matrix:

where D is the degree diagonal matrix and Dii = ∑j Si,j. Inspired by [Li and Liu, 2020], CSS takes E as the Laplacian matrix in GCN to aggregate information among vertices. Then for the sample xi, the embedding vector of xi with FD method can be obtained by

where I is the identity matrix, ν is the number of times for aggregating feature, α is a trade-off parameter to trade off the representation of neighbors and itself one. (αI +E)νi,j is the j-th element of the i-th row of (αI +E)ν .

After that, for a support set Si and a query sample xq, the classification score for class k can be obtained:

where

is the prototype of class k. Besides, the classification score for class k without FD can also be caculated:

where

Then the final loss is given by:

where η is a positive constant trading off the importance of the first and the second terms of the loss.

When predicting, CSS uses hϕ as the feature extractor to extract the features of samples. Then CSS calculates the average feature of the support set samples corresponding to each class as a prototype of the class. Then the similarity between the query sample and the prototype of each class is calculated by the similarity function simω (·, ·). Finally, the class with the highest similarity score is obtained as the classification result.

4 Experiments

In this section, the effectiveness of CSS is verified by various experiments. Three standard few-shot classification datasets (CIFAR-FS [Bertinetto et al., 2019], CUB-200 [Wah et al., 2011], mini-ImageNet [Vinyals et al., 2016]) are selected to compare the performance of our approach with previous fewshot learning methods. All the computations are performed on a GPU server with NVIDIA TITAN RTX, Intel Core i78700 CPU 3.20GHz processor and 32 GB memory.

4.1 Implementation Details

For the fair comparison, we implement all methods by PyTorch. Since [Chen et al., 2019b] shows that the gap among different methods drastically reduces as the backbone gets deeper, all algorithms use the Conv-4-64 [Vinyals et al., 2016] as the backbone and the feature embedding dimension is set to 1600. In CSS, fθ, gξ and hϕ are implement by the Conv-4-64 backbone, and the output dimension of Fω is 2048. The projection MLP head σ is a composite function, which is composed of Fω and a multi-layer neural network with {1600, 2048, 2048} units and batch normalization in each layer. The prediction MLP head δ is parameterized by a three-layer neural network with 512 hidden units and batch normalization in the hidden layer. All hidden layers use ReLU function [Glorot et al., 2011] as the activation function.

4.2 Comparison with PriorWork

In order to verify the effectiveness of CSS, several competitive and state-of-the-art fully supervised few-shot classification methods are compared, including Baseline [Chen et al., 2019b], Baseline++ [Chen et al., 2019b], MAML [Finn et al., 2017], Matching networks [Vinyals et al., 2016], Prototypical networks [Snell et al., 2017], Relation networks [Sung et al., 2018], FEAT [Ye et al., 2020] and Deep Kernel Transfer (DKT) [Patacchiola et al., 2020]. Besides, we also compare the performance of CSS with several state-of-theart self-supervision-based few-shot learning methods. ProtoTransfer [Medina et al., 2020] pre-trains a representation on self-supervision and adapts it to original few-shot classification tasks. CC+rot, PN+rot, SLA and ProtoNet+rot [Gidaris et al., 2019; Lee et al., 2020; Su et al., 2020] are multi-task few-shot learning and its varient methods.

We perform experiments to illustrate the average classification accuracies over 600 test episodes with 95% confidence intervals of different methods on CIFAR-FS, CUB-200 and mini-ImageNet datasets under 5-way 5-shot and 1-shot settings respectively. For each setting, the best result is highlighted in bold. CSS (pre-training), CSS(SSL-training) and CSS(meta-training) compose the 3-stage training pipeline proposed by us, which represent the classification model of CSS using fθ, gξ and hϕ as feature extractors respectively.

From Tables 1, 2 and 3, we can find that in most of the few-shot classification settings, CSS already achieves almost the same performance as state-of-the-art methods only through the pre-training stage and self-supervised training stage, while the meta-learning stage can further improve the accuracy of our model. In all cases, after the meta-training finishes, we achieve state-of-the-art results surpassing previous methods with a significant margin. For instance, on CIFAR-FS, CUB-200, and mini-ImageNet datasets, CSS has around 6%, 7% and 4% performance improvements compared with the vanilla prototypical networks under the 5-shot setting, while it has 13%, 15% and 6% performance improvements under the 1-shot setting. At the same time, in all settings, our approach outperforms around 2% to 5% higher than all previous leading methods.

4.3 Ablation Study

Now we explore the importance of the condition module in self-supervised learning and the effects of different stages. We design five cases respectively to investigate the performance when applying different combinations of stages. Then, we test the performance of these methods in the standard fewshot classification settings. To be brief, we use S1, S2 and S3 to represent CSS(pre-training), CSS(SSL-training) and CSS(meta-training) respectively.

• SSL: Discard S1 and S3 and replace S2 with a vanilla self-supervised learning without a conditional module.
• SL+SSL+FD: Change S2 and abandon the conditional module in our self-supervised learning method.
• SL (S1): Only execute supervised learning, i.e., using fθ in CSS as the feature extractor for classification.
• CSS (S1+S2): execute the first two stages without S3, i.e., using gξ in CSS as the feature extractor.
• CSS (S1+S2+S3): Execute the whole three stages of CSS, using hϕ as the feature extractor for classification.

The classification performance in different cases are shown in Figures 2 and 3. From Figures 2 and 3, we find that the performance of vanilla self-supervised learning (SSL) degrades significantly compared with other settings, which proves it difficult to study an effective representation in the absence of supervised learning with limited samples. The combination of supervised learning, self-supervised learning and fusion distillation (SL+SSL+FD) can improve the classification accuracy compared with SSL. However, the performance of SL+SSL+FD almost can not surpass the supervised learning model (SL) and the negative impact of SSL with limited samples still continues. In addition, by comparing the results of CSS(S1+S2) with SSL and SL, we find that the conditional module can significantly improve the performance of self-supervised learning, which can achieve comparable results with the supervised model (SL). Furthermore, CSS with entire process can achieve an obvious performance improvement and surpass other cases with large margin. The experiments illustrate that the condition module plays a crucial role in self-supervision and the effective feature fusion can further improve the model performance.

5 Conclusion

In this work, we propose conditional self-supervised learning (CSS) with a 3-stage training pipeline: the supervised pre-training stage, the self-supervised training stage and the meta-training stage, and each training stage is conducive to the improvements of the model performance. For the selfsupervised training stage, CSS leverages the supervised information learned by the pre-training stage to guide the learning of self-supervision, thus more resilient to low data regimes. For the meta-train stage, CSS utilizes a novel fusion distillation (FD) to integrate the information from the first two stages into a unified distribution so as to enrich and broaden the original representation. Detailed experiments reveal that our approach indeed leads to significant performance improvements over recent approaches and achieves state-of-the-art results.

Acknowledgments

This work was supported by National Natural Science Foundation of China (Grant No.62076062) and National Key R&D Program of China (Grant No.2017YFB1002801). Furthermore, the work was also supported by Collaborative Innovation Center of Wireless Communications Technology.