Better Embedding and More Shots for Few-shot Learning

1 Introduction

The outstanding successes of deep learning are partly accredited to the sheer amount of trainable samples. With the rapid development of deep learning, few-shot learning conceives a more challenging scenario, where only scarce labeled samples exist in each novel class. Under a standard few-shot learning setting, the embedding network is trained on substantial labeled data that sampled from base classes, i.e., no intersection with novel classes. Further, the model can be fine-tuned on the few-shot labeled data of novel classes. Recent researchs [Tian et al., 2020] [Hou and Sato, 2021] prove that embedding is the most vital aspect of fewshot image classification, which is also the focus of this paper. The embedding network training paradigm contains standard supervised pre-training [Chen et al., 2019a] and metalearning [Finn et al., 2017]. Either way, we debate that two problems remain to be solved.

∗ corresponding author

First, the embedding network is not well suited for novel classes as it is associated with base classes closely. While we can employ a few labeled samples to fine-tune the embedding in the novel-class space, there is a gamble of over-fitting. Additionally, the embedding network memorizes overmuch details specific to base classes, resulting in distorted novelclass embedding. Although the relevant art [Lee and Chung, 2021] is aimed at preventing over-fitting to the base classes via early-stage embedding reconstruction, the embedding is still not tailored for novel classes. With a more profound thought, we creatively reconstruct the topological relation instead of the instance similarity. It alleviates the excessive priority on the original feature space and pays more attention to global information. Furthermore, we put on the wings of information theory to make the reconstructed embedding more in line with the novel-class space. Nonetheless, we still face a tough nut, which leads us to the second problem.

Second, increasing the number of shots is pivotal but intractable [Cao et al., 2019]. Explicit GANs-based data augmentation techniques [Zhang et al., 2018] [Li et al., 2020] occur as feasible solutions. Besides, embedding sampling approaches [Yang et al., 2020] [Chi et al., 2021] generate pseudo-labeled embedding in metric space. Unsatisfactorily, these remedies introduce much complexity. In this paper, we unprecedentedly introduce implicit data augmentation into few-shot learning, taking inspiration from ISDA [Wang et al., 2021]. We abstractly achieve infinite augmentation with negligible extra computational cost, where the only alteration reflects in the derived upper bound of the loss function. In summary, our contributions are:

• This paper argues that two respects are required for novel-class-specific embedding learning. First, how to properly reconstruct pre-trained embedding. Second, how to make the reconstructed embedding fit the novelclass space better. Together, these two aspects constitute our Better Embedding and More Shots (BEMS).
• This paper creatively considers topological reconstruction avoiding excessive memorizing, corresponding to the Better Embedding (BE). In addition, we make the

embedding more in accordance with the novel-class space through the derived upper bound of the loss function. This bound implicitly achieves infinite shots with negligible cost, corresponding to the More Shots (MS). To the best of our knowledge, it is the first work to introduce implicit data augmentation into the few-shot learning.

• Substantial evaluations and ablation studies prove the promising performance. We also demonstrate that our method is appropriate for both standard pre-trained and meta-learning embedded networks.

2 RelatedWork

Both the embedding network and downstream module can obtain better embedding. The cross-entropy-based pretrained methods are the most common for the embedding network. Other strategies, such as self-supervised learning [Chen et al., 2021] and mixup [Mangla et al., 2020], are available for the more general-purpose embedded feature. For the downstream module, one tends to improve the metric space of the embedding. BD-CSPN [Liu et al., 2020] rectifies the prototype based on intra-class and inter-class biases. S2M2R [Mangla et al., 2020] adopts the manifold mixup for robust general-purpose representation. This paper focuses on the downstreammodule. We reveal that proper reconstruction is beneficial for better embedding.

Most methods explicitly accomplish more shots augmentation in sample or embedding space. MetaGAN [Zhang et al., 2018] generates non-perfect samples to help the classifier identify much tighter decision boundaries based on generative adversarial networks. Similarly, AFHN [Li et al., 2020] takes the few labeled samples as the conditional context to synthesize fake features. MVT [Park et al., 2020] generates virtual embedding to boost the target space. However, it is still an explicit sampling process needing additional regularization. Instead of synthesizing image instances, TriNet [Chen et al., 2019b] proposes to synthesize embedding directly. Distribution Calibration [Yang et al., 2020] and Learning2Capture [Chi et al., 2021] generate pseudo-labeled embedding based on the similarity relation. All of these methods introduce additional complex modules, while the modification in our method is only reflected in the loss function, which is more straightforward and more efficient.

3 Method

3.1 Notation

The few-shot problem is represented by the data in novel class Cn. With xi as the embedded D-dimension representation and yi as its label, the consuetudinary naming, the Nway K-shot task sampled from Cn denotes the support set S = {xi, yi}N×K i=1 , with N classes and each has K labeled samples. Corresponding with the S, the unlabeled query set is defined as Q = {xi, yi}N×K+T i=N×K+1 , where T means the volume. In this paper, we utilize the vector and matrix forms flexibly. For example, we also use (X ∈ RT×D , Y ∈ RT×N) to represent data pairs in Q. To obtain embedding, an embedding network fΦ(·) is trained on a large-scale labeled dataset belonging to the base class Cb, where Cb ∩ Cn = ∅. Typically, the embedding network can be standard pre-trained [Wang et al., 2019] or meta-learning model [Liu et al., 2019].

3.2 Overview

Figure 1 gives a general description of our method. Given the embedding xi and our topological reconstruction network gΘ(·), we implement Better Embedding (BE) based on topological reconstruction (Section 3.3), and More Shots (MS) by implicit augmentation (Section 3.4). Finally, the trained network provides Cn-specific embedding: zi = gΘ(xi).

3.3 Better Embedding

We introduce that two steps lead to better embedding. Suppose we propose to extract the embedding by fΦ(·). The first step is to maximize extraction of general representation and minimize extraction of Cb-specific information. As we have discussed in Section 2, one may utilize pre-training tricks [Chen et al., 2021] to make fΦ(·) more general or stop the embedding reconstruction in time [Lee and Chung, 2021] to prevent Cb-specific information. Instead of reconstructing the embedding directly, we rebuild the topological relation learned by fΦ(·). It makes the gΘ(·) pay more attention to the global information, avoiding over reconstruction. Formally, we have:

where Z = gΘ(X) and F denotes the Frobenius norm. We gingerly strike their trade-off balance for S and Q. Because S often acts as the precious supervised prototype [Liu et al., 2020], and Q provides more samples. Specifically, we obtain better embedding by:

where λ1 and λ2 balance the contribution between S and Q. LBE achieves the purpose of the first step, i.e., appropriate information extraction. Next, we move on to step two.

3.4 More Shots

In BE, the reconstructed embedding is still stuck in the Cb space. Ideally, the second step is to maximize the relevance between the reconstructed embedding and the Cn space. This inspiration can be depicted as:

where R(·, ·) represents the relevance. Nevertheless, scarce labeled data in Cn are not up to this challenge. Fortunately, we propose that distribution capture is a sound thought, where more or even infinite shots can be sampled. Once we have captured the class-conditional distribution, we can perform the following sampling:

where Σyi is the class-conditional covariance matrix, and α is a positive coefficient that controls the strength. Implementing such seemingly trivial sampling requires complex explicit methods, such as metric learning [Chi et al., 2021] and GANs [Li et al., 2020]. Nonetheless, this paper presents straightforward implicit data augmentation without introducing additional modules.

We primarily expand from Q. It is critical to secure accurately estimated distribution for the unlabeled ones, as otherwise, it will cast a catastrophic impact. Consequently, we embrace the prevalent prototypical classifier to acquire pseudo distribution:

where d(·, ·) can be any distance function. The prototype µj is calculated by the average of j-class S, mathematically, µj = Ez∼Sj [z]. According to these pseudo distributions, we can compute Σyi , where yi = argmin j∈N d(zi,µj). Correspondingly, we identify the meaningful semantic directions and proceed to sampling.

Assume we train the gΘ(·) with the weight matrix W and corresponding bias b. With M times sampling, we maximize R(Z, Cn) via minimizing the LQR :

Based on it, the reconstructed embedding leans toward the pseudo Cn distributions. Covetously, when M → ∞, we rewrite Equation (6) in an expectation form:

where vkj = wk - wj. The above formula, however, is hard to apply. We show the possibility to derive an easy-tocompute upper bound, achieving this unattainable goal.

Theorem 1. The upper bound of LQR is given by [Wang et al., 2021]:

Proof. According to Jensen's inequality E[logX] ≤ log E [X], we derive:

(8) According to the moment-generating function E [etX ] = etµ+ 12 σ2 t2 , where X ∼ N (µ, σ2 ), we have:

Thus, the upper bound of LQR is derived:

(10)

where α is the positive hyper-parameter. Finally, we derive the final loss function LQMS .

So far, we have implicitly carried out infinite sampling based on the unlabeled Q. In a similar spirit, we apply the upper bound of cross-entropy loss function for the labeled S.

Theorem 2. Similar to Theorem 1, we derive the upper bound of the cross-entropy loss [Wang et al., 2021]. For zi ∈ S, we have:

Proof. Similar to the proof of Theorem 1, we have the following derivation based on Jensen's inequality and momentgenerating function.

3.5 Overall Objective Function

In the mass, we minimize the overall objective given by:

When K = 1, LSMS is reduced to LSR. In brief, the first two terms work for BE and the last two terms work forMS. They provide the Cn-specific embedding together.

4 Experiments

4.1 Experimental Settings

Datasets. We perform evaluations on three popular datasets. The miniImageNet [Vinyals et al., 2016] and tieredImageNet [Ren et al., 2018] are the subsets of ImageNet. CUB-200-2011 [Wah et al., 2011] is a fine-grained bird classification dataset. All images are resized to 84 × 84. More details are depicted in Table 1.

Evaluation Protocol. We report the average accuracy and 95% confidence interval on random sampled 600 tasks. Each task contains 15 query samples of each class, i.e., T=15×N. For reported results, the number in bold means the best performance and underline means the second. All our reimplemented algorithms, as indicated by the superscript† , adopt the unified trained embedding network for fairer comparisons.

4.2 Implementation Details

Embedding Networks. We train our embedding networks using the standard cross-entropy loss on Cb based on three backbones: ConvNet, ResNet-18, andWideResNet. ConvNet comprises four blocks. Each block contains a 64-filter 3 × 3 convolution, a batch normalization layer, a ReLU activation and a 2 × 2 max-pooling layer. ResNet-18 we used is a standard 18-layer residual network that removed the first two down-sampling layers. WideResNet is the wide residual network with 28 convolutional layers and 10 widening factors. For WideResNet training, we set the label-smoothing parameter as 0.1. We use SGD optimizer and 128 mini-batch sizes. Referring to [Ziko et al., 2020], we use early stopping by prototypical classifier on the validation set. For ConvNet and ResNet-18 training, we adopt the source code provided by [Wang et al., 2019]. In addition, we extend our applicability to meta-learning in Section 4.7. This section trains all re-implemented methods with Adam optimizer and an initial learning rate of 0.001. We cut the learning rate in half every 10,000 and 25,000 episodes for miniImageNet and tieredImageNet, respectively.

Reconstruction Training. We use 2-layer fully connected layers with ReLU function and a 0.5 dropout as our reconstruction module. The first layer reduces the dimension to half the input dimension, and the second layer restores the dimension. We conduct 200 training iterations. For optimizer, we use Adam with 0.001 learning rate and 0.01 weight decay.

Preprocessing. The output of the penultimate layer of the embedding network is extracted as the embedding feature. The centering and L2-normalization are used as preprocessing tools. For centering, we first calculate the average value of the embedded features in Cb: µ = Ex∼Cb [x]. Then x ← x - µ, where x ∈ Cn. For L2-normalization, we compute: x← x ‖x‖2 .

Hyper-parameters. The hyper-parameters are tuned by validation set. We set α = t× iter ITER , where iter and ITER are the current and total iterations, respectively. We tune t in {0.01, 0.1, 0.25, 1, 10}. For λ1 and λ2, we tune them in {0.01, 0.02, 0.1, 0.5, 1}.

4.3 Improvement by our Method

We utilize the prototypical classifier as our baseline to investigate the improvement by our method. Concretely, it makes inference of Q by matching the nearest prototype, where Euclidean distance is adopted. Results of three backbones and two datasets are reported in Table 2. All backbones yield satisfactory results. The most remarkable performance is the 10%+ accuracy gain at most on 1-shot evaluation. Comparatively, 5-shot evaluation also gets a maximum 5.11% improvement. In addition, the performance gap between 1-shot and 5-shot results is significantly narrowed, which will be further proved in Section 4.8.

4.4 Comparison with Relevant Methods

We also report the performance of relevant methods in Table 2. (1) In comparison with the explicit data augmentation methods: Distribution Calibration, Learning2Capture, TriNet, MVT, and AFHN, our BEMS perform a significant improvement in a lighter augmentation manner. On all datasets and backbones, we averagely lead by 6.81% and 3.01% on 1-shot and 5-shot scenarios, respectively. (2) In comparison with the embedding adaptation methods: BDCSPN and ESFR, we surpass them by 2.18% and 0.97% averagely on 1-shot and 5-shot scenarios, respectively, because our approach pays more attention to the topological structure and alleviates the over-fitting risk on Cb. ESFR avoids the over-memorizing of Cb, which is similar to our motivation. Differently, our MS further brings the reconstructed embedding closer to Cn. (3) In comparison with the information theory based AWGIM, our BEMS leads by as much as 12.26% on miniImageNet and 12.74% on tieredImageNet. AWGIMmaximizes the mutual information between generated weights and S as well as Q to retain information within the task, while our utilization is essential for better embedding.

4.5 Cross-domain Evaluation

We conduct further evaluations on CUB and cross-domain scenario, i.e., miniImageNet →CUB. Results are shown in Table 3. We also compare the most relevant algorithms, and our method still maintains the top performance. In cross-domain problems, our BEMS is second only to Learning2Capture, which uses explicit data augmentation.

4.6 Significance of Better Embedding andMore Shots

First, we take our core components apart for ablation in Figure 2(Left). In this evaluation, we fix t = 10. We analyze the following alterations:

• Baseline denotes we directly make predictions through pre-trained embedding and prototypical classifier.

Baseline +  R + +ESFR + + +

• +LSR denotes we only optimize gΘ(·) by LSR;
• +BE and +MS denote we use the LBE and LMS terms

in Equation (12), respectively;

• +ESFR denotes we take it as a contrast to BE;
• +BES denotes the loss function without MS: LBES = λ1LSBE + λ2LQBE + LSR + LQR ;
• +LBEMS denotes the complete version.

As we have mentioned in Section 4.3, our BEMS narrows the accuracy gap between different shot numbers, which is more intuitive in the line chart. The proposed BE substantially reconstructs better embedding. The proposedMS implicitly provides infinite data volume, blurring the shot disparity and boosting the performance. Taking ESFR as a reference for reconstruction problems, BE performs better on the 1-shot setting. In addition, ESFR further supports ourMS perspective that embedding should get close to Cn space on the basis of Cb information reconstructing. Although MS shows a slightly negative effect on the 5-shot scenario, it improves significantly on the 1-shot setting. Finally, the BEMS consistently achieves the highest accuracy.

Second, we report the sensitivity of t in Figure 2(Right). In this evaluation, we fix the λ1 = λ2 = 0.1 and λ1 = 0.01, λ2 = 0.02. We observe that MS works positively in most t values. The overall trends show that the parameter sensitivity of MS is affected by BE, which is reflected in the different effects of t on performance under different λ1 and λ2. This is because BE and MS have a latent antagonistic relationship. MS makes embedding tend to Cn space, which interferes with the reconstruction in BE to some extent.

4.7 Meta-learning Embedding

As an extension study, we explore the enhancement effect of BEMS on meta-learning-based embedding. We select two representative meta-learning methods, ProtoNet [Snell et al., 2017] and TPN [Liu et al., 2019], for evaluations. Concretely, we use BEMS to boost the test phase of two algorithms, making the meta-knowledge transfer to the Cn better. Results in Table 4 strongly prove that our method also significantly improves meta-learning embedding.

4.8 Ablation Study

Based on the overall loss function, Equation (12), we conduct more detailed ablations in Table 5. We fix λ1 = λ2 = 0.1 and t = 10 in this evaluation. (1) Considering the relevant methods [Park et al., 2020] [Lee and Chung, 2021], we also try the feature reconstruction. Accordingly, FeatRec denotes we change our topological reconstruction to ‖X - gΘ(X)‖2F . The results show feature reconstruction plays a negative role in our approach, except for the 1-shot miniImageNet. (2) w/o denotes we drop the corresponding component in the loss function. Six results report the S and Q ablations. We observe that the reconstructions of both S and Q are indispensable, especially for S. The MS of S and Q have a steady performance improvement. When the S component is entirely absent, there is a certain negative impact on performance. When we drop all Q components, the performance degradation is more significant.

Parameter Sensitivity Analysis. We conduct the parameter sensitivity analysis in Figure 3. We observe higher λ1 and λ2 lead to better performances, where this phenomenon is more sensitive to λ2. Sensitivity analysis shows that our method is stable and effective when the LBE occupies a high proportion in the overall loss function.

Visualization. We use T-SNE to visualize the original and reconstructed embedding in Figure 4. The original embedding is distorted, which is reflected in poor inter-class and intra-class relations. Exhilaratingly, our BEMS solves the problem well, i.e., sharper boundaries. Notably, the unlabeled Q is brilliantly reconstructed under the guidance of the implicit augmentation.

5 Conclusion

In this paper, we first reconstruct the topological relation of embedding, and then pioneer implicit more shots augmentation. We show significant improvement in different evaluations. In ablation studies, we discuss why BEMS works. Scarce labeled data is the fundamental problem of fewshot learning. Our approach implicitly alleviates this problem while introducing only negligible complexity, which is very promising.

Љৗޕٖъ

Acknowledgments

This work is supported by Shanghai Science and Technology Program "Distributed and generative few-shot algorithm and theory research" under Grant No. 20511100600, Natural Science Foundation of China under Grant No. 62076094, and Shanghai Science and Technology Program "Federated based cross-domain and cross-task incremental learning" under Grant No. 21511100800.