Few-shot Visual Learning with Contextual Memory and Fine-grained Calibration

1 Introduction

Deep learning methods have shown the powerful learning capability in the past decade. The standard deep learning models [Ji et al., 2013; Zagoruyko and Komodakis, 2016] mainly contain millions of parameters and heavily rely on the huge amount of training data. Largely different from the deep learning model, the human cognition system exhibits remarkable abilities to infer the novel concepts effortlessly from only one or a few examples and reliably recognize them later on. To acquire the similar strong generalization ability, few-shot learning has been introduced to learn a model that can be readily adapted to new unseen classes (concepts)

∗Corresponding Author

by accessing only one or few examples. A variety of fewshot learning methods have been proposed recently, which can be roughly divided into optimization-based, generationbased, and metric-based methods.

Optimization-based methods train a desired meta learner over a variety of learning tasks and optimize it for the best performance on a distribution of tasks, including potentially unseen tasks. To accomplish this task, usually there needs an across-task meta-learner that identifies how to update the parameters of the learner's model. [Rusu et al., 2018] introduced a data-dependent meta-learning approach which learns a low-dimensional latent generative representation of model parameters, and performs gradient-based adaptation in this space. In [Sun et al., 2019], a novel meta-transfer learning method is proposed which combines the advantages of meta learning and transfer learning to transfer large scale pretrained DNN weights for solving few-shot learning tasks.

Generation-based methods attempt to augment few-shot data with a generative meta-learner or learn to predict classification weights for novel classes. [Qiao et al., 2018] proposed a novel approach that can adapt a pre-trained neural network to novel categories by directly predicting the parameters from the activations without training. In [Gidaris and Komodakis, 2019], a Denoising Autoencoder network is used to refine a set of initial classification weights to make them more discriminative with respect to the classification task at hand.

Metric-based methods have achieved considerable success by learning to compare the support and query samples in a shared feature space. The early study of [Vinyals et al., 2016] introduced the episodic training mechanism into fewshot learning and utilized a bidirectional LSTM to encode each support sample in the context of the whole support set, and matched the query sample to the support sample through an attention mechanism. The following typical methods such as [Sung et al., 2018] attempted to embed the samples by simply summing each support class in an element-wise manner.

Most of the previous few-shot learning approaches concentrated on the abstract global information for each sample. This is consistent with the human cognition system, which usually first makes a coarse recognition from the global perspective, e.g., shape, size, structure, etc. However, in practice when the object is too difficult to be distinguished from the global perspective, humans will further resort to the detailed local features. This recognition scheme follows the coarse- Select Indistinguishable Classes to-fine theory of visual perception, inspired by the Gestalt dictum that the whole is prior to the parts (the whole arises before the parts). To compensate for the weakness of the previous methods, recently a few studies [Li et al., 2019b; Li et al., 2019c] have been proposed to extract the subtle difference from the local perspective, and achieved satisfying performance on fine-grained dataset. Unfortunately, only focusing on either global or local characteristics, these methods are still far from strong generalization abilities over datasets with various classes.

Figure 1: The IPN framework.

To address this problem, in this paper we propose an inverted pyramid network (IPN) to intimate the human's coarse-to-fine cognition paradigm, inspired by the reversehierarchy theory [Ahissar and Hochstein, 2004]. Reversehierarchy theory, also known as inverse pyramid theory, suggests that the magnocellular stream provides the fast "coarse" initial sweep, while slow parvocellular signals representing "fine" analysis are processed in a later time window. Following this paradigm, the proposed IPN consists of two consecutive stages, namely global stage and local stage. At the global stage, we propose a class-specific contextual model with a memory mechanism (CCMNet) to learn the discriminative global support-query relation embeddings. Specifically, CCMNet sequentially processes the query sample and one support sample of a specific class at each time step, and learns the discriminative relation embedding between the support and query sample based on the contextual information. Besides, the information flow of the classical GRU is further modified to preserve the long-term dependencies using fewer parameters, enabling the strong sensitivity to the contextual information. At the local stage, to compensate for the weakness of the globally predicted query-to-class similarity, the fine-grained calibration can be further appended by simply comparing the query with its nearest patches, and thus targets more precise query-to-class similarity evaluation.

To the best of our knowledge, IPN is the first work that simultaneously integrates both global and local characteristics in few-shot learning, approximately imitating the human cognition mechanism. Extensive experiments conducted on two commonly-used few-shot datasets miniImageNet and tieredImageNet further verify the superiority of our IPN model. Especially, even using our CCMNet alone can achieve 66% 1-shot accuracy and nearly 83% 5-shot accuracy on miniImageNet, outperforming most state-of-the-art approaches under the same setting. Moreover, by further applying the fine-grained calibration, our two-stage framework can consistently obtain accuracy gains (up to 5.6%), on different datasets and under different few-shot settings.

2 The Inverted Pyramid Network

We develop a novel two-stage few-shot learning architecture named Inverse Pyramid Network (IPN), inspired by the coarse-to-fine theory of human visual perception, meaning that the whole arises before the parts. Therefore, the proposed IPN consists of two consecutive stages, namely global stage and local stage. At the global stage, a class-sensitive contextual memory network is proposed to progressively capture the global relations such as the similarities of shape, structure etc., between support samples from a class and query in an online setting. After that, fine-grained calibration will be conducted to further compare the local discriminative parts for indistinguishable classes.

Next, we first introduce the preliminary of the few-shot setting, then present the Class-sensitive Contextual Memory Network at the global stage and the fine-grained calibration at the local stage, and finally demonstrate the inference process of the proposed model on novel classes.

2.1 Preliminary

Let S denote a support set, which contains N different image classes (C1, . . . , CN) and K (K is small, e.g., K = 5) labeled samples per class. Given a query set Q, few-shot learning aims to classify each unlabeled sample in Q according to the set S. This setting is also called N-way K-shot classification. We adopt episodic training which is commonly em-

⋯

Figure 2: Class-sensitive Contextual Memory Network. The red arrow represents the skip link.

ployed in the literature as an effective approach to learn the transferable knowledge from a relatively large labeled training dataset with a set of classes Ctrain which has a disjoint class label space with the test dataset with novel classes Ctest, namely Ctrain ∩ Ctest = ∅.

More concretely, in episodic training, a small subset of N classes are sampled from Ctrain to construct an N-way Kshot problem as follows: a task Γ contains a support set S and a query set Q, where S = {x(1) k }Kk=1 ∪ ... ∪ {x(N) k }Kk=1 and Q = {(ˆxq, ˆ yq)}Tq=1 . Here, x(n) k denotes the k-th sample of class Cn in the support set. T is the number of query samples, and ˆxq, ˆ yq ∈ {C1, ..., CN} are the q-th query data and its ground truth label, respectively. In each episode, the model is trained to minimize the loss of its predictions of Q through learning the labeled support set S.

2.2 Global Stage: Contextual Memory

Given a sample x of a task Γ, the embedding network E is first utilized to extract the global feature representations g = E(x) ∈ R1×D . Based on the feature representations, we propose the class-sensitive contextual memory network (CCMNet) to capture the query-to-class relations, by fully exploring the context of a class. Initializing the hidden state with query features, CCMNet takes a support sample from a specific class as input and captures the support-query relation at each time step. Through learning relation embeddings sample by sample in a class-sensitive contextual environment, CCMNet enjoys a better understanding about the query-to-class similarity and thus achieves comparatively accurate classification performance.

Class-sensitive Contextual Memory Network

For the proposed CCMNet, we adopt the classical GRU [Cho et al., 2014] and modify its information flow to selectively absorb the information from past experience. GRU is a recurrent gating mechanism, where the reset gate mainly determines how much of the past information needs to be removed, and the update gate determines how much of the past information (from previous time steps) needs to be memorized and passed to the future.

We further modify the information flow of classical GRU to fit the task of relation learning from the contextual information. For one thing, we initialize the hidden state with the query representations, and feed the global feature representation from the same class into the memory network at every time step. Thus, we capture the relation between the query sample and support sample at each step and make the prediction given the class-sensitive contextual environment. For another, we add a skip link in each time step to further consider experience from the previous two time steps, in avoid of catastrophic forgetting and to better learn the contextual information across the class.

Figure 2 shows the whole structure of CCMNet with the modified GRU module. Specifically, when learning the relations between the query sample ˆxq and the n-th class, we initialize the hidden state h(n) 0 = ˆ gq. And in the time step k, we feed the k-th sample denoted as g(n) k from class Cn into the CCMNet, and the hidden state updating in our class-sensitive GRU is conducted as follows:

where h(n) k is the updated hidden state, with h(n) k-2 a skip link to h(n) k-1 , considering previous two time steps to involve more class-sensitive contextual information. When k = 1, we denote h(n) k-2 = ˆgq. Wz, Uz, Wr, Ur, Wh, and Uh are all learnable parameters, σ and φ are the sigmoid activation function and the tanh activation function, respectively. z(n) k and r(n) k represent the update gate and the reset gate.

The entire iterations explore the relations between ˆ gq and the category Cn by traversing all samples of Cn in the support set. Due to the special gating update mechanism, the hidden state h(n) k after iterative update retains the common features of the query sample and category Cn, while irrelevant interference information is forgotten. Therefore, h(n) k can be used as a relation embedding to measure the query-to-class similarity. Besides, through the skip link, not only the gradient vanishing problem in the back-propagation procedure is alleviated, but also effectively mitigates the occurrence of catastrophic forgetting of earlier data and transmits more information to the current step.

To further learn the contextual information and eliminate the influence of sequence order, we adopt the bidirectional mechanism. We concatenate the output hidden states -→h (n) k , ←- h (n) K-k+1 from two opposite directions together as the final relation embedding:

where [·, ·] denotes the concatenation operation. As a result, for the n-th class, we can obtain a set of relation embeddings {h(n) k }Kk=1 .

Learning at the Global Stage

For each relation embedding, we could learn a similarity score s(n) qk = Ψ(h(n) k ) where Ψ is a similarity measure, and s(n) qk ∈ [0, 1]. Intuitively, the larger the score is, the higher the probability that they belong to the same category is. Thus, for each training episode, we could compare the similarity score with the ground truth label and compute the loss function:

where δqk is defined as:

2.3 Local Stage: Fine-grained Calibration

As mentioned above, it is hard for the model to distinguish similar categories given the compact global representations. Therefore, we need to compare the different local details to calibrate the uncertain global prediction. Under such consideration, we reuse the backbone network E to further gain a better understanding of the input samples and mine the distinguishable characteristics from local perspective.

Here we adopt the method presented in DN4 [Li et al., 2019b]. Given a sample x, we view the features output by the last convolutional layer of E, as a set of local patch features [p1, · · · , pM] where pj is the j-th local patch feature. For each local feature pj of query sample, we find its L-nearest neighbors p′lj |Ll=1 in the local feature space of support samples from the same class Cn. Then we calculate the cosine similarity between pj and each p′ j , and sum the M ×L similarities as the query-to-class similarity:

Note that the fine-grained calibration process is nonparametric and computation effective. Following the reversehierarchy cognition paradigm, it is very simple and flexible to be appended to the global stage on demand, forming an effective two-stage coarse-to-fine few-shot learning framework, i.e., our Inverse Pyramid Network (IPN) model.

2.4 Inference on Novel Class

In the testing stage, given a query sample ˆxq, we first use the CCMNet to generate relation embeddings and compute the similarity scores for support-query pairs. Then the global query-to-class similarity can be expressed as:

In practice some classes of the task are too similar and indistinguishable from global perspective, and thus the global query-to-class similarities are very close. In this case, we should further conduct the fine-grained calibration. Specifically, assuming that Ci and Cj are the two categories with the highest query-to-class similarities, we calculate the prediction reliability τ of the task as:

We set a reliability threshold τ0 and compare it with the prediction reliability τ. If τ ≥ τ0, we consider Ci as the final prediction of the query sample directly. Otherwise we resort to the fine-grained calibration to further obtain the more precise query-to-class similarity through which we make the final prediction.

3 Experiments

In this section, we evaluate our IPN with state-of-the-art fewshot approaches on widely used datasets.

3.1 Experimental Settings

Datasets

We employ the widely used datasets in prior studies, including miniImageNet dataset [Vinyals et al., 2016] and tieredImageNet dataset [Ren et al., 2018]. The miniImageNet dataset consists of 100 classes, each of which contains 600 images of size 84 × 84, while the tieredImageNet contains 608 classes with 77915 images in total. The classes of tieredImageNet are grouped into 34 higher-level nodes based on WordNet hierarchy [Deng et al., 2009], and is further partitioned into disjoint sets of training, testing, and validation nodes, ensuring a distinct distance between training and testing classes thus making the classification more challenging. For both datasets, we adopt the common splits as previous work.

Model Architectures

We use the recently common-used feature embedding architecture WRN-28 [Zagoruyko and Komodakis, 2016] as backbone. WRN-28 whose output is a 640-dimensional feature vector is a 28-layer wide residual network with width factor 10. We pre-train the WRN-28 network by optimizing the accuracy of the multi-classes classification on the whole training set of miniImageNet or tieredImageNet, and then freeze the parameters during the training phase. The similarity measure Ψ is implemented as Multi-Layer Perceptions (MLPs) consisting of 3 fully-connected layers.

Implementation Details

Standard data augmentations including random crop, leftright flip, and color jitter are applied in the training stage. The mini-batch size for all experiments is 20. The number of training iterations on miniImageNet and tieredImageNet are 100K and 200K. We use the validation set to select the training episodes with the best accuracy. We use Adam optimizer with an initial learning rate of 0.001, and reduce the learning rate by half every 15K and 30K iterations, respectively on miniImageNet and tieredImageNet. The weight decay is set to 10-6 . When conducting finegrained calibration at local stage, the prediction reliability threshold τ0 is set to 1.5, and the number of nearest neighbors L is set to 3. As presented in [Kim et al., 2019; Liu et al., 2018], most few-shot approaches adopted two kinds of transductive inference methods to improve the classification performance. In our CCMNet, we simply concatenate the query features together as the initialization of the hidden state and learn the relation embeddings simultaneously.

Evaluation Protocols

On both datasets, we conduct 5-way 1-shot and 5-shot experiments which are standard few-shot learning settings. For evaluation, each episode was formed by randomly sampling 1 query for each of 5 classes. We report the mean accuracy (%) of 10000 randomly generated episodes as well as the 95% intervals on test set.

3.2 Comparison with State-of-the-Art

We first investigate the performance of our model, compared to state-of-the-art few-shot approaches, respectively on miniImageNet and tieredImageNet. These approaches are particularly divided into optimization-based, generationbased and metric-based. Table 1 and Table 2 list the fewshot classification accuracies of the 5-way 1-shot and 5-shot tasks along with the specifications of the backbone embedding models for feature extraction, where "Conv-4" indicates the 4-layer convolutional neural network. We use bold fonts for the two best results.

From Table 1, we can observe that as the shots increase, all the methods perform better, which is adhere to our intuition. Moreover, a deeper embedding network will lead to a better classification performance compared to methods equipped with Conv4, achieving above 70% accuracy on 5-shot setting. In most cases, our IPN model significantly outperforms others under the same experimental setting, achieving 67.42% and 83.98% accuracy on 5-way 1-shot and 5-shot setting, respectively. Though as presented in [Chen et al., 2019], as the backbone gets deeper, the gap among different methods drastically reduces, our IPN model can consistently gain nearly 5% improvements on 5-shot setting over the second best approach CTM, confirming the superiority of IPN mainly owning to the inverted pyramid paradigm. The similar trend can also be observed in Table 2. The proposed IPN shows comparable results with the state-of-the-arts, achieving 73.18% and 86.59% accuracy on 5-way 1-shot and 5-shot setting, respectively.

3.3 Ablation Study

Figure 3 demonstrates the effects of each component of the proposed IPN framework on miniImageNet. It respectively shows the few-shot classification results of the proposed IPN, IPN without the global stage (denoted local only), the proposed CCMNet, CCMNet without contextual information, and CCMNet without skip links. Since we directly adopt the DN4 model proposed in [Li et al., 2019b] to compare local fine-grained details at our local stage, we simply replace the backbone of DN4 model with ours and re-train the model under the same setting. As for removing contextual information from CCMNet, at each time step, we assign previous hidden state h(n) k-1 , h(n) k-2 with query features and thus capture the query-to-class relations without passing contextual messages.

As it could be seen from Figure 3, only considering global characteristics or local ones, the performance decline by nearly 1% and 4%, respectively. As shown in Table 1 and Table 2, comparing the performance of CCMNet and our full model, accuracy gains of fine-grained calibration is comparatively obvious on the more challenging tieredImageNet with various classes and disjoint higher-level semantic hierarchy, confirming that fine-grained calibration is effective and significative in real-world scenarios. We further investigate the effectiveness of each part of CCMNet. Without contextual information, the performance significantly drops by 10%. Without the skip link, it witnesses more than 2% decline. In contrast, with contextual information and skip link, the proposed CCMNet achieves 66.30% 1-shot accuracy and 82.89% 5-shot accuracy on miniImageNet, which outperforms most few-shot approaches. Thus, we can conclude that, with help of the CCMNet at global stage and fine-grained calibration at local stage, our IPN framework enjoys strong power to achieve the best performance compared to others.

Figure 4 shows t-SNE visualizations of relation embeddings for the proposed IPN. The model is trained under 10way 5-shot. Circles indicate the relation embeddings of a query sample and numerous support samples from 10 classes. Different colors denote different class labels of support samples. Intuitively, if two support samples are similar, they share the similar relation with the same query sample. As the Figure 4 depicted, there are obvious 10 clusters of relations. Each cluster is compact and separates from other clusters, proving that the proposed IPN model could learn the discriminative relation embeddings.

4 Conclusion

In this paper, we proposed a two-stage inverted pyramid network (IPN) for few-shot learning inspired by the inverted pyramid theory, which is the first work integrating both global and local characteristics in few-shot learning. At the global stage, the CCMNet is introduced to predict the query-to-class similarity from the global perspective. Then at the local stage, a fine-grained calibration is further appended to compensate for the weakness of the global prediction. Extensive experiments conducted on several widely-used datasets demonstrate that IPN outperforms other state-of-the-art few-shot approaches by a large margin.

Acknowledgments

This work was supported by National Natural Science Foundation of China (61872021, 61690202), and Beijing Nova Program of Science and Technology (Z191100001119050).