MemREIN: Rein the Domain Shift for Cross-Domain Few-Shot Learning

1 Introduction

In recent years, machine learning especially deep learning methods have made amazing achievements in the field of computer vision, image classification, semantic segmentation, etc. However, the promising performance heavily counts on the large amount of well-labeled training data, which provides comprehensive and diverse samples to cover all corner cases. Such a huge scale makes it difficult in real practice, thus leads to a new topic of few-shot learning [Wang et al., 2020]. Few-shot learning aims to enable models generalize to new categories (query instances) with only limited labeled samples (support instances) from each category.

Among existing few-shot learning methods, metric-based methods have attracted significant attention because of their effectiveness and intelligibility. In general, the core idea of this kind of methods is to make classifications based on the similarity between the query images and the support images via proposed similarity measures. It usually consists of two main components: (1) feature encoder and (2) metric function. Given a task with few labeled images (support set) and unlabeled images (query set), the visual features are firstly extracted via the feature encoder and then passed through the defined metric function to determine the categories of the query images. The underlying assumption is that both training and testing are from the same dataset, namely the same domain. While, when it comes to different domains, the generalization ability of the metric-based methods greatly decreases [Chen et al., 2019; Tseng et al., 2020]. However, such ability to generalize to unseen domains is of great importance in practice, e.g., expensive human annotation or timeconsuming data collection. As a result, considering the domain shift scenario within the few-shot learning has become an important yet challenging task.

Various unsupervised domain adaptation methods have been proposed [Yang et al., 2018; Xu et al., 2022] recently. These methods aim to minimize the domain gap either by learning domain-invariant representations via representation learning, projection learning, or adversarial strategies [Long et al., 2015; Kumar et al., 2018; Tzeng et al., 2017; Kundu et al., 2019]. However, these methods assume that the complete unlabeled samples from the target domain are accessible while training. We argue that this assumption may not hold in real situations, and it could leads to high computational cost in testing phase. Domain shift problem could be addressed by various domain generalization methods [Blanchard et al., 2011; Muandet et al., 2013]. However, these methods assume that the source and target domains share the same categories. In contrast, our goal is to recognize novel categories from the target domain with only a few (e.g., 1 or 5) of samples selected from novel categories.

As argued above, there are two main challenges in crossdomain few-shot learning task. (1) How to minimize the discrepancy between the source and target domain. (2) How to recognize novel/unseen classes with only limited samples.

To this end, we propose a novel MemREIN approach, which includes Memorized, Restitution, and Instance Normalization as crucial modules, to "rein" the domain shift level in few-shot scenario. The core idea of MemREIN is to enhance the generalization ability while still be able to balance the discrimination ability for subsequent classification. Specifically, on the training stage, we first present an instance normalization layer operating on features with respect to samples at the channel level. This operation aims to reserve spatial feature dependency and meanwhile remove the image-specific features, i.e., alleviate the discrepancy of these training samples. In this way, the generalization ability across different samples is enhanced. Then, the filtered out features are extracted from a residual structure. Normally, the filtered out features are considered as useless feature which could be discarded. However, we consider it still contains finegrained distinctive knowledge which could be "remembered" and "restituted". To this end, we manage to adaptively distill the long-term discriminative information from them via our proposed novel memorized approach. Then, such discriminative information is restituted to the above refined features to maintain the discrimination ability for subsequent classification. A novel reverse contrastive loss constraint in the restitution phase is designed to encourage the better separation of discriminative features and general features, which ensures the distillation process. Contributions of our work are as,

• A novel memorized and restitution strategy is proposed for discriminative information distillation. It is able to distill the long-term discriminative information from filtered out features to maintain the discrimination ability of original features for better classification.
• An instance normalization strategy is adopted to alleviate the the discrepancy across training samples, which reduces the sample-specific features and greatly enhances the overall generalization ability across features.
• A novel reverse contrastive loss is proposed to encourage the better separation of discriminative and general features, which is able to ensure the distillation process.

Our MemREIN approach is simple yet effective. It is a universal and method-agnostic approach that can be applied to various existing metric-based methods for enhancing their generalization ability to unseen domains. Extensive experiments demonstrate the effectiveness of MemREIN, which achieves consistent superior performance than existing stateof-the-art methods under the cross-domain setting.

2 RelatedWork

2.1 Cross-domain Few-shot Learning

Few-shot classification aims to recognize novel classes with a limited amount of labeled samples. Among these existing methods, metric-based methods have attracted significant attention and achieved promising performance. For instance, MatchingNet [Vinyals et al., 2016] utilizes cosine similarity with an attention Bi-LSTM for classification and ProtoNet [Snell et al., 2017] applies Euclidean distance for classification. RelationNet [Sung et al., 2018] uses convolutional neural networks and GNN [Satorras and Estrach, 2018] uses the graph convolutional framework as the metric function.

Although these methods are able to achieve promising performance, they always fail to generalize to unseen domains since the distributions among different domains have huge shifts. Recent work [Chen et al., 2019] reveals that the accuracy of existing few-shot learning methods degrades significantly under the cross-domain setting. The motivation of our work aims to enhance the generalization ability of metricbased few-shot learning methods so that these methods can better generalize to unseen domains.

Recently, promoted by the pioneer work [Chen et al., 2019], cross-domain few-shot learning problem has attracted many attentions. As an emerging task, work [Chen et al., 2019] carried out a broader study and introduced a new benchmark. Some methods [Tseng et al., 2020; Sun et al., 2021; Phoo and Hariharan, 2020; Zou et al., 2021; Islam et al., 2021; Liang et al., 2021; Fu et al., 2021; Wang and Deng, 2021] have been proposed recently and achieved promising performance under this benchmark, which greatly promoted the development. In our work, we propose a simple yet effective method from the perspective of feature level, which is a universal and method-agnostic approach.

3 Method

3.1 Preliminaries

In the few-shot classification problem, a task T is characterized as Nw way and Ns shot, which represents the number of categories and the number of labeled samples in each category. At each iteration, the metric-based few-shot learning method randomly samples Nw categories as a task T, and then constructs a support set S = {(Xs, Ys)} and a query set Q = {(Xq, Yq)}, where X and Y represent samples and labels respectively. These two sets are constructed by randomly selecting Ns and Nq samples for each of the Nw categories.

Once the data is prepared, the feature encoder E first extracts features of the samples from both support set S and query set Q. Then, the defined metric function M predicts the query samples Xq based on three parts: the label of support samples Ys, encoded query image E(Xq), and the encoded support images E(Xs), which is formulated as,

After all, the objective of the metric-based few-shot learning method is the classification loss of the samples in the query set, which is formulated as,

In this paper, we tackle the cross-domain few-shot classification problem. Given a set of few-shot classification tasks T = {T1, T2, ..., Tn} as a domain (dataset). At the training stage, given N accessible domains {T seen 1 , T seen 2 , ..., T seen N }, we aim to learn a metric-based few-shot learning model with these seen domains, then the model can generalize to an unseen domainT unseen .

3.2 MemREINMethod

The core idea of our MemREIN method is to enhance the generalization ability, including the ability to balance the discrimination of metric-based few-shot learning methods, and achieve promising performance on arbitrary unseen domains. The overall framework is illustrated in Figure 1.

Instance Normalization

As argued above, images with the same category from different domains have large discrepancies in many aspects e.g. , image style, color, quality. Generally speaking, the discrepancy between the source domain and the target domain hinders the generalization ability of the model to some extent.

To this end, we reduce the discrepancy cross samples by instance normalization in our proposed MemREIN method. Given the input feature map F ∈ Rc×h×w and the output feature map ˜ F ∈ Rc×h×w , where c, h, and w represent the number of channel, height, and width, respectively. The instance normalization is formulated as,

where µ(·) and σ(·) denote the mean and standard deviation calculated at the channel level for each sample, γ ∈ Rc and β ∈ Rc are two trainable parameters. Instance normalization was originally used in style transfer [Dumoulin et al., 2016], which is helpful to enhance the generalization ability by reducing the feature dissimilarity. It can remove instance/sample specific features out of the input, which makes more general features remained.

However, instance normalization inevitably removes some discriminative information from original feature maps [Jin et al., 2020], which weakens the extracted features discrimination ability of extracted features. To address this emerging problem, we propose a memorized restitution approach to distill the discriminative information from the filtered out features and then restitute it as the final output feature maps.

Memorized Restitution

As discussed above, in order to maintain the discrimination ability of the refined features, we propose a following memorized restitution approach to distill discriminative information. We first obtain the filtered out feature R via a residual structure, which is defined as,

where R ∈ Rc×h×w , denoting the features that we have filtered out via the instance normalization operation. Since instance normalization operation will inevitably remove discriminative information from the original features, we need to distill and purify discriminative features from the residual feature R to maintain the discrimination ability.

At the training stage, given the feature map R at each iteration (we omit the subscript of feature map R for brevity), we assume R consists of two parts: D ∈ Rc×h×w with relatively more discriminative information, and G ∈ Rc×h×w with relatively more general information, which is defined as,

where k represents the kth channel, θk denotes the learnable parameters to split the residual feature map R into D and G, which are channel-wise mutually exclusive.

Then, we employ the SE-like channel attention [Hu et al., 2018] to define an attention vector α = [α1, α2, ..., αc] as,

where avepooling represents the average pooling layer, W1 and W2 are parameters to be learned, δ and η are the ReLU and Sigmoid activation function, respectively.

Since there are limited labeled samples under the few-shot learning framework, it is highly possible that the model would overfit. Thus we further propose a memorized mechanism with a memory vector M (l) ∈ Rc to store the long-term feature maps D, which is defined as,

where M (l) k ∈ Rh×w , (l) represents the lth iteration, k denotes the kth channel. At the lth iteration, we concatenate the feature map D to the memory bank at the channel level, and update D as,

where concat represents the concatenation operation, maxpooling represents the max pooling layer.

Once we obtain the updated feature map D, we restitute it to refined feature ˜ F as the final output ˜ F + of our proposed MemREIN method, and we also restitute the relatively unimportant feature map G with feature ˜ F as the "contaminated" feature ˜ F - for following loss optimization as,

Reverse Contrastive Loss

Apart from the conventional cross-entropy loss defined in Equation 2, we also propose a novel reverse contrastive loss Lrcl to promote the disentanglement of feature D and feature G. Instead of directly separating feature D and G, we first combine them with ˜ F defined in Equation 9 for better disentanglement. The reverse contrastive loss consists of two parts: L+rcl and L-rcl , e.g. , Lrcl = L+rcl +L-rcl . Given a minibatch Xb = {X1, ..., XN} contains N samples at the training phase, we first randomly select one anchor sample referred as Xa, and then we denote samples with the same category as the positive samples Xpos, samples with different categories as the negative samples Xneg. Note that the corresponding features of these samples are denoted with their subscripts such as ˜ Fa, ˜ Fpos, and ˜ Fneg in the following paragraphs.

We first reshape features ˜ F + and ˜ F - to the size of Rchw×1 and then pass them through one fully-connected layer following the softmax function to obtain the feature vectors ˜ f + and ˜ f -. Note that these two vectors have the same size of ∈ RK×1 . It is formulated as,

where W + and W - are trainable parameters with the same size of RK×chw , K is the number of classes. Then, our proposed reverse contrastive loss is defined as,

The goal of our proposed reverse contrastive loss is to promote the disentanglement of feature ˜ F + and feature ˜ F -, where feature ˜ F + contains more discriminative information and ˜ F - contains more general information. Since feature ˜ F + has better discrimination ability, sample features with the same category are closer and those with different identities are farther apart. Therefore, we propose L+rcl to encourage the features of positive samples ˜ f + pos to gather closer and separate the features of negative samples ˜ f + neg from the anchor feature. On the other hand, feature ˜ F - is more general which is not capable of distinguishing samples with the same category. Therefore, we propose L-rcl to separate the the features of positive samples ˜ f - pos from both negative sample features ˜ f - neg and the anchor feature ˜ f - a . The whole loss is defined as,

where λ is a hyper-parameter to control the balance of these two terms in the training phase.

4 Experiments

4.1 Experimental Setup

Baselines. Extensive experiments are made on three existing metric-based few-shot learning methods: MatchingNet [Vinyals et al., 2016], RelationNet [Sung et al., 2018], and GNN [Satorras and Estrach, 2018]. We compare our proposed method with following existing cross-domain few-shot learning methods: FT [Tseng et al., 2020], LRP [Sun et al., 2021], and ATA [Wang and Deng, 2021] to demonstrate the advantages of our proposed method.

Datesets. Five widely used datasets are used: miniImageNet [Ravi and Larochelle, 2017], CUB [Wah et al., 2011], Cars [Krause et al., 2013], Places [Zhou et al., 2017], and Plantae [Van Horn et al., 2018].

Setting. We take the same leave-one-out setting which is applied in other baselines. Specifically, we select one dataset among CUB, Cars, Places, and Plantae as the target domain for testing, and using the remaining three datasets along with dataset mini-ImageNet as the source domains for training.

Implementation details. We adopt the ResNet-10 [He et al., 2016] as the backbone network for our feature encoder E. We insert our proposed MemREIN method after the last batch normalization layer of all the residual blocks in the feature encoder E at the training stage. Instead of optimizing from the scratch, we apply a strategy that pre-trains the feature extractor by minimizing the standard cross-entropy classification loss on the 64 training categories from the dataset mini-ImageNet and this strategy is also applied in all the baselines. In the training phase, we set λ = 0.1 and train 1000 trials for all the methods. In each trial, we randomly sample Nw categories with Ns randomly selected images for each support set, and 16 images for the query set. We use the Adam optimizer with the learning rate 0.001.

4.2 Experimental Results

Quantitative Results

Table 1 shows the results under the leave-one-out setting. We first select out one dataset as the unseen domain for testing and use the remaining three datasets as well as the dataset mini-ImageNet for training since we already use the dataset mini-ImageNet for pre-training. Note that the baseline [Tseng et al., 2020] has two different training strategies, one is the "learn to learn" strategy and another is using fixed hyper-parameters. We consider the better results for comparison here, which is denoted as "+LFT" in the Table 1. The results demonstrate that our proposed MemREIN method can greatly improve the performance of all three metric-based few-shot learning methods, which reflects that our method has the capability of mitigating the domain gap problem. In addition, results show that our method consistently outperforms the "+LFT" method, which validates that our proposed method can better capture the variation of feature distributions across multiple domains than the "+LFT" method, thus the generalization ability of extracted features are better enhanced. In particular, for dataset Cars (5-way 1-shot), the accuracy of our MNet+MemREIN is 35.65 ± 0.45%, which outperforms the accuracy of baseline MNet+LFT by 16.43%.

Qualitative Results

As illustrated in Figure 2, we employ the t-SNE algorithm to visualize features that obtained by the feature encoder "before/within/after" our MemREIN method, where each color represents one class. We take the GNN baseline under the leave-one-out setting on the dataset CUB as the example. We randomly select 5 categories with 60 samples of each category in the testing spilt of the dataset CUB. The first column indicates two examples of the features from conventional GNN baseline, The second column indicates the features that only applied the instance normalization operation, and the third column indicates the features that applied our proposed MemREINmethod. As shown in the first column, there exists several rough clusters but the boundaries are unclear. After instance normalization, the overall model generalization ability of features is enhanced. In comparison with the first column and the third column, the features learned by our method are more clustered and separable, which validates the effectiveness of our novel memorized restitution approach.

Performance Study of λ

We carry out performance study on the hyper-parameter λ. We take our method under the leave-one-out setting (5-way 5-shot) and dataset CUB and Cars as the example. We set four different values λ = {0.01, 0.1, 0.5, 1} and the results are shown in Table 2. It can be observed that when setting λ = 0.1, it can achieve the best performance.

Ablation Study

We carry out ablation studies of different components in our proposed method. We compare with the GNN baseline under the leave-one-out setting (5-way 5-shot) and results are shown in Table 3. Comparing the results of Variant 1 and 2, it indicates that only applying the instance normalization operation results in the decrease of the accuracy. It is reasonable because the instance normalization operation will inevitably remove some discriminative useful information. In comparison with Variant 3 and 6, it validates the effectiveness of employing the memory bank on feature D. Comparing Variant 3, 6, and 7, it indicates that when employing memory bank on feature G, it would cause performance decrease. Empirically, when applying the memory bank on the feature D and directly using feature G, it can achieve the best performance.

Different Numbers of Ways

We consider a more practical situation that Nw may be different from that at the training stage. It also reflects the generalization ability of the model and results are shown in Table 4. Note that model GNN requires the number of ways to be the same while the training and testing, thus we evaluate with method MatchingNet and RelationNet (MNet and RNet for short). The model is trained on the datasets mini-ImageNet, Cars, Places, and Plantae and evaluated on the dataset CUB with different number of ways Nw. The results indicate that our proposed method are still capable of improving the generalization ability to the unseen domain with various numbers of ways. In addition, our proposed method consistently outperforms the baseline that has considered the domain-shift issue, which validates the superiority of our method.

5 Conclusion

In this paper, we investigated the cross-domain few-shot classification problem where exists the domain gap issue. We propose a novel framework, MemREIN, which considers Memorized, Restitution, and Instance Normalization to address this issue. We first alleviate feature dissimilarity across sample features via an instance normalization algorithm to enhance the overall generalization ability. In order to avoid the loss of fine-grained discriminative knowledge between different classes, a memorized restitution approach is further proposed to adaptively remember the long-term refined knowledge and restitute the discrimination ability. Finally, a novel reverse contrastive learning strategy is proposed to stabilize the distillation process. Extensive experiments demonstrate that MemREIN well addresses the domain shift challenge, and significantly improves the performance up to 16.43% compared with state-of-the-art baselines.