Exploring Cross-Domain Few-Shot Classification via Frequency-Aware Prompting

1 Introduction

Recently, most progress on Few-Shot Learning (FSL) has followed a Meta-Learning pipeline, where the sampled classes of episodic source and target tasks are disjoint but from the same dataset, i.e., they share a similar domain. However, this underlying assumption is not applicable to real-world scenarios since the source and target domains are usually different in realistic tasks. Besides, recent works [Chen et al., 2019; Guo et al., 2020] have also revealed that most existing metalearning models suffer performance degradation when facing a domain shift between source and target tasks, and even underperform simple pre-training and fine-tuning paradigms. This intractable issue makes the new research direction, Cross-Domain Few-Shot Learning (CD-FSL) distinguishable from the conventional FSL settings.

∗Corresponding authors: Qing Cai, Junyu Dong.

CD-FSL aims to generalize FSL models across different domains. Here we focus on the most challenging CD-FSL setting where only one single source domain is available for training. To achieve that, [Tseng et al., 2020] uses the featurewise transformation layers to manipulate the feature activation with affine parameters to improve the generalization of metric-based inductive bias. This feature-wise transformation approach has influenced some later research on varying the feature distribution for target domain adaptation [Liang et al., 2021; Zhang et al., 2022; Hu and Ma, 2022], but they either have employed extra parameters of weights on backbone networks or suffered from an incompatibility with other models. Apart from the feature-wise manipulation, it is also intuitive that improving the diversity of training tasks also takes a significant role in cross-domain generalization for inductive bias since it's learned from a collection of training tasks. Therefore, [Wang and Deng, 2021] first resorted to solving a worst-case problem around the source task distribution and proposed Adversarial Task Augmentation (ATA) to construct 'challenging' virtual tasks as a task augmentation technique. Their strategy improves the robustness of various inductive biases learned from meta-learning models and achieves good performance on the CD-FSL benchmarks.

However, when tackling the generalization problem on domain shift, most previous CD-FSL methods neglect an important factor that the frequency components of images also get involved in the robustness of inductive bias learned by existing meta-learning models. Several researchers [Wang et al., 2020a; Geirhos et al., 2018; Li et al., 2021; Rahaman et al., 2019] have already shown that the generalization behaviors of deep neural networks (especially CNNs) are sometimes counter-intuitive to human understandings (e.g., adversarial examples [Nguyen et al., 2015]). One convincing explanation is that deep neural networks are more sensitive to highfrequency components of a natural image which are not easyperceivable for human beings in recognition tasks [Wang et al., 2020a; Rahaman et al., 2019]. And the experimental observation in [Wang et al., 2020a] also demonstrates a phenomenon that when making the final classification decision, deep-learning models tend to capture these specific highfrequency cues of a specific training domain for a higher accuracy after they learn from the low-frequency information at the early training stage. This phenomenon will guide the meta-learning models to learn a distribution-specific inductive bias towards the high-frequency cues during the metatraining stage on the source domain, and thus limit the performance of cross-domain generalization when meta-testing on a new domain dataset.

Therefore, in order to learn the more robust inductive bias that can be generalized to the cross-domain scenario for existing meta-learning methods, we make one of the first attempts to consider the CD-FSL challenge from a frequency-aware perspective. As shown in Figure 1, in this paper, we try to propose a frequency-aware prompting method with mutual attention to prompt the network either to stress the low-frequency semantic information or weaken the tendency on capturing the distribution-specific high-frequency cues, which aims to learn robust inductive bias and let the meta-learning models perceive more like the human when facing a new recognition task. Concretely, we first decompose the source images into corresponding low-frequency and high-frequency components by applying the Discrete Wavelet Transform (DWT) and constructing the frequency-aware augment samples by modifying high-frequency components. In addition, a mutual attention module is designed to enhance the interaction between different frequency-constructed features, and implicitly stress the low-frequency semantic information or suppress the high-frequency cues. Note that [Fu et al., 2022b] also explores the CD-FSL problem from the frequency viewpoint, but their research focuses on varying the feature distribution on a feature-wise level, whereas our frequency-aware prompting method is used at the task level and can be orthogonal with the most off-the-shelf CD-FSL method. We also employ an adversarial training procedure to make task augmentation, which is similar to [Wang and Deng, 2021]. A KL Divergency loss is also applied to restrict the final class prediction of different frequency-reconstructed inputs from the same source image and help the network converge. Overall, our contribution can be summarized as followed:

• We consider the most challenging single source domain generalization of the Cross-Domain Few-Shot classification from a frequency-aware perspective. To learn more robust domain-adaptive inductive bias for existing meta-learning models, we proposed a frequency-aware prompting method to simulate human perception when facing a new recognition task. The augment samples are elaborately constructed using the decomposed frequency components to prompt the meta-training process.
• A special mutual attention module is designed to facilitate the information interaction across different frequency-reconstructed features and thus promote the network to implicitly stress the related low-frequency semantic information which helps with generalization.
• Experimental results on two CD-FSL benchmarks [Guo et al., 2020; Tseng et al., 2020] reveal that our method can effectively improve the average classification accuracy in CD-FSL settings when compared to previous methods, and also can be used as a plug-and-play method for most existing meta-learning models.

2 RelatedWorks

In this section, we briefly introduce the previous related works on Few-Shot Learning (FSL) and extended CrossDomain Few-Shot Learning (CD-FSL).

2.1 Few-Shot Learning

Few-Shot Learning [Shu et al., 2018; Wang et al., 2020b; Lu et al., 2020; Koch et al., 2015] has been widely studied, especially in the Meta-Learning manner [Vilalta and Drissi, 2002; Hospedales et al., 2021; Sun et al., 2019; Triantafillou et al., 2019; Vanschoren, 2018]. Most recent works of metalearning methods proposed for FSL problems can be roughly divided into two types: 1) Optimization-based methods [Finn et al., 2017; Nichol et al., 2018; Ravi and Larochelle, 2017] aim to learn a generalizable model initialization that can be quickly adapted to target novel tasks with limited labeled samples. 2) Metric-based methods [Vinyals et al., 2016; Sung et al., 2018; Snell et al., 2017; Satorras and Estrach, 2018; Oreshkin et al., 2018] try to construct a metric function in the embedding space and perform classification on a novel task by comparing the similarity between query images and limited labeled support images through this metric. Among these two types, the metric-based methods have drawn more attention from researchers when considering the performance and simplicity. However, what is common to all of these FSL solutions is that they assume base and novel classes are from the same domain. Due to this limitation, the Cross-Domain FewShot Learning (CD-FSL) benchmark is proposed to focus on the domain shift problem.

2.2 Cross-Domain Few-Shot Learning

As mentioned, CD-FSL [Guo et al., 2020; Tseng et al., 2020; Du et al., 2022; Li et al., 2022; Oh et al., 2022; Yuan et al., 2022] has been trying to address a more challenging and practical scenario where the source (base classes) and target (novel classes) image domains are dissimilar, as well as their labels are disjoint. This problem was preliminarily discussed in [Chen et al., 2019] when conducting an analysis of existing meta-learning methods in the cross-domain setting. [Guo et al., 2020] then extend the study to a Broader Study of Cross-Domain Few-Shot Learning (BSCD-FSL) benchmark which consists of four image datasets, including agriculture (CropDisease), satellite (EuroSAT), dermatology (ISIC), and radiological (ChestX) images, along with a decreasing domain similarity to ImageNet. [Tseng et al., 2020] also proposed a benchmark with four datasets and use the featurewise transformation layer to simulate various feature distributions during the training stage and thus encourage existing metric-based meta-learning models to generalize to these unseen domains. Similarly, [Liang et al., 2021] then introduces a noise-enhanced supervised autoencoder (NSAE) which can teach the model to capture broader variations of feature distributions and boost the cross-domain generalize capability.

Different from focusing on diversifying feature distributions, [Wang and Deng, 2021] develops an Adversarial Task Augmentation (ATA) algorithm to construct inductive bias adaptive challenging tasks by solving a worst-case problem. This task augmentation method tries to improve the generalization capability of existing meta-learning models from the perspective that how well the inductive bias (a.k.a metaknowledge) is learned depends on the training task diversity. It is intuitive that a wider task distribution space would cover the domain gap between source and target domains.

There are also other researches [Phoo and Hariharan, 2021; Fu et al., 2021; Das et al., 2022; Fu et al., 2022a] that aim to adapt to the novel target domain by leveraging additional unlabelled data [Phoo and Hariharan, 2021] or a few labeled data [Fu et al., 2021; Das et al., 2022], whereas in this paper we consider that there is no access to any target data.

3 Methodology

3.1 ProblemFormulation

For every meta-learning task T consisting of a support set Ts and a query set Tq, it is generally called a N-way K-shot situation when Ts contains N classes with K labeled samples in each class. And Tq consists of Q query samples sharing the same classes as in Ts. Then it can be defined as T = (Ts, Tq), where Ts = (xsi , ys i )N×K i=1 and Tq = (xqj , yq j )Qj=1 are sampled from a task distribution T0. As for cross-domain few-shot classification, due to the domain shift scenario, the metatraining dataset (e.g. mini-ImageNet [Vinyals et al., 2016]) is different from the meta-testing dataset (e.g. CUB [Welinder et al., 2010]). Here we only consider the setting where only one source domain dataset (mini-Imagenet) is used for training while testing on several other different datasets. But notice that every combination of support set Ts and a query set Tq for a meta-learning task is from the same domain.

3.2 Overall Mechanism

Figure 2 shows the overall pipeline of our proposed Frequency-Aware Prompting method. Here we first give an overall explanation of our Frequency-Aware Prompting Mechanism and the training strategy. Our method takes a dual-branch framework, where the decision is made by a hyper-parameter probability p. A method for constructing frequency-aware augment samples is designed at the task level. The difference between the two branches is that a random convolution [Lee et al., 2020] layer is maintained from [Wang and Deng, 2021] for feature embedding in one branch, and followed by the mutual attention module before the next step. The max iteration number Tmax of gradient ascending serves for the adversarial training. Other details can be viewed at the Algorithm. 1 in supplements.

3.3 Frequency-Aware Augmentation

To prompt the meta-training with frequency information, we design a method of generating frequency-aware augment samples by reconstructing the high-frequency components. This manner includes two meanings. On the one hand, the extra samples will be reunited along with the original samples to form a new source task, thus improving the task diversity for meta-training. On the other hand, these prompt samples aim to weaken the tendency of networks that focus too much on high-frequency cues while paying less attention to low-frequency information.

To be specific, for a source task T0 = (X0, Y0), we conduct Discrete Wavelet Transformation (DWT) on X0 with the 2D Haar wavelets, and get four frequency subbands denote as FLL, FLH, FHL, FHH. We consider the low-frequency components as Flow = FLL and keep it unchanged while performing two different operations on the high-frequency components Fhigh = [FLH, FHL, FHH] as (1) replacing the Fhigh with the same size tensor of zeros; (2) replacing the Fhigh with the same size of tensor consisting of random numbers sampled from the normal distribution. For the first operation, we can get reconstructed Xzeros 0 after applying Inverse Discrete Wavelet Transformation (IDWT) and Xrandn 0 in the same way for the second operation. By doing so, we get a new source task ˜ T0 = {X0,Xzeros 0 , Xrandn 0 , Y0} as mentioned before, where the labels Y0 are same.

Compared to the original T0, this new ˜ T0 is more diverse as a meta-training source task. Besides, the task distribution ˜ T0 it represents is more faithfully than T0 since it consists of 'noisy' samples near the distribution boundary. Note that for images from the same category, these frequency-aware samples will not introduce semantic drift since we keep the low-frequency components unchanged. This can implicitly prompt the network to rely more on low-frequency information to make the classification decision and learned robust inductive bias which can perform better in the later crossdomain generalization than previous.

By using the gradient ascent process with early stopping (for Tmax iterations) as the adversarial training based on:

which is sampled from the wider source task distribution ˜ T0, we can get corresponding virtual 'challenging' tasks as:

where ˜ T can represent a virtual 'challenging' task sampled from a distribution ˜ T which is ρ distance away from ˜ T0.

3.4 Mutual AttentionModule

To facilitate the information interaction across different frequency-reconstructed features, we proposed a Mutual Attention Module for either stressing the correlated features or suppressing other unrelated information on extra generated features of xzeros and xrandn with their original image feature x. As shown in the Figure 2, we take xzeros and x as examples for better illustration. The attention module takes two same-size features as inputs, where for every attention operation there is one input used as the query and the other is the key tensor. The attention calculation is mutual to both inputs and the weights are applied to the corresponding value tensor before they are merged. A residual tensor is also added to modify the final outputs.

Figure. 2 has shown where to use this attention module. Overall, we consider the features of an original image x ∈ RN×K×C×H×W as an anchor, meanwhile modifying two frequency-manipulated feature xzeros ∈ RN×K×C×H×W and xrandn ∈ RN×K×C×H×W based on x. The underlying motivation is that the low-frequency features have been preserved and stressed, whereas other uncorrelated frequency features are suppressed. Therefore, the network is implicitly prompted to pay more attention to low-frequency information which is more useful for generalization across different domains. Note that the proposed Mutual Attention Modules do not change the dimension of both inputs and outputs.

3.5 Training and Inference

Network Training: Our Frequency-Aware Prompting method is a bi-level optimization process in the meta-training stage, which is similar to [Wang and Deng, 2021]. For the inner loop, it tries to find a 'challenging' input based on the new source task for the outer loop by maximizing the loss function, while the outer loop updates the model parameters (learned inductive bias) to minimize the loss function on the new challenging input. And the proposed FrequencyAware Prompting Augmentation is employed to help improve the diversity of tasks in the inner loop stage. During the meta-training stage, the objective function is first composed of three initial Cross-Entropy (CE) FSL losses from XTmax , Xzeros Tmax and Xrandn Tmax that:

and then to restrict the network prediction and help the network to converge, we also proposed to use the KL divergence loss as the special regularization in training. Specifically, there exist two KL divergence losses defined as:

where P(.) represents the distribution of the network prediction scores for classification.

Overall, the training loss can be summarized as:

Network Testing: During the meta-testing stage of the network, for each episodic task of images sampled from the target domain, we only feed the original images as inputs to the baseline meta-learning backbone with all additional processes discarded, and get their prediction. No frequencyaware augment samples as well as the mutual attention module are used. The class with the highest probability will be considered as the classification results.

Further Explanation of Methodology: Due to the limited pages, a more detailed explanation of the Methodology can be found in our supplementary materials.

4 Experiments

In this section, we evaluate the proposed Frequency-Aware Prompting method on several CD-FSL benchmark datasets and baseline methods. Following we first describe the experiments settings and the implementation details. Then we present the evaluation results and ablation studies.

4.1 Experiments Settings

Datasets: We conduct extensive experiments under the very strict cross-domain few-shot learning settings where only one single source dataset is used for meta-training, i.e., miniImagneNet [Vinyals et al., 2016]. There are two CD-FSL benchmarks used in the meta-testing phase for evaluation. One is from the FWT benchmark [Tseng et al., 2020] which includes CUB [Welinder et al., 2010], Cars [Krause et al., 2013], Places [Zhou et al., 2017], and Plantae [Van Horn et al., 2018]. The other one is the BSCD-FSL benchmark [Guo et al., 2020] which consists of ChestX, ISIC, EuroSAT, and CropDisease. For all experiments, we select the model with the best validation accuracy on mini-ImageNet for the next testing on eight target datasets.

Implementation Details: We conduct our experiments under the open-source CD-FSL framework provided by [Wang and Deng, 2021]. For a fair comparison, ResNet-10 is selected as the feature extractor and the optimizer is Adam with a fixed learning rate α = 0.001. The iteration number for early stopping is kept as Tmax = 5 in all experiments and the gradient ascending step takes a learning rate β from {20, 40, 60, 80}. We maintain the Random Convolution layer applied in [Wang and Deng, 2021] with a probability (1 - p), where p is selected from {0.5, 0.6, 0.7}. The filter size k is randomly sampled from the candidate pool K = {1, 3, 5, 7, 11, 15} and the Xavier normal distribution [Glorot and Bengio, 2010] is used to initialize the layer weights. Both stride and padding sizes are determined to keep the image size unchanged. Note that we follow the strategy of ATA which uses a pre-trained feature extractor by minimizing the standard cross-entropy classification loss on the 64 training classes in the mini-ImageNet dataset. Evaluation results are obtained in the 5-way 1-shot/5-shot settings with 2000 randomly sampled episodic tasks with 16 query samples per class, and report the average accuracy (%) with a 95% confidence interval. All experiments reported are conducted on the RTX 2080Ti GPUs.

4.2 Evaluation Results

Here in this subsection, we present the evaluation results with CD-FSL benchmarks and the baselines.

Comparison with State-of-the-arts: We follow the settings in [Wang and Deng, 2021] to make a fair comparison on eight datasets with several stat-of-the-arts CD-FSL methods. Specifically, we integrate the Frequency-Aware Prompting (FAP) into three meta-learning baselines: GNN [Satorras and Estrach, 2018], RelationNet [Sung et al., 2018], and TPN [Ma et al., 2020]. Then compare the performance improvements on the baselines using ours with FWT [Tseng et al., 2020], LRP [Sun et al., 2021], and ATA [Wang and Deng, 2021]. Note that LRP can only be applied to metricbased meta-leaning models such as GNN and RelationNet. All the methods presented select ResNet-10 as the feature extractor. Quantitative results of 1-shot and 5-shot classification average accuracies as well as their 95% confidence interval are shown in Table 2, and the parameters of ours for comparison are shown in Table 1. As presented, our method achieves considerable accuracy gaining on three baselines, and in most cases achieves the best or second-best performance over benchmark datasets. Also, we observe that our method can provide the most impressive results on the GNN baseline, which means that GNN is more robust when applied with our frequency-aware method under the CD-FSL settings. Furthermore, there are also some performance limitations on the target dataset such as the satellite images (EuroSAT).

Evaluation on Robustness and Frequency Perception: In order to verify the robustness of the inductive bias learned from our Frequency-Aware Prompting method, we also provide the evaluation results of meta-testing with different frequency-reconstructed images in Table 3. To be specific, instead of meta-testing on tasks sampled from the eight novel target datasets, we turn to change the frequency cues of original meta-testing samples into new frequency-reconstructed tasks. For a target domain novel task Tn = (Xn, Yn), we design three types of new target domain task as (1) replacing the Fhigh with zeros and get T zeros n = (Xzeros n , Yn); (2) replacing the Fhigh with random numbers sampled from the normal distribution and get T randn n = (Xrandn n , Yn); (3) directly add the Gaussian noise to images and get the T noise n = (Xnoise n , Yn). The purpose is to test the robustness of the learned inductive bias when facing a new task with frequency-disturbed cues or noise under the cross-domain settings and its tendency on frequency perception compared to the original meta-testing.

From the results, we learn that the low-frequency components take the most cues for semantic information since using T zeros n for meta-testing can already get a considerable classification accuracy in CD-FSL benchmarks compared to using original tasks Tn. Besides, the disturbance performed on high-frequency components actually influences the final classification results, which leads to the performance degradation. But our learned inductive bias can be more robust to these disturbances, suggesting that our frequency-aware mechanism prompts the model to concentrate on the most important low-frequency cues. And the testing results presented by using T noise n also demonstrate the robustness of inductive bias learned by our method in CD-FSL settings.

4.3 Ablation Studies

Effect of Gradient Ascend Learning Rate β: In all the above experiments, we keep the max iteration number for early stopping as Tmax = 5 and the probability of dualbranch decision as p = 0.5 unchanged. Thus, the most important hyper-parameter which influenced the training process is the learning rate β. Here in this ablation study, we try to explore the effect of the learning rate β used in the gradient ascend process, which aims to construct a virtual 'challenging' task (˜ T0)Tmax based on the frequency-aware augmented task ˜ T0 for gradient descending optimization.

Experiments are conducted on both 5-way 5-shot and 5way 1-shot settings with the learning rate β selected from the set of {20, 40, 60, 80} as mentioned before. Other metatraining parameters are fixed as Tmax = 5 and p = 0.5 following the previous experiments. Average classification results over eight cross-domain few-shot learning datasets are reported in Figure 3. From it we can see that different baselines prefer various β. GNN and TPN perform better on a small β while the RelationNet needs a larger one.

Effect of Probability p: The average accuracy on eight datasets of our methods is shown in Table 4. In order to make a fair comparison through different baselines (GNN, RealtionNet, TPN), we set hyper-parameter p as 0.5 unchanged. However, as p grows, the training consumption is increasing according to Algorithm 1 in Supplementary Material. Therefore, keeping p as 0.5 is the better choice to get a performance and effectiveness trade-off in our experiments.

Effect of Mutual Attention Module: We also explore the improvement provided by the Mutual Attention Module designed in our method. Ablation experiments are conducted on both 5-way 5-shot and 5-way 1-shot settings with fixed meta-training parameters as β = 40, Tmax = 5, and p = 0.5. All baselines and the ablative models are trained in the same manner for a fair comparison. Figure 4 shows the average classification results over eight cross-domain few-shot learning datasets with the settings of using baseline model, using frequency-aware prompting method without mutual attention modules, and using our complete method respectively. From the figure we can learn that either with or without the mutual attention modules, the average classification results of the frequency-aware prompting method on eight datasets still achieve higher accuracy than original baseline models.

T-SNE Visualization: Figure 5 provides the t-SNE visualization of baseline model GNN and GNN+FAP (ours) on both mini-ImageNet (source) and CropDisease (target) datasets. Concretely, the representations encoded by the feature extractor are projected into a 2D space. For each dataset, 5 categories of 100 samples are randomly sampled, and they are fed into two models samely. Different colors indicate different categories. From Figure 5, we can learn that compared to GNN, the distribution of each category is indeed more dispersed in our method as expected, hence making it easier to recognize the target categories.

Computational Burden Explanation: Generally, the proposed method mainly focuses on frequency-aware augmentation when compared with other methods. Therefore, it didn't introduce too much computational burden.

Supplements: The full paper with supplementary materials can be found in the resources link for further explanation.

5 Conclusion

In this paper, we try to explore the cross-domain fewshot learning problem from a frequency-aware perspective. For this purpose, we propose a frequency-aware prompting method with mutual attention to simulating the human perception of selecting frequency cues when facing a new task. Concretely, the generated frequency-aware augment samples can improve the diversity of the meta-training episodic task and prompt the meta-training process during training. The mutual attention modules can facilitate the information interaction across different frequency-reconstructed features to either stress or suppress the corresponding information. Experimental results on the benchmarks show the effectiveness of our method on CD-FSL settings and the robustness of inductive bias learned from existing meta-learning baselines.

Acknowledgments

This work was supported in part by the National Science and Technology Major Project of China (No.2022ZD0117201); in part by the National Science Foundation of China (No.62102338); in part by TaiShan Scholars Youth Expert Program of Shandong Province (No.tsqn202312109); in part by the China Postdoctoral Science Foundation (No.2023M733342); in part by the Qingdao Postdoctoral Innovation Project (No.QDBSH20230101001).