Multi-Attention Based Visual-Semantic Interaction for Few-Shot Learning

1 Introduction

With a large number of annotated data, deep learning methods have achieved tremendous success in various computer vision tasks [Chen et al., 2024; Huang et al., 2021; Huang et al., 2022a; Huang et al., 2022b; Huang et al., 2023; Huang et al., 2024], such as image recognition, object detection, etc. However, in many real-world scenarios, collecting a large number of labeled samples for each class can be impractical or expensive [Wu et al., 2023a; Wu et al., 2023b; Wu et al., 2024; Wang et al., 2023a; Wang et al., 2022;

∗Corresponding author: Wei Wang.

feature map

Wang et al., 2023b; Wang et al., 2021]. Therefore, FewShot Learning (FSL), which requires only a small number of samples to recognize new classes, has been receiving increasing attention recently [Liu et al., 2022; Xu et al., 2023; Cao et al., 2021]. Existing FSL approaches usually adopt the meta-learning strategy, which simulates FSL tasks using an episodic training approach on a fully annotated base dataset. The base dataset does not share any classes with the test dataset. Each episode consists of a labeled support set and an unlabeled query set, with the query set sharing the same label space as the support set.

It could be observed that humans can quickly recognize a novel category, as they could utilize semantic prior knowledge to efficiently extract discriminative visual features from a small number of samples. Therefore, existing FSL methods have started using semantic knowledge to assist in learning discriminative visual features [Chen et al., 2019; Xing et al., 2019; Peng et al., 2019]. Although these methods have achieved promising performance, the discriminability of visual features on the query set cannot be well guaranteed because semantic knowledge of the query set is unavailable. As shown in Figure 1, when semantic knowledge is not considered, the feature extractor may capture features of both the target of interest (American Redstart) and the background (branches). In the right part of Figure 1, the first row shows the case where neither the support set nor the query set considers semantic knowledge, the second row considers the semantic knowledge only on the support set, and the third row considers semantic knowledge on both sets. It can be observed that the similarity scores continuously increase when semantic knowledge is considered. Therefore, considering semantic knowledge in both sets is crucial for improving the performance of FSL.

To address the problem mentioned above, we propose a Multi-Attention based Visual-Semantic Interaction (MAVSI) approach for FSL. As shown in Figure 2, we first obtain the visual features of the support set, query set, and the semantic features of the support set, through the feature encoder and semantic encoder. Next, the proposed MAVSI employs Spatial Cross-Attention (SCAVSI) and Channel Attention (CAVSI) based Visual-Semantic Interaction modules to select discriminative visual features. As there is no semantic knowledge of the query set, we use all the semantic features from the support set for each query sample to generate visual features for the query set, with the number of features equal to the number of semantic features. Then, based on the support set, we calculate the class prototypes for each category and introduce a relation module to compute the relationship scores between each visual feature of the query sample and the corresponding class prototype. By promoting the relationship scores corresponding to the query sample's class label, we can extract discriminative visual features for the query set.

To further enhance the discriminability of the support set, we design a Visual-Semantic Contrastive Learning (VSCL) module. As shown in Figure 2, we perform a concatenation operation between the visual and semantic features in the support set. We calculate the visual-semantic similarity through a comparison network and aim to prompt the similarity between the visual features and their ground-truth semantic features. Our main contributions are summarized as follows,

• We propose a Multi-Attention based Visual-Semantic Interaction (MAVSI) approach for FSL, which contains two distinct modules, SCAVSI and CAVSI, and prompts discriminative visual features from both spatial and channel perspectives.
• We propose to utilize the semantic knowledge from the support set to guide the visual-semantic interaction on the query set with a relation module and introduce a VSCL module further to enhance the feature discriminability of the support set.
• Extensive experiments on four FSL benchmark datasets demonstrate that the proposed MAVSI outperforms state-of-the-art FSL approaches.

2 RelatedWork

FSL learns new concepts with only one or a few annotated samples and has gained extensive attention in recent years. Existing FSL methods mostly employ the metalearning strategy, which aims to transfer knowledge from an auxiliary base dataset to new tasks. Meta-learning-based FSL methods can generally be categorized into the following three types: optimization-based methods [Finn et al., 2017; Ravi and Larochelle, 2017; Rusu et al., 2019; Baik et al., 2020], metric-based methods [Vinyals et al., 2016; Snell et al., 2017], and data augmentation-based methods [Wang et al., 2018; Zhang et al., 2019; Wang et al., 2020]. The proposed approach belongs to a metric-based FSL.

In recent years, semantic information has played a crucial role in tasks such as image-text matching [Ji et al., 2021] and zero-shot learning [Zhao et al., 2023]. Particularly, in scenarios with scarce visual samples, semantic information is a preferable choice to assist model training, and existing FSL methods that utilize semantic information could be summarized into three categories.

The combination of visual features and semantic features. For instance, Xing et al. propose a mechanism that can adaptively combine information from both modalities (visual and semantic) according to novel classes to be learned. Furthermore, Schwartz et al. combine visual features with multiple and richer semantic information such as category labels, attributes, and natural language descriptions. Peng et al. propose a semantic-visual mapping network to conduct knowledge inference for novel categories from base categories.

Employing semantic information to synthesize visual features, to mitigate the deficiencies in visual features. For example, Chen et al. propose directly synthesizing instance features by leveraging semantics using a novel auto-encoder network. Xu and Le propose using a conditional variational autoencoder model to generate visual samples based on semantic embeddings.

Utilizing semantic information to refine visual features, to enhance discriminability of visual features. For instance, Yang et al. use the semantic knowledge to guide the visual perception about what visual features should be paid attention to when distinguishing a category from the others. As such, the feature embedding of the novel class, even with few samples, can be more discriminative. Afham et al. force the prototypes to retain semantic information about the class description, improving their generalization to novel classes at inference. Xu et al. propose to highlight pivotal local visual information with attention mechanism and align the attentive map with semantic information to refine the extracted visual features. Li et al. develop a class-relevant additive margin loss and a task-relevant additive margin loss with semantic information to prompt discriminative visual features.

The proposed approach belongs to the third category. Given the effectiveness of attention mechanisms in crossmodal tasks, many FSL methods employ them to facilitate interactions between semantic and visual features, thereby enhancing the discriminability of visual features, such as [Yang et al., 2023], [Yang et al., 2022], [Xu et al., 2022], and [Wang et al., 2020]. However, these methods only consider interactions between semantic and visual features within the support set. In contrast, this paper simultaneously considers semantic-visual interactions in both the support and query sets. Moreover, we introduce a multi-attention mechanism that operates in spatial and channel dimensions to enhance discriminative visual features more effectively.

���

���

3 Proposed Approach

In this section, we first provide problem formulation for FSL. Then, we detail the proposed multi-attention based visualsemantic interaction and visual-semantic contrastive learning modules. Finally, we introduce the relation network and our overall training loss.

3.1 ProblemFormulation

During the training phase, we generally have a base dataset for meta-learning purposes, and each class has abundant samples. Then, N novel classes do not overlap with the base dataset during testing. Additionally, each novel class has only K labeled samples (support set), resulting in a total of N×K samples, referred to as the N-way K-shot setting.

Following [Xing et al., 2019], we adopt the episodic training with the base dataset, performed with episodes that simulate the testing setup. Generally, our model is trained in an Nway K-shot setting, where each episode consists of N classes from the meta-training set, divided into two parts: K labeled samples in the support set and several unlabeled samples in the query set. The support set contains N ×K labeled samples with semantic knowledge.

In one episode, we have a support set Xs and a query set Xq . Then, we use a Convolutional Neural Network (CNN) to embed them into a visual space: Z = Gθ(X) ∈ RC×H×W , where θ represents a learnable parameter set. Moreover, C, H, and W are the channel, height, and width, respectively. Notably, we can regard r = H ×W as the spatial resolution of an image. Each C-dimensional vector corresponds to a position and represents a local region of an image, where some of these local regions may contain important features of the target object of interest.

3.2 Multi-Attention Based Visual-Semantic Interaction

Inspired by the self-attention mechanism, we propose a Multi-Attention based Visual-Semantic Interaction (MAVSI) approach that operates at both spatial and channel levels. With semantic knowledge, MAVSI could mitigate the negative impact of background noise by selecting discriminative visual features. Specifically, MAVSI consists of a Spatial Cross-Attention (SCAVSI) and a Channel Attention (CAVSI) based Visual-Semantic Interaction modules.

Spatial Cross-Attention Visual-Semantic Interaction

SCAVSI aims to explore the relationship between local regions and semantics with labels of support set. It selects local regions highly correlated with semantic knowledge, avoiding introducing irrelevant background noise at the spatial level. Due to the limited exploration of the relationship between semantics and local features by a single-head attention mechanism, we choose multi-head attention to find more highly correlated local features. For simplicity, we illustrate our proposed SCAVSI with only one attention head in Figure 3.

Let si0 ∈ Rd0 denotes the semantic from class i by semantic extractor. We transform the semantic to the same dimension as the visual feature through a semantic encoder. Specifically, we first pass si0 through a Multilayer Perceptron (MLP) layer to obtain si1 = MLP(si0 ) ∈ RC . Then, we duplicate si1 in the spatial dimension to obtain Si = repeat(si1 ) ∈ RC×H×W , where repeat is a replication operation.

As shown in Figure 3, we use three 1×1 convolution kernels ω1, ω2, and ω3 with learnable parameter sets to linearly

�

�3

⨂

⨂ matrix multiplication

Figure 3: The proposed SCAVSI. We pass visual and semantic features through three 1×1 convolution kernels, followed by a series of flattening and replication operations. Next, we obtain an attention map through visual and semantic information interaction, from which the spatially corrected visual feature ˜Z is derived.

transform the visual feature Zs,i j of the j-th sample from class i in support set, and the semantic Si of class i. We denote the transformed semantic as a key ω3(Si ), and denote the transformed visual feature as a query ω2(Zs,i j ) and a value ω1(Zs,i j ). As mentioned before, each C-dimensional vector corresponds to a position and represents a local region of an image. Thus, we reshape them into the size of RC×(HW) . Then, the transpose of ω3(Si ) is multiplied with ω2(Zs,i j ). This matrix multiplication calculates the similarity between the spatial feature of each semantic channel in ω3(Si ) and all visual channel features in ω2(Zs,i j ), establishing the relationship between semantic information and visual features in the spatial level.

Next, we apply the softmax function on the resulting matrix to perform spatial normalization, obtaining the visualsemantic attention map As,i 1 ∈ RHW×HW :

where γ is a scaling factor.

After obtaining the semantic-visual attention map, the next step is to perform a matrix multiplication between the value ω1(Zs,i j ) and the transpose of the attention map. The resulting matrix is then reshaped back to its original size, i.e., RC×H×W . Finally, the spatially modified feature is obtained by element-wise addition of the spatially corrected visual feature with the original input feature:

where ˜Zij denotes the modified features in spatial level. As we have no semantic knowledge of the query, we obtain its modified features with all semantics of the support set, with the number of features being equal to the number of semantics. Then, we use the relation module introduced in the following subsection, to select the modified feature with the highest score for each query sample.

Channel Attention Visual-Semantic Interaction

As some background noise may also affect discriminative information in the channel dimension, we propose a CAVSI module as a complement to SCAVSI, further enhancing the discriminative power of visual features in the channel level, as shown in Figure 4.

Given the spatially modified feature ˜Zs,i j ∈ RC×H×W of the j-th sample from i-th class and the semantic si for the i-th class, where we use the Global Average Pooling (GAP) layer to transform Si ∈ RC×H×W to si ∈ RC . Then, we use a two-layer MLP network as the channel attention generator to produce the attention weights ai2 ∈ RC . The final layer of the MLP is a sigmoid function, ensuring that ai2 consists of values in the range [0,1]:

We duplicate ai2 in the spatial dimension to enable it have the same dimension as the visual feature (i.e., Ai2 ), and conduct element-wise product with the spatially modified feature ˜Zs,i j . Then, we obtain modified features of Zs,i j in channel level as the following equation,

where W1, W2, b1, and b2 are the parameters of the MLP network, and σ is the activation function sigmoid. Similar to the modified features at the spatial level, we could obtain modified features for each query sample at the channel level.

3.3 Visual-Semantic Contrastive Learning

To further enhance the feature discriminability of the support set, inspired by contrastive learning techniques [Badamdorj et al., 2022], we propose a Visual-Semantic Contrastive Learning (VSCL) module. Since the visual and semantic features are not embedded in the same space, it is impossible to compute the similarity between them directly. Therefore, we introduce a comparison network F to deal with this problem, which aims to minimize the following equation,

where Zs is the modified feature of support set by the proposed MAVSI. N and K are the class number and sample number of each class in one episode. S+ is the semantic corresponding to the ground-truth of Zsj , and τ > 0 is a temperature parameter. Equation (5) enables the similarity between a visual feature and its ground-truth semantic larger with the comparison network F, so discriminative visual features of the support set are further promoted.

3.4 Relation NetworkModule for Classification

Following [Sung et al., 2018], the visual prototype Pi for class i could be obtained by Zs,i in support set as below,

Then, we concatenate each class prototype with each query set and calculate the similarity score as below,

where R consists of two convolutional layers and one MLP. Moreover, we utilize the loss of Mean Square Error (MSE) to train the model,

where m denotes the sample number of the query set. Different from [Sung et al., 2018], we compute the relation scores between the prototypes of the support set and query samples.

After introducing the modules in our proposed model, we could obtain the overall loss function as below,

where λ is a hyper-parameter used to balance the importance between the two losses.

4 Experiments

In this section, we validate the effectiveness of our proposed model on four benchmark datasets for FSL.

4.1 Datasets

miniImageNet [Vinyals et al., 2016] consists of 100 classes. These classes are divided into 64, 16, and 20 for training, validation, and testing. tiredImageNet [Ren et al., 2018] contains 608 classes, split into 351, 97, and 160 for training, validation, and testing. CIFAR-FS [Bertinetto et al., 2019] consists of 100 classes. These classes are divided into 64, 16, and 20 for training, validation, and testing. CUB-200-2011 [Wah et al., 2011] contains images from 200 bird species, where 200 species are divided into 100, 50, and 50 for training, validation, and testing, respectively.

4.2 Implementation Details

Similar to previous works [Xing et al., 2019; Schwartz et al., 2022; Yang et al., 2022], we utilize ResNet-12 as the backbone network, and modify the number of convolutional filters from [64, 128, 256, 512] to [64, 160, 320, 640]. In all cases, the comparison network F is the MLP with a LeakyReLUactivated hidden layer, and the relation network consists of convolutional layers and the MLP. We use Glove [Pennington et al., 2014] as the semantic extractor, which is pre-trained on a large corpus. Our experiments are implemented under 5way 1-shot and 5-way 5-shot settings. The input image size is 84×84. Following [Peng et al., 2019], we train the model for 150 epochs, with 800 episodes in each epoch. We use the Adam optimizer with a learning rate of 5e-3 and weight decay of 5e-6. The learning rate is dropped by half every 6,000 episodes, and other parameters such as λ, γ, and the temperature parameter τ are adjusted during end-to-end training. We conduct 5-way 5-shot and 5-way 1-shot classification tasks on each dataset during the testing phase. The final classification accuracy results are obtained by averaging over 10,000 episodes and we report it with a 95% confidence interval.

4.3 Main Results

To evaluate the effectiveness of our proposed model, we conduct extensive experiments on miniImageNet, tieredImageNet, CUB, and CIFAR-FS. The comparison results with state-of-the-art methods are shown in Tables 1, 2, and 3. Compared to previous methods that utilize semantic information to address FSL problem (e.g., KTN [Peng et al., 2019], TriNet [Chen et al., 2019], AM3 [Xing et al., 2019], SEGA [Yang et al., 2022], MultiSem [Schwartz et al., 2022]), our model achieves the best performance in both 1-shot and 5-shot settings on all benchmark datasets. Particularly, as shown in Table 1, in the 1-shot and 5-shot settings of tieredImageNet, our method outperforms SEGA by 2.43% and 3.17% improvements, respectively. This is because we conduct visual-semantic interaction on both the support and query sets, allowing us to capture more discriminative visual features. Additionally, our proposed approach consistently achieves the best results across almost all datasets and settings compared to all methods. Due to our aim at maintaining consistent parameters across all datasets and all FSL settings, we do not surpass DeepEMD on the miniImageNet with the 5-shot setting. These results could verify the effectiveness and superiority of our proposed FSL model.

4.4 Empirical Analysis

This section provides a deeper analysis and discussion of the proposed model.

Ablation Study. In Table 4, we record the accuracy results of our model using three proposed modules, including SCAVSI, CAVSI, and VSCL, as well as their combinations. It can be observed that SCAVSI, CAVSI, and VSCL are all highly effective, with average 1-shot accuracy improvements of 4.91%, 3.89%, and 2.58% on the three datasets, respectively. Moreover, when combining them together, further improvements in 1-shot learning accuracy are achieved. These results indicate that the three proposed modules all play important roles in the final performance of the model.

Analysis for Hyper-Parameter λ. λ is used to embody the importance of our proposed VSCL module. We conduct experiments on validation of miniImageNet and CIFAR-FS to analyze its impact on the model's performance. As shown in Figure 5 (a), we observe that, in 5-way 1-shot setting, the optimal values for λ are all within the range of 0.3 to 0.6. In comparison results, we set the value of λ as 0.5.

Analysis for Number of Attention Heads. By using different numbers of attention heads and their corresponding scaling factors [Vaswani et al., 2017], we evaluate their impact on the performance of the miniImageNet dataset. As shown in Table 5, the model's performance shows a trend of initially increasing and then decreasing. We speculate that when the number of attention heads is too small, the model's feature learning capacity is insufficient. Moreover, when the number is too large, it introduces significant redundancy.

Analysis for Temperature Parameter τ . There exists a temperature parameter τ in the proposed VSCL module. We select different values of τ and assess their impact on the performance of the tiredImageNet validation. As shown in Figure 5 (b), the accuracy curves remain relatively stable with increasing values of τ in both the 1-shot and 5-shot settings.

(a) �

(b) �

These results indicate that our proposed model is robust to the choice of τ. In both settings, the highest accuracy results are achieved when τ is set to 0.1.

4.5 Visualization Results

We visualize four feature maps from support and query sets, respectively. As shown in Figure 6, the visual features without visual-semantic interaction exhibit some background noise, which makes the visual features less discriminative. In contrast, background noise has been almost completely eliminated, demonstrating that our proposed approach could effectively enhance discriminative visual features. Additionally, we visualize feature distributions on the miniImageNet dataset, as illustrated in Figure 7. Through visual-semantic interaction, the dots representing different classes in the query set become more distinguishable and discriminative.

5 Conclusion

We propose a novel FSL approach named MAVSI, where two different attention modules are designed to enhance discriminative visual features at spatial and channel levels. To conduct MAVSI on both the support and query sets, we introduce a relation module and utilize the semantic knowledge from the support set to guide the visual-semantic interaction on the query set. Moreover, we propose a visual-semantic contrastive learning module to promote discriminative visual features of the support set. Extensive experiments on four benchmark datasets demonstrate that the proposed model could outperform existing mainstream FSL approaches. In future work, we will explore higher-quality semantic extractors, such as vision-language large models, and investigate the cross-domain FSL problem where a distribution shift exists between base and unseen classes.

Acknowledgments

This work was supported by the Key Natural Science Project of Anhui Provincial Education Department (No.KJ2021A0043) and the National Natural Science Foundation of China (No.61602004, No.62025604, No.62306343), Shenzhen Science and Technology Program (No.KQTD20221101093559018).