Connecting Giants: Synergistic Knowledge Transfer of Large Multimodal Models for Few-Shot Learning

1 Introduction

Deep learning models have achieved remarkable success in numerous computer vision tasks [Li et al., 2019]. However, their effectiveness typically relies on deep neural architectures [He et al., 2016] and large-scale training datasets [Russakovsky et al., 2015], which hinders their applicability in real-world scenarios where annotated data are scarce. In contrast, humans exhibit an exceptional ability to acquire new concepts and recognize categories from only a handful of samples, aided by extensive prior knowledge and contextual understanding [Ralph et al., 2017]. This gap has motivated researchers to investigate few-shot learning (FSL) [Tang et al., 2020; Wu et al., 2022], where a model classifies query samples into one of N novel classes, each provided with only K labeled examples.

∗ Corresponding author.

The effectiveness of FSL heavily relies on leveraging prior knowledge to address data scarcity. Existing methods commonly transfer knowledge [Tang et al., 2022; Zha et al., 2023] from a disjoint base dataset to novel classes. Early works primarily focused on efficiently exploiting visual prior knowledge, including metric-based [Snell et al., 2017] and optimization-based paradigms [Ravi and Larochelle, 2016], both striving to train a base learner capable of rapid adaptation to novel classes with limited training data. While these methods have achieved promising results, there remains a huge gap in comparison to how humans utilize accumulated knowledge and experiences. As a result, semantic-based methods have emerged to explore various types of semantic knowledge as auxiliary information to improve FSL performance. This semantic knowledge can be obtained either manually (i.e., attribute annotations) or automatically (i.e., word vectors). Unfortunately, acquiring attribute annotations requires substantial human effort and may be infeasible for large-scale datasets, while word vectors derived from a single class name tend to be noisy or lack contextual richness. Hence, how to effectively collect and utilize high-quality prior knowledge in FSL is worthy of further investigation.

Perceptual filling-in [Neumann et al., 2001] is a fundamental characteristic of the human visual system, in which the brain employs prior knowledge and contextual cues to intuitively "fill in" missing information, resulting in a coherent and comprehensive perception. This phenomenon becomes particularly apparent in scenarios where visual stimuli are limited or partially obscured, enabling a seamless visual experience despite incomplete data. Inspired by this phenomenon, we hypothesize that transferring rich external knowledge to the off-the-shelf few-shot learner can further improve performance. Recently, Large Multimodal Models (LMMs) [Ouyang et al., 2022; Radford et al., 2021] containing abundant implicit knowledge have emerged as powerful repositories of external knowledge in various domains. These models encompass extensive understandings of the visual world, language, and inter-entity relationships, encapsulating diverse knowledge and information. A natural question thus arises: can we replicate this human-like cognitive process by connecting the rich, multimodal knowledge of these "giants" to compensate for incomplete visual data?

In this paper, we propose a Synergistic Knowledge Transfer (SYNTRANS) framework to harness the extensive knowledge embedded in large multimodal models to empower the small few-shot learner. Three key challenges arise in achieving this goal: ❶ effectively distilling desired visual and semantic knowledge from these models, ❷ transforming explicit or implicit knowledge into a usable form, and ❸ adaptively integrating them with limited visual data to improve FSL performance. As illustrated in Figure 1, SYNTRANS addresses these challenges in three stages. First, we introduce a vast CLIP model as a strong teacher, adding a linear projection layer after the frozen few-shot vision encoder as a weak student to distill semantic-aligned visual knowledge via an unsupervised proxy task. Next, our Synergistic Knowledge Mining (SynMine) module exploits a large language model to generate comprehensive text descriptions by tapping into the implicit knowledge through chain-of-thought prompting. These descriptions are then refined by a visuallanguage model into rich semantic descriptors, producing deeper, context-aware understanding of class characteristics. Central to the semantic transfer stage is the Visual-Semantic Bridging (VSBrid) module, which leverages a dual encoderdecoder design to facilitate bi-directional knowledge transfer between the visual and semantic spaces, ultimately mapping these high-quality visual embeddings and semantic descriptors to practical class-specific classifier weights. Finally, a visual weight generator and a semantic weight reconstructor are incorporated for dynamic visual-semantic knowledge fusion, constructing robust and adaptable multimodal classifiers.

We evaluate SYNTRANS on four benchmark datasets, demonstrating its state-of-the-art performance even when equipped with a simple few-shot vision encoder. To the best of our knowledge, SYNTRANS is the first framework that systematically integrates knowledge from large multimodal models to empower small few-shot learners, opening new avenues for bridging the gap between human-like intuition and machine learning in FSL.

2 RelatedWorks

Visual-based FSL Methods. Visual-based FSL methods [Tang et al., 2023; Fu et al., 2023; Fu et al., 2024] transfer prior visual knowledge from base classes to novel classes. Broadly, these approaches can be categorized into two branches: optimization-based methods and metric-based methods. Their core distinction lies in how they leverage the support set, either by fine-tuning an end-to-end network or by directly generating classifiers for novel classes. Optimization-based methods [Finn et al., 2017; Ravi and Larochelle, 2016] focus on learning an effective initialization or optimization strategy, enabling rapid model adaptation to novel tasks with only a few fine-tuning steps. Despite their flexibility, such methods can face meta-overfitting issues when limited labeled data are available. In contrast, metric-based methods [Vinyals et al., 2016; Snell et al., 2017; Sung et al., 2018] learn a metric space where samples from the same category lie close together while those from different categories are farther apart. Another emerging paradigm [Chen et al., 2019] involves pre-training a powerful feature extractor on base data and directly generating classifier weights for new classes using, for instance, a cosine classifier [Luo et al., 2018]. However, purely visual approaches may struggle under limited training samples, as real-world data often contain background noise and significant intraclass variation. Consequently, relying solely on visual cues may be insufficient for robust recognition of novel categories. This limitation naturally leads to the question of how high-quality semantic knowledge can be integrated to complement visually dominated FSL methods.

Semantic-based FSL Methods. To overcome the shortcomings of purely visual approaches, semantic-based FSL methods [Li et al., 2023; Lu et al., 2023] leverage auxiliary semantic information such as attributes [Lampert et al., 2009], word embeddings [Xian et al., 2019], or even knowledge graphs [Miller, 1995]. For example, AM3 [Xing et al., 2019] combines label embeddings derived from class names with visual prototypes to generate semantic prototypes, which are then adaptively fused. Knowledge graphs also provide valuable correlation cues: KTN [Peng et al., 2019] builds a graph convolutional network (GCN) with node representations and edges derived from label embeddings and semantic relationships, allowing knowledge transfer from base to novel categories. Nevertheless, these methods often rely on either sparse attribute annotations or word vectors obtained from limited textual sources (e.g., Glove [Pennington et al., 2014], Word2vec [Mikolov et al., 2013]), which may lack sufficient contextual richness. As a result, the potential noise in these external semantics can hinder performance. Encouraged by the rise of large multimodal models [Ouyang et al., 2022; Radford et al., 2021] that encapsulate abundant implicit knowledge, we investigate how to extract and distill higherquality prior knowledge to complement visual-based FSL. Our experiments show that the desired knowledge distilled from these large multimodal models can significantly boost FSL performance, even when used with a relatively simple pre-trained backbone.

3 Method

3.1 ProblemFormulation

We consider the standard FSL setting [Vinyals et al., 2016], where two disjoint sets of classes are given: a base set Cbase and a novel set Cnovel, such that Cbase ∩ Cnovel = ∅. The model first trains on samples from Cbase, denoted by Dbase = {(x, y)}, where each pair (x, y) corresponds to an image x and its one-hot label y drawn from Cbase. For semantic reference, each label y can be mapped to a specific class name c, such as "House Finch" or "Robin". During evaluation, we adopt the "N-way K-shot" protocol, which randomly selects N categories from Cnovel to construct two subsets: a support set and a query set. The support set S = {(xi, yi)}N×K i=1 contains K examples for each of the N novel classes, while the query set Q = {(xi, yi)}N×Q i=1 includes Q samples from the same N categories. Here, xi denotes the i-th image, and yi is one of the novel class labels in Cnovel. The main objective of FSL is to leverage the base knowledge (learned from Dbase) along with the limited support samples in S to classify new query images in Q.

3.2 Overall Framework

As shown in Figure 1, the SYNTRANS framework consists of three stages: visual knowledge distillation, semantic knowledge transfer, and multi-modal knowledge fusion. These stages work synergistically to enhance a few-shot learner by integrating visual and semantic knowledge. Notably, unlike traditional FSL methods, our method does not require fine-tuning the vision encoder at any stages. This unique characteristic highlights the efficiency of SYNTRANS, making it readily applicable on top of existing few-shot learners.

Visual Knowledge Distillation. In this stage, we train a lightweight projector fφ and a cosine classifier fΦ. The projector fφ enables SYNTRANS to perform CLIP-like visionsemantic alignment, while the classifier fΦ distills more taskrelevant visual knowledge from the few-shot vision encoder for subsequent knowledge transfer. Specifically, the parameters of fΦ are optimized using cross-entropy loss, while the parameters of fφ are optimized via the teacher-student distillation paradigm [Wu et al., 2024].

Semantic Knowledge Transfer. This stage consists of two phases: semantic knowledge mining and visual-semantic knowledge bridging. First, we introduce the train-free Synergistic Knowledge Mining (SynMine) module, which efficiently extracts implicit knowledge from visual- and language models to generate high-quality semantic descriptors. Next, the Visual-Semantic Bridging (VSBird) module facilitates bidirectional knowledge transfer between the visual and semantic spaces, mapping the visual embeddings and semantic descriptors to category-specific classifier weights.

Multi-modal Knowledge Fusion. In this stage, the frozen few-shot learner computes visual prototypes for all classes in the support set. Simultaneously, the VSBird module generates classifier weights based on these visual prototypes and the semantic descriptors. To integrate these weights into robust multimodal few-shot classifiers, we introduce a visual weight generator and a semantic weight reconstructor, which function as meta-learners, adaptively combining both types of classifier weights for current FSL task.

3.3 Visual Knowledge Distillation

In this stage, we introduce a large CLIP teacher model to distill semantic-aligned visual knowledge, empowering the frozen few-shot learner with the ability to perform CLIP-like vision-semantic alignment. Unlike the heavy CLIP vision encoder, the frozen few-shot vision encoder is lightweight and compatible with existing FSL methods, such as IER [Rizve et al., 2021] and SMKD [Lin et al., 2023], enabling training from scratch for simplicity.

As shown in Figure 1(a), we align the few-shot vision encoder with the CLIP vision encoder by learning a linear projector. To achieve this, we treat unsupervised visionsemantic alignment as a proxy task, inspired by [Li et al.,

2024], and use knowledge distillation to align the output distributions of both models. For a given FSL task with images {xi}N×(K+Q) and their class names {cj}N from Cbase, the CLIP teacher model first processes the category names using a fixed prompt template (e.g., "a photo of a {CLASS}"), then passing images and category names through the image encoder f t I and text encoder f tT to obtain normalized teacher image features uti and text features wtj . We then input the same images into the frozen few-shot vision encoder f s I to obtain the normalized student image features usi . The learnable projector fφ(·) is introduced to match the feature dimensions at minimal computational cost while ensuring alignment quality. The teacher and student image features, along with the generated teacher text features, are used to compute the output logits qt i and qs i for the teacher and student models, respectively. The knowledge distillation loss is formulated using Kullback-Leibler divergence:

where σ(·) is the softmax function and τ is the temperature.

Additionally, the cosine classifier fΦ is trained to acquire transferable visual knowledge from the base dataset Dbase using the pre-trained few-shot vision encoder. Let Wbase = {wc }c∈Cbase denotes the weight vectors of classifier fΦ. The classification loss, Lce, is computed with cross-entropy over the base classes Cbase:

This allows the learned classifier can provide more meaningful and task-relevant visual knowledge for subsequent knowledge transfer and fusion within our SYNTRANS. Finally, we combine the knowledge distillation and classification losses into a multi-task objective to ensure the projector fφ and classifier fΦ perform their respective tasks effectively within the same FSL task as Lvis = Lce + Lkd.

3.4 Synergistic Knowledge Mining

In leveraging the capabilities of large multimodal models, our proposed SynMine first utilizes the rich common-sense knowledge embedded in large language model (LLM) to generate detailed class descriptions. Additionally, SynMine takes advantage of the advanced image-text alignment capabilities of pre-trained visual-language model (VLM) to refine these descriptions into high-quality semantic descriptors, enhancing their relevance for FSL.

As illustrated in Figure 2, SynMine follows a multi-step process to enhance the comprehensive understanding of class characteristics. Inspired by the "chain-of-thought" prompting technique [Wei et al., 2022], which improves LLM performance through intermediate reasoning, we guide LLM using this technique. Initially, a concise definition matching the visual content is provided to the LLM to eliminate ambiguity and help it generate class descriptions focused on visual features. The first prompt is as follows:

Prompt 1: [DEFINITION] is the definition of the [CLASS]. Can you describe the visual features associated with this category?

Here, [DEFINITION] is a brief class definition obtained fromWordNet [Miller, 1995], and [CLASS] refers to the class name. Next, we refine these descriptions by focusing on distinctive visual attributes based on the initial responses. The second prompt is as follows:

Prompt 2: Please describe the [CLASS] in a maximum of five sentences, focusing on discriminative visual features. Make the description more detailed and aligned with scientific facts, avoiding general summaries and subjective interpretations.

The generated descriptions are then processed by the text encoder of a pre-trained VLM, producing high-quality semantic descriptors. This approach is preferred over traditional word embedding models, such as Word2Vec [Mikolov et al., 2013], as it captures deeper contextual meanings and nuances, enhancing the discrimination of visual features across classes.

3.5 Bidirectional Visual-Semantic Bridging

After distilling high-quality semantic descriptors through the SynMine module, we introduce the Visual-Semantic Bridging (VSBird) module, as shown in Figure 3. VSBird employs a dual autoencoder architecture to establish bidirectional mappings within and between the visual and semantic spaces, aiming to reconstruct multimodal classifier weights based on visual embeddings and semantic descriptors. This step is crucial for mapping visual and semantic knowledge into actionable classifier weights for FSL.

In this module, the classifier weights fΦ are treated as visual embeddings encoding distinctive visual knowledge. We then use the corresponding semantic descriptors to facilitate the visual-semantic bridging. Specifically, let Wbase = {wc }c∈Cbase denote the classifier weights and Tbase = {tc }c∈Cbase the semantic descriptors. Our goal is to learn a mapping ΨVS : S → V, where S and V represent the semantic and visual spaces, respectively. The VSBird architecture consists of two encoder-decoder subnetworks: the visual encoder VE : V → Z, the semantic encoder SE : S → Z, the visual decoder VD : Z → V, and the semantic decoder SD : Z → S, with Z as the latent space. These components define the desired semantics-to-weights mapping as ΨVS (tc ) = VD(SE(tc )). To avoid bias towards base classes, we introduce self- and cross-reconstruction objectives. These ensure both modalities preserve their structures within their respective spaces while also aligning the latent spaces. The self-reconstruction objectives minimize the following terms:

Here, cos(·, ·) denotes the cosine distance function. While the two autoencoders preserve structure within their respective spaces, they do not ensure alignment between the two latent spaces. To address this, we introduce two additional crossreconstruction objectives in a symmetric manner:

The final objective of VSBird is a balanced combination of these reconstruction terms:

where α is a weight coefficient used to control the balance between self-reconstruction and cross-reconstruction.

During inference, for a novel class u ∈ Cnovel, the semantic encoder SE and visual decoder VD infer the semanticderived classifier weight:

and for a unseen visual prototype wu , the visual encoder VE and semantic decoder SD infer the visual-derived classifier weight:

3.6 Multi-modal Knowledge Fusion

In the knowledge fusion stage, we create N-way K-shot meta-tasks from the base training set Dbase to mimic the fewshot scenario during testing. The main goal is to develop a visual weight generator and a semantic weight reconstructor to combine visual-based and semantic-based classifier weights, forming robust multimodal classifiers for FSL tasks.

Given the pre-trained few-shot vision encoder f s I , we calculate the visual-based classifier weight wm v for each class m in the support set S as: wm v = 1 ∥∑Ki=1 f s I (xi)∥2 ∑Ki=1 f s I (xi), where K is the number of samples per class m. Next, similar to Equations (6) and (7), we transfer visual and semantic knowledge from SynMine to generate visual-derived classifier weight wm v′ and semantic-derived classifier weight wm s′ for each class m ∈ S. These weights complement each other: the visual-based weight wm v and semantic-derived classifier weight wm s′ complement one another, and similarly, the semantic-based weight wm s and visual-derived weight wm v′ complement each other.

To facilitate visual-semantic weight fusion, we introduce the visual weight generator G and semantic weight reconstructor R. The generator G, consisting of a fully connected layer followed by a sigmoid function, adaptively produces a weight coefficient as β = 1 1+exp(-G(wm s′ )), with val-

ues restricted to the range [0, 1]. This coefficient β is used to balance the contributions of wm s′ and wm v in the visualdominated classifier as wm V = β · wm s′ + (1 - β) · wm v . Similarly, the semantic-based weight wm s and visual-derived weight wm v′ are concatenated and passed through the reconstructor R to form the semantic-dominated classifier as wm S = R (wm s ,wm v′ ) = σ ([wm s · wm v′ ]⊤ W1) W2, where σ is an activation function, and W1 and W2 are learnable weights. Afterward, we generate multimodal classifiers for the N-way K-shot meta-task: WV = {wm V }Nm=1 and WS = {wm S }Nm=1 . Given an image q from the query set, we compute the probabilities Pv and Ps using cosine similarity as Pv = exp(cos(f s I (q),wm V )) ∑Nm′=1 exp(cos(f s I (q),wm′ V )) and Ps = exp(cos(fφ(f s I (q)),wm S )) ∑Nm′=1 exp(cos(fφ(f s I (q)),wm′ S )) . We use the softmax crossentropy loss function to train the generator G and reconstructor R.

Inference. During inference, the visual-dominated classifier and the semantic-dominated classifier are complementary to each other. Therefore, we propose a fusion mechanism to obtain the final class logits. Given a test image xt, the prediction is made as follows:

where λ is a positive balancing coefficient, empirically set to 1K in our SYNTRANS.

4 Experiments

4.1 Datasets

Following the settings in [Zhang et al., 2024], we evaluate the performance of the proposed SYNTRANS framework on four widely used benchmarks in FSL. Two of these datasets are derived from the ImageNet dataset [Russakovsky et al., 2015]: MiniImageNet [Vinyals et al., 2016] and TieredImageNet [Ren et al., 2018]. Another two dataset are CIFARFS [Lee et al., 2019] and FC100 [Oreshkin et al., 2018].

4.2 Implementation Details

Architecture. In all experimental setups, we utilize ResNet-12 and ViT-Small (ViT-S) as the few-shot vision encoders. Specifically, the ResNet-12 encoder is pretrained using the training strategy described in IER [Rizve et al., 2021], while the ViT-S encoder follows the strategy reported in SMKD [Lin et al., 2023]. For the ResNet-12 encoder, visual features are obtained by averaging the outputs from the final residual block, resulting in a feature dimension of 640. For the ViT-S encoder, visual embeddings are computed by averaging the hidden states from the last transformer block, yielding a feature dimension of 384. During the visual knowledge distillation stage, we use both the vision and text encoders from Res50x4 CLIP [Radford et al., 2021] as a strong teacher and employ two linear layers to construct the linear projector fφ. In the SynMine module, we leverage the GPT3.5-turbo model as the large language model and Res50x4 CLIP [Radford et al., 2021] as the visual-language model. The SynMine module facilitates the alignment of features across vision and language modalities. The encoders and decoders of the VSBird module consist of single-layer linear mappings, with ReLU activation following the encoder mappings. A simple fully connected layer serves as the learnable weight generator G. The weight reconstructor R combines visual and textual features using two fully connected layers followed by a LeakyReLU activation function. The hidden layer has a dimension of 2048.

Training Details. During the visual knowledge distillation stage, we freeze both the few-shot vision encoder and CLIP's vision encoder, focusing on optimizing the linear projector and cosine classifier. For ResNet-12, we follow the methods in [Li et al., 2023] and resize the input images to 84 × 84. For ViT-S, we resize the input image to 320 × 320 for MiniImageNet and TieredImageNet, and to 224×224 for CIFARFS and FC100, maintaining consistency with SMKD [Lin et al., 2023]. In the knowledge transfer stage, we freeze all parameters in the SynMine module and only train the parameters of the VSBird module. In the meta-training stage, we train only the parameters of the weight generator and weight reconstructor. For both stages, we employ the Adam optimizer [Kingma and Ba, 2015] with an initial learning rate of 0.0001 and weight decay of 5 × 10-4 . In particular, the VSBird module is trained for 50 epochs with the hyperparameter α set to 0.7, while the weight generator and weight reconstructor are trained for 10 epochs.

Evaluation protocol. The proposed method is evaluated under 5-way 1/5-shot settings on the novel dataset, with 600 few-shot tasks randomly sampled from it. Each task consists of 15 query samples per class. We report the average accuracy (%) with 95% confidence intervals.

4.3 Benchmark Comparisons and Evaluations

Tables 1 and 2 summarize the performance of recent stateof-the-art FSL methods on the MiniImageNet, TieredImageNet, CIFAR-FS, and FC100 datasets, focusing on the 5way 1/5-shot tasks. The experimental results demonstrate that the SYNTRANS framework achieves outstanding performance across all datasets, particularly when visual information is limited. In the 5-way 1-shot scenario, SYNTRANS outperforms the most relevant semantic-based method, SenFew [Zhang et al., 2024], by a margin of 2.98% due to its flexible knowledge transfer framework that mines rich knowledge from diverse large models. Notably, SYNTRANS shows greater improvement in the 1-shot setting compared to the 5-shot setting. In the 5-way 5-shot scenario, SYNTRANS still maintains an advantage over all state-of-the-art methods, though the improvements are less pronounced than in the 1shot setting. This suggests that even with more visual data, high-quality semantic knowledge from large models can still enhance performance. Overall, the results highlight the effectiveness of SYNTRANS in various scenarios, demonstrating its ability to leverage both semantic and visual knowledge.

4.4 Ablation Studies

Influence of Semantic Knowledge Quality. As shown in Table 3, we evaluate the impact of knowledge quality using different sources and encoders. First, we compare common semantics (class names) with generic descriptions from WordNet (i.e., "Short Definitions") and richer descriptions from LLMs via SynMine (i.e., "Rich Descriptions"). Results show that LLM-generated descriptions yield the best performance. This suggests that rich descriptions provide deeper semantic understanding, adding nuanced attributes to the classifier weights that simple class names cannot. Next, we compare Word2Vec models with VLMs for encoding semantic knowledge. VLMs outperform Word2Vec in both 1shot and 5-shot settings, benefiting from their multi-modal training, which enhances semantic understanding. The strong performance of VLMs, particularly when paired with LLMgenerated descriptions, highlights the effectiveness of combining large models for optimal knowledge transfer.

Influence of Hyper-parameter α. The VSBird module is essential for bridging visual and semantic spaces, where the dual autoencoder architecture encourages both selfreconstruction within each space and cross-reconstruction between spaces. Figure 4 shows the performance of the visualdominated classifier WV on MiniImageNet with varying weight coefficients α in Equation (5). A larger α enhances the significance of self-reconstruction loss during VSKB module training. When α is too small, self-reconstruction loss is suppressed by cross-reconstruction loss, leading to lower accuracy. This suggests that while cross-reconstruction is crucial for aligning latent spaces, preserving the individual structures of visual and semantic spaces is equally important. However, beyond α = 0.7 for the 1-shot setting and α = 0.8 for the 5-shot setting, accuracy slightly decreases, indicating that excessive self-reconstruction diminishes the benefits of cross-reconstruction. Thus, balancing self-reconstruction and cross-reconstruction is crucial for optimal performance, with α = 0.7 achieving the best trade-off across settings.

Effect of Multi-modal Knowledge Fusion. As shown in Figure 5, we utilize t-SNE visualization to present classifier weights for all novel categories of MiniImageNet. Figure 5(a) illustrates classifier weights derived solely from visual data in the 1-shot setting, revealing loosely defined clusters with significant category overlap. Conversely, Figure 5(b) shows the 1-shot results with fused visual and semantic knowledge, where clusters are more compact and distinct. This highlights the substantial benefit of semantic knowledge in scenarios with limited samples. Figure 5(c) shows the 5-shot classifier weights based solely on visual data, showing more tightly grouped clusters with less overlap. Figure 5(d) presents the 5shot results with multi-modal knowledge fusion, where clusters are even more distinct and compact. This demonstrates that rich knowledge still improves FSL performance even with a higher number of samples.

5 Conclusion

In this paper, we delve into the previously unexplored potential of harnessing the extensive knowledge available in large multimodal models to empower the pre-trained few-shot learner. As a result, we propose a Synergistic Knowledge Transfer (SYNTRANS) framework that effectively transfers diverse and complementary knowledge from both visual- and large-language models to address FSL tasks. The essence of SYNTRANS lies in its proficient capability to distill and transform explict visual knowledge and implicit semantic knowledge from these large models into practical classifier weights, thereby significantly improving FSL performance through a multimodal knowledge fusion manner. Experimental results on four benchmark datasets demonstrate the superior efficacy of SYNTRANS compared to the state-of-the-art methods.

Acknowledgments

This work is supported by the Shenzhen-Hong Kong-Macao Science and Technology Plan Project (Category C) under the Shenzhen Municipal Science and Technology Innovation Commission (Project No. SGDX20230821092359002), and a grant for Collaborative Research with World-leading Research Groups of The Hong Kong Polytechnic University (Project No. G-SACF). Additional support is provided by the Guangdong Natural Science Funds for Distinguished Young Scholars (Grant No. 2023B1515020097), the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG3-GV-2023-011), and the Lee Kong Chian Fellowships.