A₃B₂: Adaptive Asymmetric Adapter for Alleviating Branch Bias in Vision-Language Image Classification with Few-Shot Learning

1 Introduction

Large-scale Vision-Language Models (VLMs) with dual branches are foundational for downstream transfer learning tasks due to their strong generalization. CLIP [Radford et al., 2021], trained on 400 million image-text pairs, aligns visual and textual modalities in a shared feature space via separate encoders and contrastive learning. However, full finetuning VLMs for specific tasks is computationally expensive and prone to overfitting in few-shot settings.

To address the aforementioned issues, efficient transfer learning methods have been proposed [Zhou et al., 2022b; Yang et al., 2024; Guo and Gu, 2025a; Guo and Gu, 2025b], enabling VLMs to adapt to different downstream tasks with minimal additional parameters. Previous approaches have mainly focused on prompt tuning and adapter-based methods. Prompt tuning aims to address the limitation of VLMs that require manually designed text prompts, which requires expert knowledge and is time-consuming. For instance, in the OxfordPets [Parkhi et al., 2012] dataset, CLIP's text encoder computes class embeddings using the prompt "A photo of a [CLASS], a type of pet.". These embeddings are then matched with the outputs by the image encoder, and the most similar one is selected to predict the image category. Most prior prompt tuning methods [Zhou et al., 2022b; Guo and Gu, 2025a] introduce learnable continuous vectors into the text encoder to dynamically model prompts, optimizing them during training while freezing other parameters, thereby enabling adaptation to specific datasets. In contrast, adapter-based methods [Yang et al., 2024] add lightweight, architecture-agnostic modules, making them easily integrable into different VLMs and allowing the models to adapt to potential downstream task spaces.

∗Corresponding author.

Although prior methods have achieved impressive few-shot performance in data and model adaptation, most CLIP-based methods follow a fixed fine-tuning paradigm: prompt tuning methods optimize the text encoder, while adapter-based methods are attached to the image encoder [Li et al., 2024; Zhang et al., 2024]. This raises a compelling research question: which matters more in different few-shot tasks, CLIP's text or image encoder? (We refer to this as the CLIP's Branch Bias issue) Recent studies have sparked interest in this issue. Yang et al. train only the image encoder while keeping the text encoder fixed, achieving surprising results on compositional understanding and long-text description tasks. Gong et al. propose a kernel-based method, significantly improving the image encoder's performance on zero-shot target recognition, fine-grained spatial reasoning, and localization tasks. Fu et al. find that in vision-centered benchmarks (e.g., depth estimation, correspondence), VLMs perform worse than their image encoders, with performance nearing random levels. In open-vocabulary semantic segmentation tasks, most past methods [Li et al., 2022; Liang et al., 2023; Xu et al., 2023] freeze the text encoder to preserve generalization, while recent studies [Peng et al., 2025] show that jointly fine-tuning both encoders leads to better segmentation results. However, there is no consensus on the Branch Bias issue in the general tasks of image classification.

To address gaps in prior work, we conduct extensive exploratory experiments on three visual tasks: base-to-novel generalization, cross-dataset evaluation, and domain generalization. Crucially, our insights reveal a significant phenomenon: fine-tuning the image encoder does not always yield performance gains and may even degrade robustness on out-of-distribution (OOD) tasks. While a straightforward response to this discovery would be to rigidly lock the image encoder for OOD scenarios, such a hard-coded approach is cumbersome and impractical, as it requires prior knowledge of the target data distribution to manually toggle the adapters. To resolve this dilemma, we propose a novel Adaptive Asymmetric Adapter for alleviating Branch Bias (A3B2) in vision-language image classification. Unlike rigid switching mechanisms, A3B2 achieves adaptability through a uncertainty-aware optimization mechanism. Specifically, we introduce Uncertainty-Aware Adapter Dampening (UAAD). This mechanism dynamically monitors prediction confidence; when the model encounters high-uncertainty samples (indicative of potential OOD data), it automatically suppresses the magnitude of the image adapter's contribution via a regularization penalty. This soft-constraint approach effectively operationalizes our insights without requiringmanual intervention. Structurally, inspired by theMixture of Experts (MoE) design [Fedus et al., 2022; Xue et al., 2022; Tian et al., 2024; Gao et al., 2024a; Zhou et al., 2025], A3B2 consists of a single dimensionality-reduction matrix and multiple dimensionality-expansion matrices, further enhanced by a Load Balancing Regularization to ensure diverse expert utilization [Shazeer et al., 2017].

Our contributions are summarized as follows:

• We systematically explore the Branch Bias issue in image classification for transformer-based CLIP models. Three insights discovered help build the optimal adapter structure for vision-language image classification tasks.
• We propose a task-adaptive, structure-asymmetric adapter to alleviate Branch Bias in vision-language image classification. Empirical evidence, theoretical support for the asymmetric structure, and theoretical guarantees of the method are provided in the extended version, respectively.
• Evaluations on three few-shot tasks show that, across 11 image classification datasets, A3B2 consistently achieves leading performance compared to 11 competitive baselines. The experiments also confirm the generality of our insights.

2 RelatedWork

2.1 CLIP's Branch Bias in Various Tasks

Recent studies have sparked significant interest in the CLIP's Branch Bias issue, particularly in tasks requiring fine-grained visual discrimination. For compositional and spatial tasks, the image branch often requires more adaptation. Yang et al. demonstrate that training only the image encoder while keeping the text encoder fixed yields superior performance in compositional understanding and long-text description. Similarly, Gong et al. propose a kernel-based alignment method that focuses on the image encoder, significantly improving zero-shot target recognition and localization. Fu et al. further reveal that in strictly vision-centered benchmarks (e.g., depth estimation), full VLMs can paradoxically perform worse than their frozen image encoders, suggesting textual interference. In contrast, for open-vocabulary semantic segmentation, the consensus has shifted. While earlier methods [Li et al., 2022; Liang et al., 2023; Xu et al., 2023] froze the text encoder to preserve generalization, recent work by Peng et al. argues that jointly fine-tuning both encoders is essential for aligning pixel-level features with semantic concepts. Despite these diverse findings in structured prediction and reasoning tasks, the manifestation of this Branch Bias in general image classification remains largely underexplored.

2.2 Efficient Transfer Learning in Vision-Language Image Classification

Prompt learning methods enhance the adaptability of VisionLanguage Models (VLMs) by dynamically adjusting pretrained representations. CoOp [Radford et al., 2021] introduces the learnable prompt vector, replacing fixed templates with optimized contexts to improve flexibility, though this sacrifices CLIP's zero-shot generalization. CoCoOp [Zhou et al., 2022a] addresses this limitation by incorporating dynamic instance-conditioned prompts, improving generalization to unseen categories. KgCoOp [Yao et al., 2023] further constrains the semantic gap between learned and handcrafted prompts, preserving general textual knowledge and enhancing model generalization. RPO [Lee et al., 2023] employs a masked attention mechanism to freeze the pretrained representations, reducing internal representation changes and improving model stability, particularly in data-scarce scenarios. MaPLe [Khattak et al., 2023] achieves cross-modal alignment by coupling visual-text prompts, enhancing adaptability to new categories and unseen domains. TCP [Yao et al., 2024] innovatively embeds category semantics into prompt tokens, strengthening generalization for new categories and few-shot settings. MMRL [Guo and Gu, 2025a] proposes a shared learnable representation space to facilitate efficient visual-text interaction, with regularization to prevent overfitting, especially suited for few-shot learning and task adaptation. MMRL++ [Guo and Gu, 2025b] further optimizes this framework by employing a parameterefficient low-rank alignment and progressive feature composition, which jointly facilitate robust generalization and training stability with minimal overhead.

Recent adapter-style methods provide another effective approach for model adaptation, typically integrating lightweight modules into VLMs for downstream task. CLIPAdapter [Gao et al., 2024b] introduces a residual MLP layer at the image encoder's end, significantly improving performance in visual tasks. MMA [Yang et al., 2024] proposes a multimodal adapter method, aggregating image and text features into a shared space, enhancing consistency between modalities and allowing cross-modal gradient flow, thereby improving the model's overall adaptability and task transfer capability. However, none of these methods have systematically investigated the Branch Bias in vision-language image classification, typically adhering to static adaptation paradigms that fail to dynamically balance the dual encoders. Consistent with previous research [Zhou et al., 2022b; Zhou et al., 2022a; Guo and Gu, 2025b], we use CLIP (transformerbased ViT-B/16 backbone) as the base model for our method.

3 Methodology

3.1 Preliminary

The CLIP model consists of an image encoder V and a text encoder T , encoding images and corresponding texts.

Image Encoder The image encoder V consists of L Transformer blocks [Vaswani et al., 2017], denoted as {Vi}Li=1 . The input image I ∈ RH×W×3 is divided into M patches, which are then embedded to obtain the initial features x0:

where x0 ∈ RM×dv (dv is the image embedding dimension). The patch embeddings xi-1, along with a learnable class (CLS) token ci-1, serve as inputs to the next Transformer block Vi and are processed sequentially:

Finally, a patch projection layer maps the CLS token cL from the last Transformer block VL into a shared VisionLanguage (V-L) latent space to obtain the final image rep-

where x ∈ Rd , and d is the feature dimension of the shared V-L space.

Text Encoder For a given text T (e.g., "A photo of a [CLASS]."), it is first tokenized and converted into word embeddings w0 using a Text Embedding layer:

where w0 ∈ RN×dt , with N as the token length and dt as the text embedding dimension.

At each layer, wi-1 is sequentially processed by the next Transformer block Ti to extract text features wi:

The final text representation w is obtained by projecting the last token output from the final Transformer block TL into the shared V-L space via a Text Projection layer:

where w ∈ Rd .

Zero-shot Classification Given the image representation x and text representations {wc}Nc c=1 , where Nc is the number of classes, CLIP computes the cosine similarity between them:

where | · | denotes the L2 norm. The classification probability for each class c is then predicted as:

where τ is a learnable temperature parameter in CLIP. The class with the highest probability is selected as the final prediction.

3.2 A3B2: Adaptive Asymmetric Adapter

Experimental Observations on Branch Bias

We conduct extensive exploratory experiments to investigate CLIP's Branch Bias in image classification tasks. Specifically, we apply 11 diverse image datasets across three widely used image classification tasks (base-to-novel generalization, cross-dataset evaluation, and domain generalization). By fine-tuning a basic adapter (a pair of dimensionality reduction and expansion matrices) on each layer of the image or text encoders in three different CLIP variants (ViT-B/32, ViTB/16, ViT-L/14), we evaluate performance across different tasks. Fig. 1 shows the average performance across these tasks, along with the parameter sizes of the image and text encoders in various CLIP variants. Further details (experimental setup and additional results) are available in the extended version. We summarize the following three insights:

Insight I In various tasks, fine-tuning additional parameters on the image encoder results in lower performance gains compared to the text encoder.

Insight II In in-distribution tasks (e.g., base-to-novel generalization), fine-tuning adapters on both encoders achieves the best performance.

Insight III In out-of-distribution tasks (e.g., crossdataset evaluation and domain generalization), fine-tuning additional parameters on the image encoder may harm the transferability of VLMs on non-distribution data.

Task-adaptive and Structure-asymmetric Adapter

Based on the insights above, we propose an asymmetric architecture where adapters are always active on the text encoder (Insight I), while the image encoder's adaptation is dynamically regulated (Insight II & III).

As shown in Fig. 2, we integrate the A3B2 adapter (details in Fig. 3) into the CLIP backbone. For the text encoder, we modify Eq. 5 by adding the adapter At to each layer:

where underline indicates trainable modules, and α controls the adaptation strength. ReLU(·) is the activation function. W t downi ∈ Rdht×r and W tup ki ∈ Rr×dht , where dht represents the hidden layer dimension of the text encoder and rank r ≪ dht, represent the "Down" matrix and k-th "Up" expert matrix in text encoder layer Ti, respectively. Additionally, a dynamic router computes the expert weights:

where tW g i ∈ Rdht×n , n is the number of experts.

Similarly, for the image encoder, we introduce an adapter Av at each layer to modify Eq. 2:

where W v downi ∈ Rdhv×r and W v up ki ∈ Rr×dhv are the "Down" expert matrix and k-th "Up" matrix of image encoder layer Vi, respectively, with dhv as the hidden dimension of the image encoder. vwi ∈ Rn represents the gating weights from the router:

Rather than rigidly locking the image encoder, we fine-tune the image adapters while constraining their optimization via a novel uncertainty-aware mechanism, described below.

Optimization Objectives

To effectively alleviate Branch Bias and ensure structural diversity within the adapters [Mu and Lin, 2025], we introduce two auxiliary objectives: Uncertainty-Aware Adapter Dampening (UAAD) and Load Balancing Regularization.

Uncertainty-Aware Adapter Dampening Insight III suggests that aggressive modification of image features on out-of-distribution (OOD) data degrades performance. To address this without manual locking, we propose UAAD. This mechanism leverages the model's prediction confidence to gauge whether a sample lies within the distribution. Low confidence implies potential OOD data, triggering a penalty on the image adapter's activation magnitude [Hendrycks and Gimpel, 2016].

Let p(ˆ y|x) be the softmax probability distribution for the image feature x. We define the confidence score as κ = max p(ˆ y|x). The bias alleviation loss Lbias is formulated as:

where B is the batch size, and ∆vi,j = Avi (zj) represents the feature modification vector produced by the image adapter at layer i. When the model is uncertain (low κ), Lbias forces the adapter output ∆v towards zero, effectively reverting the model to the robust pre-trained CLIP features.

Load Balancing Regularization Standard MoE routing often suffer from expert collapse. To prevent this, we enforce a uniform distribution of expert usage across the batch. Unlike methods relying on selection frequency [Fedus et al., 2022], we minimize the variance of the routing probabilities directly. Let ¯wl,j,k (derived from Eq. 10 and Eq. 12) denote the gating probability of the k-th expert for the j-th sample in the l-th layer, averaged over the batch. We adopt a coefficient of variation [Shazeer et al., 2017] based loss:

where 1/n represents the target uniform probability. Minimizing this objective penalizes deviations from a balanced distribution, ensuring diverse expert utilization.

Total Objective The model is primarily optimized using the cross-entropy loss, derived from the classification probabilities defined in Sec. 3.1. For a batch of size B, given the ground-truth label yj for the image future xj , this loss is defined as:

The final training objective is a weighted sum of the crossentropy loss and the auxiliary regularizations:

where λbias governs the suppression strength for uncertain samples, and λbal enforces the diversity of expert utilization. This formulation allows A3B2 to adaptively balance between general linguistic knowledge and task-specific visual adaptation, dynamically alleviating branch bias during training.

4 Experiments

We comprehensively evaluate A3B2's performance around three core image classification tasks in few-shot setting and validate the effectiveness of the discovered insights.

4.1 Experimental Setup

Base-to-Novel Generalization Consistent with previous studies [Brown et al., 2020; Guo and Gu, 2025a], this evaluation divides the dataset into base and novel categories. The model is trained on base categories under a few-shot setting and tested separately on both base and novel categories. This evaluation includes 11 visual classification benchmark datasets: ImageNet [Deng et al., 2009], Caltech101 [FeiFei et al., 2004], OxfordPets [Parkhi et al., 2012], StanfordCars [Krause et al., 2013], Flowers102 [Nilsback and Zisserman, 2008], Food101 [Bossard et al., 2014], FGVCAircraft [Maji et al., 2013], SUN397 [Xiao et al., 2010], DTD [Cimpoi et al., 2014], EuroSAT [Helber et al., 2019], and UCF101 [Soomro et al., 2012].

Cross-Dataset Evaluation This evaluation measures a model's discriminative ability on unseen datasets and its grasp of general knowledge. Following CoCoOp [Zhou et al., 2022a], the model is trained on ImageNet with 1k classes and 16 shots per class, then directly evaluated on 10 other datasets mentioned in the base-to-novel generalization task.

Domain Generalization This task evaluates the model's robustness to domain shifts and its discriminative ability to out-of-distribution data. Following the setup of CoCoOp [Zhou et al., 2022a], we fine-tune the model on ImageNet and test it on four of its variants: ImageNetV2 [Recht et al., 2019], ImageNet-Sketch [Wang et al., 2019], ImageNetA [Hendrycks et al., 2021b], and ImageNet-R [Hendrycks et al., 2021a].

Baselines We compare the proposed A3B2 with 11 competitive methods, including: CLIP (zero-shot) [Radford et al., 2021], CoOp [Radford et al., 2021], CoCoOp [Zhou et al., 2022a], KgCoOp [Yao et al., 2023], RPO [Lee et al., 2023], MaPLe [Khattak et al., 2023], CLIP-Adapter [Gao et al., 2024b], TCP [Yao et al., 2024], MMA [Yang et al., 2024], MMRL [Guo and Gu, 2025a], and MMRL++ [Guo and Gu, 2025b].

Implementation Details Consistent with previous studies [Yang et al., 2024; Guo and Gu, 2025b], all experiments are conducted using a pre-trained transformer-based ViT-B/16 backbone of the CLIP model under a 16-shot setting (i.e., 16 training samples per class). To ensure consistency with prior work [Radford et al., 2021; Zhou et al., 2022a; Yang et al., 2024] as much as possible, we set the batch size to 8, with 5 epochs for ImageNet and 10 epochs for other datasets. The same image augmentation methods are used, and the SGD optimizer is employed with an initial learning rate of 0.0015, a momentum of 0.9, and a weight decay of 0.0005, combined with a cosine annealing scheduler with 1 epoch of warm-up. The text templates for the proposed A3B2 follow [Gao et al., 2024b], with the default number of experts n = 3, low-rank r = 32, coefficient α = 0.001, suppression strength λbias = 0.3 and balance parameter λbal = 0.1. The optimization strategies for other baselines follow the settings of the original papers. For fairness, all baselines are re-implemented under the same experimental setup and trained with mixed-precision acceleration on a single NVIDIA A800 GPU. Final performance is reported as the average over three random seeds (seed=1,2,3).

4.2 Base-to-Novel Generalization

In this task, we report the model's accuracy on both base and novel classes, along with their harmonic mean (HM), as shown in Table 1. Overall, our method consistently outperforms all baselines across all metrics and datasets. The key findings are as follows:

• Among all baselines, the recently proposed MMA and MMRL++ achieve the best average performance. MMRL++ leverages its decoupled inference strategy and progressive inter-layer composition to achieve leading novel-class accuracy with high parameter efficiency. MMA, on the other hand, integrates features from different branches into a shared space, enhancing gradient flow across branches, thus improving consistency between modalities and achieving optimal base-class recognition.
• The proposed A3B2 consistently leads all metrics across 11 datasets. Compared to MMA, which performs best on baseclass recognition, and MMRL++, which excels on the novel and HM metrics, A3B2 shows a significant advantage in average performance. These impressive results validate the effectiveness of our insights.
• In this in-distribution task, methods with dual-branch optimization (e.g., MaPLe, MMA, and MMRL++) generally outperform single-branch optimization methods (e.g., CoOp, CoCoOp, KgCoOp, and TCP), further confirming the universality and broad applicability of our Insight II.

4.3 Cross-Dataset Evaluation

We have compared the top 7 methods in the base-to-novel generalization task with the proposed A3B2 in the crossdataset evaluation task, as shown in Table 2. We observe that MMA performs poorly in our experimental setup, which we attribute to MMA simultaneously adapting the image encoder in out-of-distribution tasks, contrary to our Insight III. And this experiment validates the broad applicability of the Insight III: Text-only encoder optimization methods (e.g., CoOp, CoCoOp, KgCoOp) generally outperform methods with image encoder optimization (e.g., RPO, MaPLe, MMA). Notably, the MMRL and MMRL++ with image encoder optimization even outperform the proposed A3B2 in some cases. We believe these unexpected results stem from their use of a loss-function-driven model exploration strategy during optimization, differing from other methods, whereas we adopt a more general and adaptive structural design based on exploratory observations.

4.4 Domain Generalization

As shown in Table 3, the text-only encoder optimization methods significantly outperform the methods with image encoder optimization, further validating the effectiveness of the Insight III. Meanwhile, we also observe that the MMRL and MMRL++ methods perform poorly on this task, while our A3B2 method consistently achieves leading performance. This suggests that the A3B2, which focuses more on model structure optimization, is more stable, providing strong support for its future application and the insights derived for related research.

4.5 In-depth Analysis

We conduct analytical experiments on 11 datasets for the base-to-novel task; detailed results are in the extended version.

Few-Shot Learning As shown in Fig. 4, A3B2 consistently achieves the best performance across 11 datasets under all shot settings, with its advantage increasing as the number of shots grows. This demonstrates its strong transfer capability and robustness, even in data-scarce few-shot scenarios.

Sensitivity Analysis We analyze the sensitivity of the A3B2 hyperparameters, as shown in Table 4.

Number of Experts n. Too few experts limit model expressiveness, while too many may lead to overfitting. The optimal balance is achieved when n = 3.

Low-rank r. The low-rank r faces a similar trade-off to the number of experts. Performance is best when r = 32.

Balance Coefficient α. The coefficient α controls the relationship between pre-trained and learnable weights, which are more effective for base and novel settings, respectively. A good balance is achieved when α = 0.001.

Suppression Strength λbias. This term controls the strength of uncertainty-aware dampening on image adapters. Moderate suppression (λbias = 0.3) best balances robustness and adaptability, while overly large values over-constrain visual adaptation.

Balance Parameter λbal. This parameter regulates expert utilization diversity. A moderate value (λbal = 0.1) effectively prevents expert collapse and yields the most stable performance across base and novel classes.

Ablation Study As shown in Fig. 5, removing any component degrades performance, confirming their effectiveness. Notably, removing Lbias + Lbal leads to the largest drop, highlighting their synergy. Applying Lbias to the text encoder ("w. bi-encoder Lbias") also causes a significant decline, validating the applicability of Insight II & III.

We provide a cost analysis of A3B2 and additional experiments on ResNet-based CLIP in the extended version.

5 Conclusion

This paper identifies the Branch Bias issue in few-shot visionlanguage classification, showing that fine-tuning the image encoder can be detrimental to out-of-distribution robustness. We propose A3B2, an adaptive asymmetric adapter that modulates the image branch based on prediction uncertainty. Extensive experiments demonstrate that A3B2 consistently improves performance across various few-shot settings, providing an effective solution for branch-aware adaptation.

Acknowledgments

This research was supported by grants from the "Pioneer" and "Leading Goose" R&D Program of Zhejiang under Grant No. 2025C02022 and the National Natural Science Foundation of China (No.62307032).