CoDA: Co-adaptive Dual-path Alignment for Vision-Language Models

1 Introduction

Vision-language models (VLMs), modeling visual and textual information jointly, have enabled unified vision-language understanding. By contrastively training on large-scale image-text pairs, VLMs like CLIP [Radford et al., 2021] learn a shared embedding space, where semantically related image-text pairs are pulled closer while unrelated ones are pushed apart. Trained on over 400 million image-text pairs, CLIP shows remarkable zero-shot generalization across diverse downstream tasks.

Despite their strong generalization ability, the massive amount of parameters of VLMs makes full fine-tuning computationally expensive and prone to overfitting, especially in low-data regimes such as few-shot learning. To address this issue, adaptation strategies that freeze the backbone and introduce a small set of task-specific parameters, including adapter-based methods [Gao et al., 2024; Yu et al., 2023; Li et al., 2023] and prompt-learning methods [Zhou et al., 2022a; Khattak et al., 2023a; Ouali et al., 2023], have become the dominant choice under limited supervision.

During adaptation, the importance of explicitly modeling cross-modal alignment has been stressed by recent studies. For shared representation of modalities, representative works like MMA [Yang et al., 2024] introduce a multimodal adapter that shares weights between the image and text branches. However, under limited supervision during adaptation, overly aggressive cross-modal alignment can distort the intrinsic structure of intra-modal representations, leading to degraded generalization.

We argue that robust VLM adaptation requires coordinately addressing two coupled yet distinct objectives: (i) cross-modal alignment, which governs how semantic information should be adequately exchanged between modalities, and (ii) intra-modal consistency, which governs how the intra-modal structure learned from large-scale pre-training should be preserved. Neglecting either objective may result in over-coupled representations or structurally degenerate features, particularly in few-shot and distribution-shift settings.

To this end, we propose Co-adaptive Dual-path Alignment (CoDA), a new adaptation framework that explicitly disentangles and coordinates cross-modal semantic alignment and intra-modal structural consistency. Rather than collapsing the two objectives into a single one during adaptation, CoDA simultaneously promotes cross-modal semantic interaction and calibrates intra-modal feature structure.

Specifically, we first introduce a channel-wise module, Decoupled Squeeze-and-Excitation (dcSE), that reduces modality discrepancy prior to our proposed dual-path alignment. dcSE aggregates token features into global channel statistics and projects them to a shared semantic space, enabling crossmodal interaction at the level of semantic responses rather than explicit token or instance alignment. This design avoids fragile one-to-one token-level alignments under ambiguity and domain shift.

Then building on dcSE, we propose a dual-path module, Distribution-Structure Co-adaptation (DiSCo), to refine intermediate representations. Concretely, a β-path computes a distribution-aware cross-modal bias (an additive semantic offset) via entropy-regularized optimal transport, enabling soft cross-modal semantic emphasis without enforcing hard token alignment. In parallel, a γ-path summarizes dominant intra-modal structure using principal component analysis and converts it into channel-wise calibration signals, preserving intra-modal structure during adaptation.

Finally, inspired by hierarchical human perception, we introduce a Parent Class Generation (PCG) strategy that augments labels with LLM-generated parent concepts. This hierarchical prompt design injects coarse-to-fine semantic priors into textual representations and improves alignment stability under limited supervision.

Our contributions are summarized as follows:

• We propose CoDA, a new vision-language adaptation framework achieving both cross-modal semantic alignment and intra-modal structural consistency.
• We introduce dcSE and DiSCo, two complementary modules that operate at channel and feature levels, respectively, enabling controlled semantic exchange and structure-aware representation refinement.
• We further propose a PCG mechanism that injects hierarchical semantic priors via LLMs, improving robustness under few-shot and distribution-shift scenarios.

2 RelatedWork

2.1 Vision-Language Models

Large-scale VLMs pre-trained on web-scale image-text pairs have become a cornerstone for multi-modal understanding. A representative example is CLIP [Radford et al., 2021], which learns a shared image-text embedding space via contrastive learning, exhibiting strong zero-shot generalization across diverse downstream tasks. Building upon CLIP, subsequent models such as ALIGN [Jia et al., 2021], FILIP [Yao et al., 2021], and SigLIP [Zhai et al., 2023] further improve vision-language representations by scaling training data and model capacity, refining contrastive objectives, or enhancing token-level interactions. Despite these advances, fine-tuning large VLMs to downstream tasks remains challenging due to their scale and sensitivity to limited supervision, motivating parameter-efficient adaptation strategies.

2.2 Parameter-efficient Adaptation of VLMs

Parameter-efficient adaptation techniques originally developed in NLP have been extended to VLMs [Zhou et al., 2022a; Khattak et al., 2023a; Gao et al., 2024]. Promptlearning methods adapt VLMs by optimizing learnable context tokens in the text encoder. CoOp [Zhou et al., 2022b] replaces handcrafted prompts with continuous learnable vectors, while CoCoOp [Zhou et al., 2022a] further conditions prompts on image features to improve generalization. Subsequent works explore multi-modal prompts [Khattak et al., 2023a] and regularization strategies to mitigate overfitting under limited supervision. Adapter-based methods instead introduce lightweight trainable modules into frozen backbones. CLIP-Adapter [Gao et al., 2024] and Tip-Adapter [Zhang et al., 2021] focus on refining visual representations, whereas more recent approaches inject adapters into both visual and textual branches to enable multi-modal adaptation [Jiang et al., 2022; Yang et al., 2024]. However, most of these methods focus merely on final embedding alignment, without explicitly regulating cross-modal distribution discrepancies or intra-modal representation structure.

2.3 Modality Alignment for Adaptation of VLMs

Recent studies have highlighted the importance of explicitly modeling modality alignment during VLM adaptation. Representative methods such as MMA [Yang et al., 2024] enforce mutual alignment between visual and textual representations by encouraging consistency across modalities during adaptation. Multi-modal prompt learning approaches [Khattak et al., 2023a] also improve alignment by jointly optimizing prompts in both visual and textual branches. Other works implicitly promote alignment through instance-level conditioning or shared embedding supervision. Despite their effectiveness, existing alignment-based methods predominantly assume that the internal structure of each modality remains stable during adaptation. Such assumptions may be fragile under few-shot or distribution-shifted settings, where aggressive cross-modal coupling can distort intra-modal representations. In contrast, our method disentangles cross-modal alignment from intra-modal structural calibration and coordinates them, enabling controlled semantic interaction while preserving the intrinsic intra-modal structure during adaptation.

3 Methodology

As illustrated in Figure 1, we propose CoDA, a new unified vision-language adaptation framework that explicitly coordinates both cross-modal semantic alignment and intra-modal structural consistency. Coordinating these two objectives is critical for robust adaptation under limited supervision.

3.1 Decoupled Squeeze-and-Excitation (dcSE)

In multi-modal adaptation, na¨ıve feature fusion or tightly coupled attention often leads to over-coupling between modalities, where modality-specific representations are suppressed. While conventional Squeeze-and-Excitation (SE) layers [Hu et al., 2018] effectively recalibrate channel responses within a single modality, they neither support semantic exchange across modalities nor preserve intra-modal structure during adaptation.

To address these limitations, we propose Decoupled Crossmodal Squeeze-and-Excitation (dcSE), a channel-wise module that enables controlled semantic exchange between visual and textual representations while avoiding aggressive crossmodal alignment. As illustrated in Fig. 1, dcSE consists of two stages. First, a decoupled squeeze that summarizes intramodal channel statistics and projects them into a shared feature space, making their representations more comparable. Then, a decoupled excitation that transforms this shared representation back into modality-specific channel responses.

Let X(l) ∈ RB×T×C denote the intermediate features from the l-th Transformer block, where B is the batch size, T is the number of tokens, and C is the channel dimension. dcSE first aggregates token-level features into a global channel descriptor by average pooling over the token dimension:

The descriptor is then projected into a shared semantic subspace of dimension dcommon:

Within the shared semantic subspace, dcSE applies two consecutive SE-style transformations to balance modalityspecific adaptation and cross-modal semantic consistency. Both PrivateSE and SharedSE adopt the standard two-layer bottleneck structure of SE blocks. Specifically, Wp1,Ws1 ∈ Rdr×dcommon and Wp2,Ws2 ∈ Rdcommon×dr are learnable projection matrices, where dr denotes the reduced channel dimension. PrivateSE uses modality-specific parameters, whereas SharedSE shares parameters across modalities to enforce semantic consistency.

The PrivateSE operation uses modality-specific parameters and is defined as

where ϕ(·) denotes the ReLU activation and p(l) ∈ RB×dcommon .

The subsequent SharedSE operation shares parameters across modalities to enforce semantic consistency:

where σ(·) denotes the sigmoid function and q(l) ∈ RB×dcommon .

Finally, the shared semantic signal is mapped back to the original channel space through a modality-specific excitation:

which is broadcast to all tokens and applied channel-wise:

By decoupling semantic aggregation and channel excitation, dcSE enables soft cross-modal semantic interaction at the level of global channel statistics, rather than enforcing explicit token- or instance-level alignment, while preserving intra-modal feature modulation.

3.2 Distribution-Structure Co-adaptation (DiSCo)

To further refine intermediate representations after dcSE, we introduce Distribution-Structure Co-adaptation (DiSCo), a dual-path module that operates directly in the feature space. Rather than explicitly fusing visual and textual features, DiSCo produces complementary features from two parallel paths: (i) a β-path that captures cross-modal semantic bias through distribution-aware aggregation; and (ii) a γ-path that performs structure-aware channel calibration to preserve intra-modal structure. These features are subsequently injected back into the original representations.

β-path. For each sample, we extract a global semantic descriptor from the current modality using the [CLS] token, denoted as h(b) self ∈ RC , b ∈ {1, . . . ,B}. From the other modality, we collect the token-level features as a set of semantic bases H(b) other = {h(b,j) other }Tj=1 , where T denotes the number of tokens and j indexes tokens in the other modality. These bases are taken from the intermediate token features of the other modality at the same block after dcSE.

Although dcSE brings visual and textual features into a shared semantic space, correspondence between individual tokens across modalities remains inherently ambiguous. Different modalities may emphasize different attributes or components of the same concept, and spurious correlations under domain shift further undermine the reliability of one-toone token matching. As a result, enforcing explicit or hard alignment risks amplifying early mismatches and destabilizing adaptation. Moreover, since h(b) self summarizes global semantics, collapsing it onto a single token basis is unnecessarily restrictive; a distributed association better reflects that multiple tokens may jointly represent the same concept.

The β-path models cross-modal interaction at the level of semantic emphasis. Specifically, we treat the global descriptor h(b) self as distributing its semantic contribution over the token bases {h(b,j) other } from the other modality, yielding a soft and distributed form of alignment.

To this end, we first measure semantic discrepancy using cosine distance:

where smaller values indicate higher semantic affinity. We then formulate semantic emphasis alignment as a mass redistribution problem, in which a unit amount of semantic contribution from h(b) self is softly allocated over the token bases according to Mbj .

This formulation naturally leads to entropy-regularized optimal transport (OT) [Cuturi, 2013], which yields a smooth and normalized transport plan under a mass-conservation constraint, preventing degenerate concentration on a single token and providing an interpretable distribution of semantic emphasis. For each sample b, we solve

where M (b) ∈ R1×T is the b-th row of M, 1 ∈ RT is an allones vector, and H(Γ(b) ) = - ∑Tj=1 Γ(b) j log (Γ(b) j ) is the entropy regularizer with ε > 0 controlling the smoothness.

The resulting transport weights Γ∗(b) encode how strongly each token basis contributes to cross-modal adaptation. We aggregate them into a distribution-aware semantic bias:

Injected as an additive bias, β encourages the two modalities to emphasize consistent semantic content without enforcing hard alignment.

γ-path. While the β-path encourages cross-modal semantic alignment, adaptation may still distort the intrinsic intramodal structure learned from pre-training, especially under domain shift. The γ-path therefore derives a structure-aware calibration signal from token statistics to stabilize channel responses within each modality.

Let X ∈ RB×T×C denote the intermediate token features of the current modality. For each sample b, we view X(b) = Xb,:,: ∈ RT×C as T observations in a C-dimensional channel space and compute a covariance statistic Σ(b) = 1 T ( ˆX(b) )⊤ ˆX(b) ∈ RC×C , where ˆX(b) = X(b) - 1(¯x(b) )⊤ and ¯x(b) = 1 T ∑Ti=1 X(b) i,: . We obtain the dominant structural direction u(b) ∈ RC as the leading eigenvector of Σ(b) , approximated by a few steps of power iteration [Golub and Van Loan, 2013]. Its strength is measured by the Rayleigh quotient λ(b) = (u(b) )⊤Σ(b)u(b) , and the channel-wise calibration vector is defined as

The resulting γ is broadcast over tokens and applied as a multiplicative modulation signal, highlighting structurally salient channels and reducing intra-modal structural drift during adaptation.

Feature Integration. Finally, the outputs of the two paths are injected into the original token-level features X:

Through the complementary effects of β and γ, DiSCo refines representations by coordinately leveraging cross-modal semantic distributions and intra-modal structural cues, without enforcing hard feature alignment.

3.3 Parent Class Generation

To further enhance textual semantics with hierarchical knowledge, we propose a parent class generation strategy based on LLMs. This design is motivated by human hierarchical cognition: when recognizing unfamiliar objects, humans often reason from coarse categories to fine-grained concepts.

Given a label set C = {c1, . . . , cN}, we employ DeepSeek [Liu et al., 2024] to infer a semantically meaningful parent concept for each class using carefully designed prompts, e.g., "Based on function or form, assign {class name} to a broad object category." For each class cn, the LLM generates a concise parent label pn, such as

Each class is paired with its parent concept to construct a hierarchical textual prompt:

where {vk} are learnable context tokens.

4 Experiments

To evaluate our method, we follow the experiment setup established in previous works such as MaPLe [Khattak et al., 2023a] and HPT [Wang et al., 2024]. We compare the performance of CoDA with the baseline zero-shot CLIP [Radford et al., 2021] and state-of-the-art (SOTA) fine-tuning methods, including Co-CoOp [Zhou et al., 2022a], MaPLe [Khattak et al., 2023a], promptSRC [Khattak et al., 2023b], MetaPrompt [Zhao et al., 2024], HPT [Wang et al., 2024], 2SFS [Farina et al., 2025] and FCPrompt-G [Liu et al., 2025]. Comprehensive ablation studies are also conducted to demonstrate the effectiveness of each component in CoDA.

4.1 Datasets

We report results on 11 datasets used in CoOp [Zhou et al., 2022b]: ImageNet [Deng et al., 2009], Caltech [FeiFei et al., 2004], OxfordPets [Parkhi et al., 2012], StanfordCars [Krause et al., 2013], Flowers [Nilsback and Zisserman, 2008], Food101 [Bossard et al., 2014], FGVCAircraft [Maji et al., 2013], EuroSAT [Helber et al., 2019], UCF101 [Soomro et al., 2012], DTD [Cimpoi et al., 2014] and SUN397 [Xiao et al., 2010].

4.2 Implementation Details

We use ViT-B/16 as the vision backbone and perform prompt tuning on the pre-trained CLIP model. For base-to-new generalization, we optimize the model using SGD with an initial learning rate of 0.017 for 10 epochs and a batch size of 32. Following prior work, we compute image-text similarity scores using the CLIP logit scale and apply a softmax to obtain class probabilities, and train with cross-entropy loss. In accordance with CoOp, we use 16 shots for training and evaluate on the full test set.

4.3 Quantitative Performance on Various Tasks

Base-to-novel generalization. Similar to existing CoOpbased methods, an important evaluation of fine-tuning involves a base-to-new generalization setting composed of two disjoint subsets: base classes and novel classes. The trainable parameters are obtained by using labeled base classes and evaluated on both base classes and unseen novel classes.

Table 1 reports the base-to-new generalization results across 11 datasets. CoDA consistently improves performance on novel classes while maintaining competitive accuracy on base classes, leading to a superior harmonic mean compared with existing methods.

Few-shot learning. Table 2 presents the few-shot learning results under 1, 4 and 16-shot settings. It shows that as the number of shots increases, our method achieves a higher average accuracy across 11 datasets. While our method does not always achieve the highest accuracy on every dataset, it generally ranks among the top performers, demonstrating strong overall performance. These results show that CoDA performs robustly in low-shot settings. By leveraging distributionaware semantic cues and structure-aware calibration, the proposed method mitigates overfitting under limited supervision, and exhibits steady performance gains as more training samples become available.

Domain generalization. In Table 3, we compare the performances on the domain generalization task following prior benchmarks [Roy and Etemad, 2023; Park et al., 2024]. CoDA achieves the best average accuracy of 60.97%, demonstrating its generalization capability across different domain shifts.

4.4 Ablation Study

Effects of DiSCo, dcSE, and PCG. We conduct ablation studies to examine the individual and combined effects of the proposed components including DiSCo, dcSE, and PCG.

As shown in Table 4, each component contributes positively to overall performance, demonstrating their effectiveness. When all components are jointly enabled, CoDA achieves the best harmonic mean. In particular, DiSCo and dcSE yield substantial improvements on novel classes, indicating enhanced robustness under distribution shift.

Notably, PCG provides moderate gains when used alone, but becomes significantly more effective when combined with DiSCo or dcSE, suggesting that hierarchical semantics are best leveraged under explicit cross-modal interaction.

Effect of cross-modal interaction. To further examine the role of cross-modal interaction, we evaluate two additional variants that restrict adaptation to a single modality: text-only, where only the textual branch is adapted, and vision-only, where adaptation is applied exclusively to the visual branch. Both variants remove explicit cross-modal interaction while preserving comparable parameter budgets.

As reported in Table 5, single-modality adaptation consistently underperforms the full CoDA model. In particular, vision-only adaptation suffers from noticeable degradation on novel classes, highlighting the difficulty of generalization without textual guidance. Text-only performs better than vision-only on novel classes, suggesting textual adaptation is particularly important for generalizing to unseen categories. In contrast, jointly adapting both modalities leads to the best overall performance, confirming that CoDA's gains stem from coordinated vision-language co-adaptation rather than stronger uni-modal tuning.

Effect of the depth of tuning layers. We further study how the depth of applying CoDA affects base-to-new generalization. Table 6 reports results from the baseline model (without CoDA) to progressively deeper adaptation ranges, where "12" denotes adapting only the last Transformer block and "10→12" (resp. "7→12", "2→12") denotes adapting the last three (resp. six, eleven) blocks in both branches.

Without CoDA, the model achieves strong base accuracy but generalizes poorly to novel classes. Adapting only the last block substantially improves novel accuracy, but causes a notable drop on base classes, indicating that shallow adaptation can overly perturb the decision boundary learned for base classes. As the adaptation depth increases (10→12, 7→12, 2→12), base accuracy steadily recovers while novel accuracy continues to improve, yielding consistent gains in harmonic mean. Applying CoDA to all layers achieves the best overall results, suggesting that CoDA benefits from stable and cumulative refinement across depth rather than relying on a singlelayer correction.

More effect of PCG. We further analyze PCG to verify that its gains come frommeaningful semantic guidance rather than a superficial prompt change. Table 7 compares three variants: (i) removing PCG (w/o PCG), (ii) using "xxx" as the incorrect parent token for all classes (Incorrect PCG), and (iii) the full CoDA with semantically correct PCG.

Introducing an incorrect PCG already improves over w/o PCG, suggesting that a hierarchical prompt scaffold can act as a useful inductive bias. Replacing incorrect PCGs with semantically meaningful ones consistently yields further gains on novel classes and the harmonic mean, indicating that semantic correctness is crucial for robust generalization. We further visualize class-center embeddings with t-SNE in Figure 2, in which correct PCG produces more compact clusters, supporting the quantitative improvements observed in Table 7.

4.5 Visualization of Saliency Consistency

To further analyze how CoDA adapts vision-language representations at the perceptual level, we visualize saliency heatmaps produced by the visual encoder under CoDA. Figure 3 presents representative examples across four datasets.

Despite notable variations in pose, background and appearance, CoDA exhibits highly consistent saliency patterns within each category. For instance, in OxfordPets-Beagle, the attention is consistently concentrated on the head and facial regions, while in Caltech101-Face and ImageNet-Tench, the model focuses on the most discriminative semantic regions across different instances. Similarly, for OxfordFlowersWallflower, saliency maps highlight characteristic petal and floral structures rather than background textures.

These observations suggest that CoDA does not rely on spurious visual cues, but instead learns category-level semantic features that are stable across samples. Such intra-class saliency consistency complements the quantitative gains reported earlier, indicating that CoDA improves generalization by encouraging structured and semantically grounded feature adaptation, rather than instance-specific memorization.

5 Conclusion

We proposed CoDA, a new dual-path adaptation framework that effectively coordinates cross-modal semantic alignment and intra-modal structural consistency for adapting VLMs.

CoDA consists of three components: dcSE performs decoupled squeeze-excitation to enable controlled semantic exchange via global channel modulation; DiSCo refines intermediate representations with distribution-aware cross-modal semantic bias and structure-aware intra-modal calibration; and PCG injects hierarchical semantic priors into textual prompts.

Extensive experiments show that CoDA achieves consistent improvements over SOTA methods on base-to-new generalization and few-shot learning across multiple datasets. Ablation studies validate the contribution of each component and their complementary effects. Overall, our results suggest that coordinately modeling cross-modal alignment and intramodal structure is key to robust and transferable adaptation of VLMs.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (Grant 62225601, U23B2052), the Beijing Natural Science Foundation Project No.L242025, the Gansu Province Top Leading Talents Foundation, the Scientific Research Innovation Capability Support Project for Young Faculty under Grant SRICSPYF-BS2025009, the Gansu Provincial Key Talent Project under Grant 2025RCXM002, the Fundamental Research Funds for the Beijing University of Posts and Telecommunications under Grant 2025AI4S15, the Guizhou Province Science and Technology Plan Project (No.QianKeHeZhongDa [2025]031), the Beijing Key Laboratory of Multimodal Data Intelligent Perception and Governance, and the Royal Society under International Exchanges Award IEC\NSFC\201071.