DSSG: Dual-Stream Semantic Guidance for Source-Fully-Free Adaptation of Vision-Language Models

1 Introduction

Traditional unimodal source-free domain adaptation (SFDA) follows a two-stage pipeline of source-domain pre-training and target-domain adaptation. While effective, this paradigm heavily relies on task-specific source priors and is restricted by practical deployment bottlenecks, such as data privacy concerns and model black-box constraints (e.g., API-only access). Recently, Vision-Language Models (VLMs) represented by CLIP, pre-trained on large-scale image-text pairs, exhibit strong zero-shot generalization, serving as generalized source models rich in transferable knowledge. To eliminate dependency on task-specific source training, we focus on Source-Fully-Free Domain Adaptation (SFF-DA)-also known as Unsupervised Fine-Tuning (UFT) [Li et al., 2025]. We directly leverage VLMs for adaptation on unlabeled target domains without accessing any source data or models, aiming to transfer their inherent universal knowledge to domainspecific contexts under distribution shifts.

*Corresponding authors.

While parameter-efficient methods (e.g., Prompt Learning [Zhang et al., 2022b], Adapters [Zhang et al., 2022a]) preserve VLM priors by freezing backbones, their limited learnable parameters hinder the acquisition of intricate domain-specific knowledge. In contrast, directly fine-tuning foundation models in SFF-DA encounters a twofold challenge: (i) error accumulation and catastrophic forgetting (Fig. 1(b,c)), where the lack of source guidance accumulates pseudo-label errors, exacerbating confirmation bias and category collapse, thereby corrupting large-scale pre-trained knowledge; and (ii) evolutionary asymmetry and static semantic drift (Fig. 1(a, d)), where the vision encoder evolves toward a domain-specific manifold while template-derived embeddings wk remain fixed in the frozen text space, inducing a spatiotemporal misalignment that undermines crossmodal consistency, termed static semantic drift.

Existing CLIP-based SFDA methods generally follow two paradigms. Firstly, CLIP-assisted approaches (e.g., DIFO, ProDe [Tang et al., 2024b; Tang et al., 2025]) employ CLIP as an external teacher to calibrate task-specific source models. However, the reliance on pre-trained source weights limits their applicability in SFF-DA. Secondly, the unsupervised fine-tuning paradigm adapts VLMs by optimizing prompt variables [Long et al., 2024], the vision encoder [Mirza et al., 2023], or joint tuning [Hu et al., 2024]. While updating encoders improves adaptability, it is prone to damaging universal priors. This leads to a fundamental stability-plasticity dilemma: prompt-based tuning lacks the plasticity for targetspecific distributions, while encoder-based tuning damages universal pretrained knowledge.

Motivation: Drawing inspiration from ImCapDA [Xiang et al., 2025], we consider generating descriptive captions as semantic proxies to jointly fine-tune encoders, which has proven effective in enforcing cross-modal alignment. However, in the rigorous SFF-DA scenario lacking source label supervision, this introduces a grave risk of dynamic semantic drift. Generated captions exhibit an inherent bias towards non-discriminative instance-level details or background noise, inducing an unconstrained semantic flow that causes visual features to diverge from their intrinsic class manifolds. This semantic divergence, coupled with the rigidity of fixed class embeddings wk, constitutes a Dual Semantic Drift dilemma. It hinders the model from reconciling the capture of domain-specific with the preservation of global categorical structures.

To address this, we present the Dual-Stream Semantic Guidance (DSSG) framework. First, the DSG module integrates a caption stream and a class-anchor stream to construct a complementary semantic space-utilizing captions to provide instance-specific context for mitigating forgetting, while employing anchors to preserve intrinsic category semantics. Second, the Dynamic CMKD module calibrates the evolving teacher distribution via self-distillation to enforce teacherstudent consistency. This coupled and complementary design ensures robust cross-modal alignment and global categorical consistency via dual semantic guidance.

Our main contributions are summarized as follows:

1. We identify Dual Semantic Drift-the conflict between static embedding rigidity and caption divergence-as a core stability-plasticity dilemma in SFF-DA.
2. We introduce the DSSG framework, which integrates parallel semantic guidance with a dynamic distillation module to reconcile fine-grained adaptation and global consistency.
3. Extensive experiments on mainstream benchmarks demonstrate that our method significantly outperforms existing approaches, establishing a new state-of-the-art.

2 RelatedWork

Unimodal Source-Free Domain Adaptation. Unimodal SFDA approaches typically rely solely on visual features, leveraging source prior models for self-training on unlabeled targets. SHOT [Liang et al., 2020] and AaD [Yang et al., 2022] employ mutual information and clustering for feature alignment. For robustness, TPDS [Tang et al., 2024a] and DIPE [Wang et al., 2022] investigate distribution flow search and domain-invariant parameter mining, respectively. Recent strategies explore Transformer adaptability [Sanyal et al., 2023] and advanced augmentation [Hwang et al., 2024], while TiGM [Zhu et al., 2025] uses multi-dimensional metrics for pseudo-label selection. However, this unimodal limitation leads to a lack of explicit semantic guidance, limiting generalization across complex domain shifts.

Vision-Language Model Assisted SFDA. Recent works leverage CLIP as an external teacher to guide knowledge transfer via two primary paradigms: (1) Prediction Calibration: Co-learn++ [Zhang et al., 2025] and DPC [Zhan et al., 2024] ensemble zero-shot priors within dual-branch frameworks, while ProDe [Tang et al., 2025] utilizes CLIP as a noisy agent for probabilistic denoising. (2) Structural Alignment: DIFO [Tang et al., 2024b] maximizes mutual information for alignment, while MMGA [Chen et al., 2025] and DTKI [Zhan et al., 2026] enforce topological consistency to anchor features to semantic prototypes. Yet, requiring source supervision and complex auxiliary deployment restricts their SFF-DA feasibility and yields suboptimal performance compared to direct VLM adaptation

Multimodal Source-Fully-Free Domain Adaptation (SFFDA). Multimodal SFF-DA directly fine-tunes VLMs on unsupervised target domains. Existing works follow three paradigms: (1) Prompt Tuning, which optimizes learnable prompts with frozen backbones (e.g., UPL [Huang et al., 2022], TFUP [Long et al., 2024]); (2) Visual Tuning, which fine-tunes only the visual encoder to align with fixed textual features (e.g., LaFTer [Mirza et al., 2023], DPA [Ali et al., 2025]); and (3) Joint Fine-tuning, which optimizes both visual and text encoders for cross-modal feature alignment (e.g., ReCLIP [Hu et al., 2024], POUF [Tanwisuth et al., 2023], ImCapDA [Xiang et al., 2025]). However, these methods face the Dual Semantic Drift dilemma: visual-only tuning leads to static semantic drift due to fixed class embeddings, whereas caption-based joint tuning induces dynamic semantic drift under unconstrained semantic flow.

3 DSSG Framework

We propose the DSSG framework, which integrates parallel semantic guidance with a dynamic distillation module to address the Dual Semantic Drift dilemma in SFF-DA (Fig. 2). The framework consists of VLM-based caption generation (Sec. 3.2), the Dual-Stream Guidance (DSG) module (Sec. 3.3), and Dynamic CMKD (Sec. 3.4).

3.1 Preliminaries and ProblemDefinition

SFF-DA Setting: Given an unlabeled target dataset Dt = {xi}Nt i=1 , S SFF-DA adapts a pre-trained vision-language model Θclip via unsupervised fine-tuning, without access to either source data Ds or source model Θs.

Revisiting CLIP: CLIP comprises a visual encoder EV and a text encoder ET , jointly trained to align images and text into a shared feature space via large-scale contrastive pretraining. During inference, given a category set C = {ck}Kk=1 , fixed class embeddings wk = ET (tk) are computed from prompt templates tk (e.g., "a photo of a ck"). For an image xi with feature fi = EV (xi), the prediction probability is defined as:

where sim(·, ·) is the cosine similarity and τ is the temperature parameter.

3.2 Data Priors: Generative Instance-level Captions

In SFF-DA, fixed class embeddings wk derived from templates tk remain anchored in the static semantic space, failing to track dynamic visual manifolds and inducing evolutionary asymmetry. Furthermore, fine-tuning the vision and text encoders {θV , θT } without source guidance is prone to category collapse and catastrophic forgetting. To mitigate these issues, we leverage the generative captioning strategy from ImCapDA [Xiang et al., 2025], proven to facilitate domainspecific knowledge learning and alleviating such forgetting. Specifically, a frozen VLM generates a caption stream for each unlabeled target sample xi ∈ Dt to construct instancelevel priors:

where advanced VLMs (e.g., BLIP-3 [Xue et al., 2024]) capture rich instance-level context, such as background textures and object attributes. Crucially, contrastive alignment of target images with the generated caption stream is essential to drive parameter updates in ET (·; θT ). This process enables coupled evolution of the semantic space with the visual manifold to mitigate static drift, acting as a semantic soft constraint to counteract confirmation bias. Consequently, Tcap,i serves as the critical input for our DSSG framework.

3.3 Dual-StreamGuidance

While Tcap introduces domain context, relying solely on its unconstrained semantics inherently induces dynamic divergence, thereby trapping the model in a dual semantic drift dilemma. Diverging from prior paradigms constrained to either fixed prompt templates [Zhou and Zhou, 2024] or caption-only guidance [Xiang et al., 2025], we introduce a Dual-Stream Guidance (DSG) mechanism that integrates a caption stream and a class-anchor stream in parallel. Specifically, for a target image xi, its visual representation fi = EV (xi; θV ) is aligned with two complementary textual embeddings encoded by a shared text encoder θT :

where ganc is derived from class-generic prompt templates {tk}Kk=1 and preserves intrinsic category semantics, thereby providing global semantic rigidity against dynamic semantic drift. Conversely, gcap,i captures instance-specific knowledge from VLM-generated descriptions and introduces local semantic plasticity to alleviate static semantic drift. This dualstream design maintains cross-modal alignment within a stable semantic manifold.

To jointly fine-tune θV , θT , we adopt a self-distillation paradigm following prior works [Zhou and Zhou, 2024;

Xiang et al., 2025], which employs a similarity-based teacher predictor and a linear student classifier h(·). Within this framework, we formulate objectives that exploit the dynamic adaptability of the caption stream while enforcing the semantic rigidity of the anchor stream:

Caption-guidedContrastiveAlignment. To promote finegrained cross-modal feature alignment, we employ a symmetric InfoNCE loss that maximizes the similarity of matched image-caption pairs (fi, gcap,i) while minimizing similarities to mismatched pairs. The objective is formulated with an image-to-text loss Li2t and a text-to-image loss Lt2i:

The overall objective is Lcon = 0.5(Li2t + Lt2i).

Dynamic Anchor-based Self-training. Since gradients from Tcap propagate to update θT , retaining fixed class embeddings wk induces misalignment with the evolving space (static drift), while instance-level captions cause visual features to diverge from their intrinsic class manifolds (dynamic drift). To address this dual-semantic drift, we recalibrate class anchors {gk anc}Kk=1 under the current θT , formulating the dynamic teacher distribution:

where P tea∗ provides calibrated guidance compared to the static baseline P tea . Consequently, we derive pseudo-labels from the dynamic teacher: ˆ yi = argmaxk P tea∗ (y = k|xi). This dynamic recalibration is performed during training with marginal computational overhead and introduces no additional cost at inference stage (see Appendix).

Following ImCapDA [Xiang et al., 2025], we compute the student prediction P stu by feeding the concatenated visual and caption features into a linear classifier h(·)

To guide and accelerate student training, we filter highconfidence pseudo-labels from the dynamic teacher via a threshold γ as supervision signals, minimizing:

where I(·) denotes the indicator function.

3.4 Dynamic Cross-Modal Knowledge Distillation

To harness the robust generalization capability of visionlanguage pre-training models for UDA, CMKD [Zhou and Zhou, 2024] leverages the static semantic guidance from frozen text encoders to transfer generic knowledge to the unlabeled target domain. However, a critical limitation remains: fixed class embeddings wk fail to track the continuously evolving visual representations during training, inevitably leading to severe cross-modal misalignment. To bridge this gap, we propose Dynamic CMKD, which utilizes the refined dynamic teacher (P tea∗ ) and student (P stu ) distributions derived in Sec. 3.3 to enforce distribution consistency via a dynamic distillation loss.

Optimization Objective. To quantify the prediction consensus between the student and the dynamic teacher, a consistency coefficient coe = sg(exp(-KL(P stu ‖ P tea∗ ))) is computed, where sg(·) denotes the stop-gradient operation. Furthermore, to mitigate the over-confidence risk inherent in standard entropy minimization, the joint objective is constructed based on Gini Impurity:

where G(P) = 1 - ‖P‖22 denotes the Gini Impurity, and P mix = 0.5(P stu + P tea∗ ) represents the averaged mixture distribution. Specifically, when consensus is high (coe → 1), Ltask drives the student to maximize prediction certainty. Conversely, when predictions diverge (coe → 0), Ldistill steers the student towards P mix reduces bias through dynamic teacher priors. Furthermore, Lreg encourages the sharpness of the teacher's distribution to prevent over-smoothing. We adhere to the original hyperparameter settings of α = 0.25 and β = 0.025, and employ λ3 to balance the overall objective.

3.5 Overall Optimization Objective

To achieve robust SFF-DA, we propose an end-to-end DSSG framework that integrates the generative caption stream Tcap and class anchor stream Tanc to jointly fine-tune the visual encoder θV and textual encoder θT via the total objective:

where Lcon ensures caption-guided alignment, while Lst and L∗cmkd leverage dynamic self-training and consensus-aware distillation to mitigate dual semantic drift in a unified objective. λ{1,2,3} are trade-off hyperparameters.

Total Gradient. The model parameters are updated via the aggregated gradient dynamics: ∇θV ,θT Ltotal = λ1∇θV ,θT Lcon + λ2∇θV Lst + λ3∇θV ,θT L∗cmkd .

4 Experiments

4.1 Experiments Setting

Datasets. Experiments are evaluated on four benchmarks: (i) Office-31, 4,110 images in 31 classes across three domains: Amazon (A), Webcam (W), and DSLR (D); (ii) Office-Home, 15,588 images in 65 classes across four domains: Art (A), Clipart (C), Product (P), and Real-World (R); (iii) Mini-DomainNet, a DomainNet subset with 140,006 images in 126 classes across four domains: Clipart (C), Painting (P), Real (R), and Sketch (S); (iv) VisDA-2017, a synthetic-toreal benchmark with 55,388 target images across 12 classes.

Baselines. We evaluate our method against representative baselines from three categories: (i) Unimodal Source-Free DA (U), including SHOT [Liang et al., 2020], AaD [Yang et al., 2022], DIPE [Wang et al., 2022], DSiT [Sanyal et al., 2023], SF(DA)2 [Hwang et al., 2024], and TPDS [Tang et al., 2024a]; (ii) Multimodal Source-Free DA (U+M), employing an external multimodal teacher to guide the sourcesupervised unimodal student, represented by DIFO-C [Tang et al., 2024c], ProDe [Tang et al., 2025], MMGA [Chen et al., 2025], Co-Learn [Zhang et al., 2025], DTKI [Zhan et al., 2026], and DPC [Zhan et al., 2024]; and (iii) Multimodal Source-Fully-Free DA (M), such as UPL [Huang et al., 2022], POUF [Tanwisuth et al., 2023], TFUP-T [Long et al., 2024], ReCLIP [Hu et al., 2024], and ImCapFusionSFDA [Xiang et al., 2025].

Implementation Details. We adopt CLIP pre-trained on WIT-400M with ViT-B/16 and ResNet50 backbones (RN101 for VisDA-2017). We use SGD with momentum 0.9, weight decay 5e-4, and batch size 32. The learning rate is 3e-6 for the encoders and 1000× higher for the student head. The confidence threshold γ is empirically set to 0.9 across all benchmarks. All experiments are implemented in PyTorch on an NVIDIA RTX 4090D GPU. See Appendix for more details.

4.2 Experimental Results

We compare DSSG with representative baselines from unimodal (U), external-teacher-guided multimodal (U+M), and directly fine-tuned multimodal (M) settings.

Results on Standard Benchmarks: Office-Home and Office-31. As shown in Table 1 for Office-Home, DSSG achieves 89.3% with ResNet-50, exceeding the 84.5% of both ImCapFusionSFDA and ProDe-V by a 4.8% margin. With the ViT-B/16 backbone, DSSG improves accuracy to 92.9%against ImCapFusionSFDA's 90.9%.

Table 2 exhibits consistent gains on Office-31. While U+M methods leverage external priors on ResNet-50, DSSG secures 90.6%, significantly surpassing ImCapFusionSFDA at 84.2%. With ViT-B/16, DSSG attains 94.9%, outperforming both ImCapFusionSFDA (90.2%) and DPC (93.3%), demonstrating robustness across architectures.

Results on Large-Scale Benchmarks: Mini-DomainNet and VisDA-2017. Table 3 shows that DSSG consistently outperforms the baseline ImCapFusionSFDA across various backbones, achieving 84.1% and 90.0% accuracy on ResNet-50, and 89.3% and 92.1% on ViT-B/16 for MiniDomainNet and VisDA-2017, respectively. This sustained improvement underscores the effectiveness and generalization ability of DSSG. Detailed per-category results for VisDA-2017 are provided in Appendix A.2.

Trade-offs and Fair Comparison: U+M vs. M Paradigms. On ResNet-50, U+M methods like DIFO-R and ProDe-R (with same-level teachers) lag behind DSSG. Competitive performance for U+M emerges only when employing external teachers with significantly stronger backbones (e.g., CLIP-ViT-B/16 or L/14), resulting in a marginal edge on Office-31 (≈ 3.8%) and Mini-DomainNet/VisDA-2017 (≈ 1.0%), whereas DSSG maintains a significant lead on OfficeHome (+4.8%). Conversely, on the ViT-B/16 backbone, DSSG consistently outperforms U+M methods across all datasets, demonstrating superior generalization without external teacher guidance. These results indicate that U+M's edge on the weaker ResNet-50 backbone is contingent upon robust weight initialization via supervised source pre-training and strong external teacher guidance. In contrast, DSSG directly fine-tunes a single VLM to capture domain-specific knowledge applicable to various downstream tasks, without the strict source dependency and dual-model deployment overhead inherent in U+Mmethods.

4.3 Ablation and Analysis

Ablation Study on DSSG Framework. We present a component-wise analysis in Table 4 to verify the effectiveness of the proposed framework, where 1 serves as the optimized baseline ImCapFusionSFDA. We use Office-Home with ResNet-50 for representative analysis. (1) Contribution of Individual Modules. Independently integrating the DSG module ( 2 ) or static CMKD ( 3 ) boosts the 84.50% baseline to 87.38% and 87.09%, respectively. This indicates that DSG introduces structural constraints to alleviate semantic drift, whereas CMKD leverages self-distillation to transfer CLIP priors to the student task head. (2) Impact of Dynamic Calibration. Removing the DSGmodule ( 4 † ) causes severe performance degradation, verifying the inherent dependence of dynamic CMKD on the co-evolutionary semantic space. Conversely, the integration of DSG with static CMKD ( 5 ) yields 89.04%, validating the effectiveness of their joint optimization. Furthermore, incorporating dynamic CMKD ( 6 ) advances performance to 89.28%, suggesting that dynamic teacher-student calibration alleviates feature misalignment inherent in static class embeddings. (3) Cross-Dataset Generalization. The progressive gains from individual modules to the full DSSG framework are consistently observed across all benchmarks, including Office-31, Mini-DomainNet, and VisDA-2017, demonstrating the framework's robust generalization across diverse datasets and backbones.

Ablation Study on DSG Module. We investigate the impact of distinct guidance mechanisms in the DSG module, as summarized in Table 5. (1) Catastrophic Forgetting of Fine-tuning VLM. Standard fine-tuning (FT w/o Stream) and anchor-only alignment (Tanc) suffer from severe catastrophic forgetting. Even when their peak performance is obtained via a lower learning rate and 100-iteration evaluation (denoted as ‡), accuracy drops to 66.23% and 65.32% (vs. 74.90% baseline), indicating a severe loss of pretrained knowledge. (2) Semantic Alignment and Divergence. VLM-only alignment (Tcap) recovers 85.09% accuracy via cross-modal alignment to alleviate forgetting. However, without source supervision, relying on unconstrained caption semantics risks semantic divergence. (3) Effect of Dual-Stream Integration. DSG module consistently surpasses the Tcap baseline, yielding +1.86% on RN50 and +0.94% on ViT-B/16. These gains demonstrate that dynamic class anchors provide structural regularization to stabilize the unconstrained caption stream, effectively preventing semantic divergence by balancing stability and plasticity.

Hyper-Parameter Sensitivity Analysis. We investigate the sensitivity of DSSG to hyper-parameters λ{1,2,3} in Eq. 10. Since ImCapFusionSFDA lacks reported results on these benchmarks, we re-implemented and optimized it via grid search on λ1 and λ2, establishing a rigorous lower bound (see Appendix A.3). Given identical loss definitions (Lcon and Lst), these optimal hyperparameters are directly inherited by DSSG. We then vary λ3 and observe consistent outperformance over the baseline across a wide range, confirming the robustness of λ3 (see Appendix A.4). This stability further validates that performance gains stem from structural designs, not specific tuning.

Computational Efficiency. We benchmark the efficiency of DSSG across Params, GFLOPs, Iteration Speed, and Latency. Although the dual-stream architecture slightly increases training costs, DSSG maintains inference efficiency comparable to the baseline (≈ 4.4 ms latency) by caching fixed semantic anchors (detailed in Appendix A.5).

Visualization Analysis. t-SNE visualizations in Fig. 3 show that DSSG yields denser target clusters with clearer boundaries across all benchmarks, demonstrating superior discriminative capability. Grad-CAM results are available in Appendix A.6.

4.4 Discussion

Since the DSSG framework is built upon VLM-generated textual descriptions, several methodological questions naturally arise regarding this paradigm. These include the use of stronger captioning VLMs to guide fine-tuning, the need for an auxiliary text classifier, the presence of class-name tokens (and their removal/masking), and the scalability to larger backbones like ViT-L/14. As ImCapDA [Xiang et al., 2025] has already extensively validated these basic mechanisms, we do not repeat these analyses and refer readers to the original work for further discussion.

Instead, we focus on caption-based SFDA and compare DSSG with LLaVA-v1.6 [Liu et al., 2023], InstructBLIP [Dai et al., 2023], and ShareGPT4V [Chen et al., 2024b]. These methods employ large VLMs for caption generation and an STS module [Chen et al., 2024a] for classification (see Table 6). DSSG integrates visual features with captions via smaller VLMs (e.g., CLIP 0.12B), surpassing large-scale VLMs (up to 34B) in both accuracy and efficiency, even when accounting for BLIP-3 (4.1B) [Xue et al., 2024]. Moreover, DSSG enables flexible deployment by supporting lightweight models, API-based generation, or pre-computed captions.

5 Conclusion

In this paper, we investigate the challenging Source-FullyFree Domain Adaptation (SFF-DA) task, identifying Dual Semantic Drift-the conflict between static embedding rigidity and caption divergence-as the primary obstacle to effective adaptation. To address this, we propose the DSSG

framework, which reconciles the stability-plasticity dilemma by integrating complementary semantic streams. Specifically, the DSG module harnesses domain-specific knowledge from generated captions while anchoring global categorical consistency via class anchors. Furthermore, the Dynamic CMKD module calibrates the evolving teacher distribution via selfdistillation to enforce teacher-student consistency. Extensive experiments on multiple benchmarks demonstrate that DSSG significantly outperforms existing methods, establishing a new state-of-the-art. Our work provides a novel perspective on balancing universal knowledge preservation and domainspecific knowledge acquisition in vision-language foundation models, paving the way for more robust unsupervised deployment in privacy-sensitive and source-constrained scenarios.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grants 62372167 and 62272156, and in part by the Natural Science Foundation of Hunan Province of China under Grants 2025JJ70455 and 2024JJ5097.