Towards Dynamic-Prompting Collaboration for Source-Free Domain Adaptation

1 Introduction

Domain adaptation (DA) [Wang et al., 2022b; Nejjar et al., 2023] refers to the process of adapting a model trained on a labeled source domain to an unlabeled target domain. Traditional methods for this task assume access to labeled source data during the adaptation process. However, in the real world, practical constraints such as privacy and security often limit our ability to access source data directly. Consequently, the field of source-free domain adaptation (SFDA) [Roy et al., 2022; Tang et al., 2023] has gained significant attention. SFDA aims to adapt a source-trained model to an unlabeled target domain without the need for direct access to source data.

Many SFDA methods prioritize transferring domaininvariant knowledge from source-trained models to the target domain by iteratively fine-tuning models through selfsupervised techniques, such as the use of pseudo-labels. In an ideal scenario, when the source and target domains exhibit a high degree of similarity, the source-trained model can efficiently capture domain-specific information from the target data and learn accurate classification boundaries. However, practical situations often deviate from this ideal, and the target data may significantly differ from the source hypothesis. This can lead to unreliable pseudo-labels and introduce bias into the learning process. Existing efforts aim to ameliorate these issues by employing techniques such as local structural adjustments [Qu et al., 2022], entropy-based strategies [Litrico et al., 2023], and historical consistency measures [Huang et al., 2021]. Nevertheless, the inherent domain discrepancy can persist, resulting in an accumulation of irreparable bias and a loss of domain-invariant information throughout training [Fang et al., 2022; Yu et al., 2023].

∗Corresponding authors (seanzhuxf@gmail.com).

Recently, large-scale pre-trained vision-language models, such as CLIP [Radford et al., 2021], have demonstrated exceptional generalization capabilities with invariant feature encoding across diverse visual domains. However, foundational models like CLIP struggle to effectively represent domainspecific knowledge, a crucial factor for achieving success in downstream tasks [Jia et al., 2022]. To address this limitation, prompt learning has emerged as a solution, involving the encoding of domain-related context as learnable parameters

(referred to as prompts) at the input end [Zhou et al., 2022b]. This approach also offers a viable solution to challenges like domain adaptation [Fahes et al., 2023]. Nevertheless, optimizing these learnable prompts often demands careful data curation and annotation, which can be difficult SFDA, where labeled data is scarce, and extracting domain-specific information poses significant challenges.

Motivated by the aforementioned insights, our work aims to tackle SFDA by complementing the traditional methodology with a large vision-language model. Our goal is to extract domain-specific information from unlabeled target data reliably while consolidating the domain-invariant knowledge embedded within the source model. However, the integration of these two distinct modeling paradigms into a collaborative SFDA framework presents significant challenges. The first challenge is how to facilitate effective interaction between these two models. A well-designed integration of both paradigms can result in mutual reinforcement and improved performance. Conversely, a suboptimal solution may lead to undesired outcomes. The second challenge is how to ensure reliable learning from pseudo-labels. As the pseudo-labels in SFDA can be susceptible to noise and errors.

To address these challenges, we propose a collaborative framework centered around an innovative dynamic prompting method. As illustrated in Figure 1 (c), our approach fosters collaboration between a source-trained model and a vision-language model, such as CLIP, not only during the later stages of loss functions but also through joint prompting mechanisms at earlier feature and input stages. This unique design helps reconcile the disparities between the two distinct paradigms and promotes the exchange of complementary information within each component. Furthermore, to mitigate the disruption caused by unreliable pseudo-labels, we introduce a dynamic mask prompting (DMP) mechanism. When presented with a pseudo-labeled image, DMP is trained to analyze the corresponding class activation maps to identify noisy and distracting elements within the image. These identified pixels are then suppressed by replacing them with visual prompts. Consequently, subsequent model tuning can focus more on task-related information and be less susceptible to noise and background interference. We summarize the contributions of this work as follows:

• We present a collaborative framework in which a traditionally source-trained model dynamically collaborates with large vision-language models to address the SFDA task.
• We recognize the adverse effects of incorrect pseudolabels in SFDA scenarios and propose a dynamic mask prompting mechanism to enhance the learning process.
• We conduct extensive experiments on multiple datasets, and the results show that our method outperforms previous state-of-the-arts in SFDA.

2 RelatedWorks

2.1 Source-Free Domain Adaptation

Previous SFDA methods can be divided into two directions: model-centric and data-centric. Model-centric methods assume the optimal target hypothesis to be closely related to the source hypothesis. Therefore, by exploring the outputs of the source training model, models can be fine-tuned using a selftraining scheme [Liang et al., 2020]. BMD [Qu et al., 2022] proposed a dynamic multi-centre pseudo-labelling strategy for updating pseudo-labels during domain adaptation. JMDS [Lee et al., 2022] proposed to mitigate the effect of noisy samples by using confidence scores as sample weights. With the success of Transformer in vision tasks, DSiT [Sanyal et al., 2023] proposed a fully ViT-based solution for SFDA and achieved outstanding performance. Though achieving promising performance, this type of method relies on taskspecific knowledge learned on the source domain, which can result in biased pseudo-labels and potentially suffer from the loss of domain-invariant information in the progressive learning process [Zhang et al., 2023a].

Data-centric models aim to enhance the pre-trained taskspecific knowledge by reconstructing data for the missing source domain, based on which existing DA methods can be easily extended to SFDA scenarios [Li et al., 2020]. However, with only the unlabelled target data, it is challenging to effectively represent the task-specific information revealed by the source data [Chen et al., 2022; Tang et al., 2020]. Recently, large-scale pre-trained vision-language models have become a popular paradigm for transferring pre-trained models into downstream tasks. In domain adaptation i.e., AutoLabel [Zara et al., 2023] utilized CLIP to discover candidate target categories, enhancing the capture of discriminative information for video domain adaptation. PADCLIP [Lai et al., 2023] introduced adaptive biasing learning to address the issue of noisy labels in domain adaptation. However, the lack of target annotations limits their application in SFDA.

2.2 Prompt Learning

Prompt learning is becoming a popular way to enhance large-scale pretrained models with domain-specific knowledge [Zhou et al., 2022b]. Hard prompt learning employs discrete tokens in the vocabulary as prompts, while soft prompt learning introduces learnable parameters in the text embedding and tunes these parameters [Jia et al., 2022; Jiang et al., 2024]. To address vision tasks, CoOp [Zhou et al., 2022b] designed class-specific prompts through back-propagation. DAPL [Ge et al., 2023] proposed ad-hoc prompting to learn disentangled domain and category representations. ADCLIP [Singha et al., 2023] introduced a domain-agnostic prompt learning strategy for CLIP and achieved state-of-the-art results. Although these methods showing promising performance in domain adaptation, prompt learning with large vision language models requires labeled training data, and the limited capacity of prompts can be difficult to model complex domain-specific knowledge of target domains.

3 Methodology

Problem Definition. The task of SFDA involves a labeled source dataset Ds = {xsi , ys i }ns i=1 and an unlabeled target dataset Dt = {xti }nt i=1 , where x and y denote the image and label, respectively. Typically, Ds and Dt follow distinct distributions, and Ds is only available during the pre-training stage of the source-trained model fθ. The primary objective of SFDA is to adapt the model fθ to the target domain using solely Dt . This paper mainly focuses on the general SFDA setting, wherein both domains share the same set of K classes (closed-set setting). Nevertheless, in our experiments, we also explore partial and open-set scenarios.

Overview. An overview of our method is depicted in Figure 2. The framework comprises three key components: the source-pretrained branch, the vision-language branch, and the dynamic mask prompting mechanism. In the sourcepretrained branch, we adhere to the traditional model-centric SFDA approach, progressively fine-tuning the entire model based on pseudo-labels to encode domain-specific knowledge from the target domain. The vision-language branch leverages the CLIP model and employs learnable textual prompts to extract task-related domain-invariant information. The dynamic mask prompting module plays a pivotal role in our method. It identifies and enhances the reliability of pseudolabels by introducing dynamic visual prompts. This module facilitates collaboration between the two branches, enhancing the overall performance of our approach. This comprehensive design allows our method to effectively acquire domainrelated knowledge from the target data while ensuring the preservation of domain-shared information related to classification tasks. In the following, we will provide detailed descriptions of the source-model-based protocol, introduce the vision-language model-based paradigm, and propose the dynamic mask prompt mechanism.

3.1 Source-Pretrained Branch

The source-pretrained branch is primarily dedicated to adjusting model parameters to align with the target data, aiming to learn domain-specific knowledge relevant to the target domain. Given the absence of annotations for the target data, we employ a common self-training strategy used in SFDA methods. We start with an initial classification model fθ (e.g., ViT [Dosovitskiy et al., 2020]) and tuning it on the target dataset Dt by optimizing the cross-entropy loss on the target image xt :

where ˜ yt represents the pseudo-label generated based on the average of the predictions from both branches, pm(·) denotes the predicted probability of the target model, and ˆxt refers to the prompted input image, which will be comprehensively introduced in Section 3.3.

3.2 Vision-Language Branch

We leverage the exceptional generalization capabilities of CLIP [Radford et al., 2021] to extract domain-invariant task information (i.e., category semantics) from the target data. CLIP consists of a vision encoder fv and a text encoder ft. The vision encoder fv transforms the image xt into a visual representation fv(xt ) ∈ Rd . Simultaneously, a set of class embedding ft(wk) ∈ Rd is generated by feeding template prompts wk (e.g., "A photo of a [CLS]") into text encoder ft. Hence, the classification probability of xt is defined as:

where sim(·, ·) represents cosine similarity, and τ is a temperature factor. Following CoOp [Zhou et al., 2022b], we convert the class name k into a text embedding wk ∈ Rd and enhance this embedding with a set of learnable prompts {vi}Ti=1 . Formally, the input for the text encoder becomes Vk = {v1, v2, . . . , vT ,wk}. By normalizing the representations for each class k, we can calculate the classification probability using Eq. (2). During training, we optimize the learnable parameters on the target data by minimizing the crossentropy loss:

where ˜ yt denotes the pseudo-label of image xt , generated based on the average of the predictions from the sourcetrained model and CLIP model. ˆxt represents the prompted input image, which will be introduced in the next section.

3.3 Dynamic Mask Prompting

Due to the domain discrepancy, pseudo-labels can be prone to limitations such as noise and errors. Learning directly from such pseudo supervision may result in suboptimal performance and limited generalization on the target domain. To ensure the reliable transfer of domain-shared knowledge from the source model to the unlabeled target data, we start by analyzing the visual content that supports these pseudo-labels. As depicted in Figure 3, effective pseudo-labels correspond to relevant regions within the image, whereas incorrect pseudolabels often get distracted by unrelated or confusing areas. This observation motivates us to enhance the quality of learning by suppressing distracting responses in the images. We introduce a novel mechanism called Dynamic-Mask Prompting (DMP), as illustrated in Figure 2 (c). In DMP, we initially obtain a pseudo-label for an image based on predictions from both the source-pretrained branch and the vision-language branch. Subsequently, we generate corresponding class activation maps, which are guided by a Gaussian Mixture Model (GMM) [Permuter et al., 2006] to estimate regions with a negative impact. These negative regions are then masked out from the input image. The masked pixels are replaced with visual prompts, further enhancing the interaction between the two branches.

Mask Prompt Generation. To identify image regions that negatively impact learning, we begin by analyzing the corresponding class activation maps of pseudo-labels. Given the input image xt and the pseudo-label, we employ GradCAM [Selvaraju et al., 2017] to generate the activation map for the model in the source-pretrained branch. Specifically, we first obtain feature tokens Fv ⊆ Rb×b×d , where b represents the patch size (e.g., 16). Then, we compute the activation map as follows:

where GradCAM(·) is the activation map generation function [Selvaraju et al., 2017].

For CLIP, we utilize the similarity between the text encoding of the pseudo-label and the visual features of the image to obtain the class attention map. Specifically, we compute the similarity between the features of image tokens Fi ⊆ Rb×b×d and the features of texts Ft ⊆ Rb×b×d with l2 normalization along the feature channel dimension, as shown in the equation:

Then, we combine both activation maps by averaging,

As a result, the activation map reflects the correlation between pixel locations and the pseudo-label, revealing how the visual contents will be activated during learning. When the pseudo-label is correct, such activations assist the model in concentrating on the most relevant regions. However, in cases of an incorrect pseudo-label, the high-response areas can divert the model's attention away from informative content, resulting in less effective knowledge transfer.

The absence of annotations poses challenges in assessing the quality of pseudo-labels. Guided by our empirical observation that correct pseudo-labels yield lower loss values and vice versa, we train a Gaussian Mixture Model (GMM) that takes the loss values as input. By applying a thresholding strategy with a hyperparameter φ we can roughly distinguish between correct and incorrect pseudo-labels. Based on the prediction scores, we devise an adaptive strategy to suppress the most irrelevant visual content during model tuning. For samples predicted as incorrect, we mask out the ξ highest activated pixel locations based on their activation map Au. Conversely, for samples predicted as correct, we mask out the ξ lowest activated pixel locations, as these regions are most unrelated to the task.

Mask Prompt Padding. After obtaining the masked images, simply filling the masked patches with constant values and passing them into an image encoder can lead to undesired effects during model training. This approach neglects the semantic information on the textual side, which is essential for effective learning. To address this, we introduce a dynamic visual prompting strategy that interacts with the semantic information from the textual side. Specifically, we define a set of learnable visual prompts g while simultaneously extracting domain-shared task-specific information from the text encoder through a lightweight MLP network:

where h(·) represent the lightweight MLP network, ˆ g is the visual prompt. The masked pixels are then filled with the visual prompt ˆ g. These prompted images are subsequently fed into the image encoder for pseudo-label-based self-training.

3.4 Optimization

Throughout the training process, we jointly optimize the textual prompts, visual prompts, and the parameters θ of the source-trained model within a collaborative framework. The pseudo-labels are derived from the averaged predictions of both branches. Additionally, to prevent the model from converging to a trivial solution where it predicts all data as a specific class, we incorporate the prediction diversity loss Ldiv into our framework:

where ¯ pkf (ˆxt ) represents the empirical label distribution of the k-th class, and q is the uniform distribution defined as q = 1/K, where K represents the number of classes. KL(· || ·) denotes the Kullback-Leibler divergence. Hence, together with Eqs.(1) and (3), the overall loss is presented as:

where µ1 and µ2 are non-negative parameter weights. Eq. (9) simultaneously collaborates with two independent modelling paradigms to achieve optimal performance in SFDA.

4 Experiments

4.1 Experimental Setup

Datasets. We evaluate our proposed approach on three standard benchmarks in domain adaptation including Office-31 [Saenko et al., 2010], Office-Home [Venkateswara et al.,

2017], and DomainNet [Peng et al., 2019]. Office-31 contains 4,652 images for 31 classes collected from three distinct domains, which are Amazon (A), DSLR (D), and Webcam (W). Office-Home is a medium-scale dataset consisting of images for everyday objects, and can be divided into four domains, i.e., Artistic (Ar), ClipArt (Cl), Product (Pr), and Real-world (Rw), with 65 classes for each. DomainNet is a large-scale benchmark with six domains, each with 345 classes. Following the setting of [Litrico et al., 2023], we select four domains: ClipArt (C), Real (R), Painting (P), and Sketch (S), with 126 classes as the SFDA benchmark. All reported results are the average of three independent runs.

Implementation Details. As introduced in the methodology, our proposed method contains two network branches. In the Source-Pretrained Branch, we follow the experimental settings of [Sanyal et al., 2023] and utilize ViT-B/16 (input size 224 × 224, patch size 16×16, resulting in 14×14 patches per input). For Vision-Language Branch, we adopt the experimental settings of CoCoOp [Zhou et al., 2022a] and use CLIP based on ViT-B/16 as the vision encoder. We optimize training objectives via the Stochastic Gradient Descent (SGD) [Zinkevich et al., 2010] optimizer, given a mini-batch size of 16, the momentum of 0.9, and weight decay ratio of 1×10-4 , respectively.

4.2 Experimental Results

Source-Free Domain Adaptation. Tables 1, 2, and 3 report the performance of our method and previous approaches on SFDA. As indicated in the tables, we compare with traditional CNN-based methods, Transformer-based methods, and vision-language-based models. Results demonstrate that our source-free method outperforms the previous state-ofthe-art SFDA methods, and achieves similar performance to the source-dependent models. Specifically, it surpassed the prior best method DSiT† by 0.3% Office-31, by 4.9% Office-Home, and method AaD† by 12.9% on DomainNet, thereby establishing a new state-of-the-art for SFDA. Besides, Compared to source-dependent DA approaches, our proposed method is equally competitive. Specifically, our method improves over the VLM-based DAPL by 1.4% on Office-Home. The reasons can be summarized as follows. First, our proposed approach unites the traditional sourcepretrained model and the vision-language model in a collaborative framework, complementing to enhance their advantages and address their shortcomings. The domain-specific knowledge encoded in the tuned source pre-trained model helps to unleash from VLM the reliable domain-invariant task information as well as the generalization ability to fill the domain gap. Moreover, benefiting from the proposed dynamic mask prompt learning, our proposed approach mitigates the noise and erroneous in pseudo-label-based self-training.

Open-set/Partial-set SFDA. We also evaluate our method with more practical settings like open-set and partial-set scenarios. In the open-set setting, the target domain contains some classes that are agnostic in the source domain. In contrast, in the partial-set scenario, the target domain contains only some of the classes in the source domain. We compare the performance for the open-set and partial-set settings in Table 4. In both scenarios, the proposed method achieves the best performance. Compared with the previous best method CRS, ours improves by 5.2% on average. This shows the superior performance of our methods. In addition, compared to the sub-optimal method AaD, our method improves by 6.35% on average, which demonstrates the ability to better enhance the generalization capacity in vision-language models.

Ablation Analysis. In this part, we analyze the impact of different components in the proposed framework and report the ablation results in Table 5. Specifically, "SPB" and "VLB" denote applications of the Source-Pretrained Branch and the Vision-Language Branch, respectively. "DMP" indicates the dynamic mask prompt learning. As shown in the Table 5, each component makes a unique contribution to the final performance. We observe that the SPBonly baseline outperforms the VLB-only baseline on easier datasets (e.g., Office-31), and underperforms on more complex datasets (e.g., Office-Home, DomainNet). This may be because the VLB model is less effective in encoding domainspecific knowledge under small domain gaps, whereas the SPB model struggles to extract domain-invariant task information under large domain discrepancies. Thus, both separate branches have their inherent disadvantages. The fusion of VLB and SPB allows for the integration of their respective strengths. As shown, the simple combination "SPB +VLB" is 1.2% better than "SPB" on Office-31 and 1.7% better than "VLB" on Office-Home. By further encouraging the interaction between the two paradigms, our dynamic mask prompting mechanism leads to a more adaptive and robust model, thereby enhancing performance across datasets of varying scales and complexities, e.g., boosting the performance by

1% and 1.3% on Office-31 and Office-Home, respectively, compared to the simple fusion strategy. The effectiveness of applying dynamic mask prompt learning to individual or both branches is confirmed. In Figure 4, we also visualize the masks generated in the dynamic mask prompting component. As shown, the background and distracting content are successively suppressed, hence enabling the model to better learn from the target data.

Hyper-parameter Analysis. We investigate the impact of hyper-parameters in our proposed method, i.e., the threshold φ for GMM and the percentage ξ for selecting on activation maps. We conduct the classification task on Office-Home (i.e., Cl-Ar, Cl-Pr) by varying the value of φ in the range of [0.7,0.9] and the value of ξ in the range of [0.05,0.25]. The results are illustrated in Figure 5. The accuracy decreases when the value of φ becomes small (i.e., 0.7). This is because the initial pseudo-labels of the model are of good quality, and when setting at a lower φ, it is easy to confuse the correct labels leading to poor training results. If the value of ξ is too large or small, it is easy to get poor results. Too many mask patches may lose discriminative information, while too few mask patches will contain noisy information. This also shows the effectiveness of our proposed dynamic masked prompt.

5 Conclusion

In this paper, we present a collaborative framework for source-free domain adaptation, which exploits the inherent advantages of traditional model-centric protocol and largescale vision-language model to complement each other. To enhance the interaction between the two paradigms, we further introduce a novel dynamic mask prompting mechanism that adaptively suppresses noisy and distracting visual content during training. Extensive experiments have been conducted to analyze the proposed approach, and the results demonstrate that our method outperforms previous state-ofthe-art of source-free domain adaptation.

Acknowledgments

This work was supported in part by the National Key Research and Development Program of China under Grant 2022YFA1004100, the National Natural Science Foundation of China under Grant 62306066, and the Sichuan Province Innovative Talent Funding Project for Postdoctoral Fellows under Grant BX202313.