Counterfactual Knowledge Maintenance for Unsupervised Domain Adaptation

1 Introduction

Deep learning has achieved significant success with large datasets [Huang et al., 2022], but models often require vast amounts of labeled data and are highly dependent on datadriven features. When applied to different domains with similar tasks, performance can degrade due to domain discrepancies. Data-driven models, assuming independent and identically distributed (i.i.d.) data, are vulnerable to distribution shifts from training data [Zhao et al., 2025]. Unsupervised domain adaptation (UDA) offers a solution by allowing models to be trained on labeled data from a source domain and tested on unlabeled data from a target domain, reducing the General Methods impact of domain differences. Traditional UDA methods aim to minimize the distribution gap using techniques like entropy minimization [Wang and Deng, 2018], moment matching [Li et al., 2021], and adversarial learning [Du et al., 2021]. However, aligning visual features without addressing distribution differences can cause semantic distortions and reduce class distinguishability [Tang et al., 2020a]. Additionally, converting text labels into numerical labels limits the model's ability to capture rich, nuanced class information, leading to suboptimal performance [Ge et al., 2023].

∗Corresponding author

Figure 1: Current prompt-based UDA methods lead to a coarsegrained alignment that encompasses both domain-related and semantically-related features. Our method decouples semantic information and domain features by counterfactual disentanglement, only aligning semantic related features. This approach minimizes the bias introduced by domain-specific knowledge.

To overcome the limitations of traditional UDA methods, large-scale pre-trained visual language models (VLMs) have been increasingly leveraged. VLMs, such as CLIP [Radford et al., 2021] and ALIGN [Jia et al., 2021], are pre-trained on extensive image and text pairs, aligning visual and textual features within a joint embedding space. This pre-training process encodes rich visual and textual information, enhancing the adaptability and generalization of the model across different domain contexts through alignment. Therefore, VLMs possess a robust conceptual knowledge foundation and strong zero-shot transfer capabilities, making them highly effective for addressing domain adaptation challenges [Bai et al., 2024; Ge et al., 2023; Singha et al., 2023].

However, these methods primarily leverage the pretraining knowledge of VLMs to enhance generalization and build statistical dependencies between data and labels, without elucidating the underlying relationship. UDA requires learning not only statistical correlations between visual features and labels across domains but also potential causal mechanisms [Wang et al., 2023]. By establishing causal relationships instead of correlations between visual features and labels, models can better classify the same visual feature in various scenarios. Existing UDA methods often lead to spurious correlations between inference results and visual features. Additionally, VLMs involve two distinct modalities: visual and textual. For classification, visual features may be more influential for some samples, while textual features may be more useful for others. For example, categories with distinct visual characteristics, like chairs and sofas, are easier to classify using visual features. In contrast, categories with similar visual features, such as filing cabinets and shelves, are better classified using textual features. Current UDA methods mainly rely on the extensive visual knowledge in CLIP for classification, leading to suboptimal results when visual semantic information is ambiguous.

To address these challenges, we propose a novel UDA method using counterfactual disentanglement and discrimination knowledge maintenance. First, we introduce a structural causal model that identifies semantic information relevant to data types and categories as causal factors, while domain-specific information unrelated to categories is treated as non-causal factors. Only semantic features causally affect category labels. As shown in Figure 1, given a source domain sample x, the coupled feature f can be obtained by the encoder. By intervening in the domain information D = d and setting it as a counterfactual with target domain knowledge, the sample ˜x in the target domain can be obtained. During this process, semantic information remains unchanged across domains. By employing counterfactual disentanglement, we decouple semantic and domain features, thereby mitigating biases inherent in CLIP's knowledge base. Secondly, we adaptively integrate the visual and textual embeddings extracted by VLMs, assessing which modality is more conducive to inference for each sample. By maintaining discrimination knowledge, we preserve the distinctive features of each modality and clarify class-specific ambiguous visual information. Our method not only eliminates spurious correlations caused by confounding factor domain confidence, but also synergistically combines the advantages of visual and textual modalities to enhance task performance.

Overall, our significant contributions can be summarized as follows:

• We propose a novel counterfactual knowledge maintenance framework and construct a causal model from a causal perspective for the UDA problem.
• To solve the confusion between semantic and domain features, counterfactual disentanglement is proposed to decouple and represent mixed knowledge, obtaining domain-invariant knowledge.
• To distinguish complex scenes with slight semantic details, discrimination knowledge maintenance is used to clarify class-specific ambiguous visual information

based on modal preferences of discriminative features.

• Extensive experiments are conducted on public datasets to demonstrate the effectiveness of the proposed method.

2 RelatedWork

2.1 Causal Mechanism

Deep learning models traditionally rely on vast datasets to generate predictions, which can lead to capturing spurious correlations [Yue et al., 2023]. This results in models that lack interpretability, robustness, and generalization capabilities. Causal learning is used to establish robust causal relationships within complex and unstructured data. To mitigate the impact of confounding factors in data, Zhu et al. [Zhu et al., 2023] introduced a method that leverages causal relationships to reweight and resample online data, thereby enhancing model performance. Ding et al. [Ding et al., 2023] employed causal models to solve the problem of inconsistent data distribution in the training dataset. Furthermore, Yue et al. [Yue et al., 2021] proposed a transport causal mechanism to identify the representation of confusion layers and domain invariant solution causal mechanisms. Yang et al. [Yang et al., 2023b] introduced a learned causal representation.

2.2 Prompt Learning

Visual language models integrate rich visual and textual information from large datasets and excel in various computer vision tasks. Recently, prompts have been integrated into VLMs to learn universal visual representations. CLIP [Radford et al., 2021] is the most groundbreaking. Zhou et al. [Zhou et al., 2022b; Zhou et al., 2022a] proposed the CoOp and CoCoOp method, which model prompts using continuous representations, automatically learning task-related prompts. However, these methods overlooked domain transfer issues. DAPL [Ge et al., 2023] processes distribution transitions in UDA by learning domain-independent and domain-specific cues. PADCLIP [Lai et al., 2023] introduces domain names and dynamically adjusts depolarization strength and momentum in prompts to bridge domain gaps. PDA [Bai et al., 2024] proposed a prompt-based distribution alignment method that incorporates domain knowledge into prompt learning. ADCLIP [Singha et al., 2023] addresses the domain adaptation problem in the prompt space with a domain-independent CLIP prompt learning strategy. DAMP [Du et al., 2024] proposes domain-agnostic mutual prompts, which align visual and textual embeddings with domain-invariant semantics.

However, previous methods directly use coupled domain and semantic knowledge from CLIP, without considering the varying effectiveness of different modalities for specific tasks. To address this, we employ causal decoupling to separate semantic knowledge from domain-specific knowledge while maintaining discrimination knowledge for ambiguous visual semantic information.

3 ProposedMethod

3.1 Overview of Our Framework

Formally, the problem definition and symbolic representation of UDA can be described as follows: given labeled source domain image data Dsd = {xsd i , ysd i }Nsd i=1 and unlabeled target domain image data Dtd = {xtd i }Ntd i=1 . Nsd and Ntd represent the number of samples from the source and target domains, respectively. The objective of UDA is to train a model on the source domain data with distribution xisd ∼ Psd(X) such that it performs well on the target domain data with distribution xitd ∼ Ptd(X). The primary challenge in this process is that the samples originate from two distinct distributionsPsd(X) ̸= Ptd(X), which is often referred to as the domain shift problem.

Therefore, based on the CLIP framework, the counterfactual disentanglement is first introduced to decouple domainrelated and semantic-related features, thereby mitigating the confounding effects of domain-specific information. Secondly, a discrimination knowledge maintenance strategy that tailors the combination of text and visual information to the needs of individual samples, enhancing task performance. The framework is illustrated in Figure 2.

For clarity, superscripts in the text denote domains: sd for the source domain and td for the target domain. Symbols without superscripts apply to both domains. Specifically, input the sample x from the source or target domain into the frozen visual encoder V to extract the coupled visual feature f. Construct a naive prompt {ti}Ki=1 for "a [DOM] photo of a [CLA]", where [DOM] represents domain names, [CLA] represents category names, and K represents the number of categories. Input the prompt information into frozen text encoder T to obtain the text feature t. Then, the coupled visual features f are decoupled by counterfactual disentanglement. The domain-related features fD and semantic features fS are separated to eliminate the confounding caused by domainrelated features. Inspired by [Du et al., 2024], the semantic feature fS and text ti are mutually prompted to fully integrate visual and textual information, yielding updated visual feature f ′ and text feature t′ . By orthogonally decomposing the updated visual feature f ′ , the text-specific feature f ′ t and the visual-specific feature f ′v are extracted. The final logits yall are obtained by modal adaptive fusion f ′ t and f ′v using weight w, where w is a learnable parameter. Finally, the obtained yall is used to calculate the cross entropy loss with the source domain label yisd and the pseudo label of the target ˆ yi td domain for training.

3.2 Counterfactual Disentanglement

To explore the causal relationship between data and labels, a Structural Causal Model (SCM) is integrated into the CLIPbased UDA framework. The visual-language model (VLM) P extracts both visual information V and textual information T. In V , causal variables are those directly related to categories and labels, like the "shape" feature in digit recognition, and these relationships are consistent across domains. Noncausal variables, such as "handwriting style" in digit recognition, are domain-specific. Each sample X is a combination of causal variables C and non-causal variables U, with only C having a direct causal impact on the category label Y , as shown in Figure 3.

The objective of the model is to identify and extract causal factor C from the raw data X, and establish a stable causal relationship framework. Before that, two basic theorems [Reichenbach and Morrison, 1956] are given.

Theorem 1 (Common Cause Principle). If variables X and Y exhibit statistical correlation, it implies the presence of a common causal factor C that influences both X and Y , accounting for their observed associations. To put it succinctly, the correlation between X and Y vanishes when the influence of C is considered, rendering X and Y conditionally independent given C.

According to Theorem 1, formalize the structural causal model of Figure 3.

where, V1 and V2 are independent noise variables that cannot be explained. The functions F, H, and G are unknown structural functions.

In addition, causal factors need to meet the requirement of mutual independence [Peters et al., 2017], as expressed in theorem 2.

Theorem 2 (Independent Causal Mechanism Principle). Each variable's distribution, conditioned on its causal factors, is independent and does not influence or inform other causal mechanisms.

The joint distribution of causal factors can be decomposed into the following conditions, namely causal decomposition:

where, PVi is the parents of ci in causal graph.

In essence, the model must identify key causal factors which are independent of domain-specific factors and can significantly affect classification outcomes.

Given a sample x, we generate counterfactual samples based on the following three steps.

• Abduction "given the fact that f = Encoder(x)". Given evidence X = x, we obtain the source domain sample attribute Encoder(x).
• Action "had D been d". d ∈ {sd, td} is the domain label. In this step, we intervene in D by discarding the inferred value and setting D as d.
• Prediction "X would be ˜x". Under the condition of inferring f = Encoder(x) (fact) and intervention target D = d (counterfactual), we can generate counterfactual samples ˜x from Pθ(X|f = Encoder(x), D = d).

Counterfactual is used to decouple the causal factor C and non-causal factor U by generating new data with perturbed domain-related information. The generated data differs in non-causal factors U but retains the same causal factor C, ensuring the representation remains unchanged. Specifically, for an image sample x, it is fed into a fixed visual encoder V

to extract the coupled visual features f. To decouple domainrelated and semantic-related features, an attention network N is utilized, which splits the features into two distinct components. Multiply the coupled visual features by the attention network weights to obtain semantic-related features fS , while the rest are domain-related features fD.

After extracting domain-related and semantic features, the feature map's dimensionality is reduced using global average pooling and projection. Separate projection layers are used for each type of feature due to their significant differences. A counterfactual approach is applied to decouple these features, separating domain-related and semantic features while preserving semantic information. This process integrates semantic features from one domain with domain features from another.

To further ensure the independence of fD and fS, a domain classifier is employed to differentiate the domain-specific information between the original and counterfactual features. The source domain is labeled as 0, and the target domain as 1. For counterfactual features, f sd cou encapsulates the semantic information from both the source and target domains, as does f td cou . Consequently, the domain discrimination loss is:

3.3 Discrimination Knowledge Maintenance

After decoupling to extract the transferable causal feature fS, the mutual prompting method [Du et al., 2024] is employed to effectively integrate visual and textual knowledge with the causal feature fS and the textual prompt t, ultimately obtaining updated visual f ′ and textual prompt t′ . However, both visual and textual cues are unique and modality-specific, and certain samples are best classified using specific modalities. Pre-trained CLIP may sometimes misclassify these items into visually similar categories. For complex items with subtle semantic differences, visual classifiers might make errors, while CLIP can leverage its extensive knowledge base for accurate zero-shot predictions. In summary, the visual branch excels at recognizing visual features of specific categories, while the text branch clarifies uncertainties in classification through semantic information. Therefore, discrimination knowledge is maintained to adaptively fuse visual and textual information for each sample, enhancing classification performance.

Specifically, the updated visual feature f ′ is decomposed into text-specific feature f ′ t and visual-specific feature f ′v through a modal partitioning network by linear projection.

During this process, orthogonal loss is used to ensure the independence of separation.

After obtaining modality-specific features, each modality is processed to obtain the final output. For the text specific modality f ′ t ,

where, temp is temperature in pretrained CLIP. Since the target domain data has no labels, we first obtain the pseudo label ˆ ytd of the target domain data, and then calculate the cross entropy loss. For source data, cross-entropy loss is directly applied using labeled source data.

where the value of α can be adjusted based on the influence of supervised source domain data.

For the visual specific modality f ′v , pass it through two learnable linear layers ψ1 and ψ2 to obtain the final output.

Utilizing Eq. (8) and Eq. (10), the ensemble output ˆ yens is formulated as:

where, w is a learnable parameter obtained by the weight generation network W.

Like Eq. (9), we will calculate the cross entropy loss for the adaptive fusion features yens and labels obtained. In addition, to further embed the updated target domain under the learned semantic structure, and enhance individual discriminability and global diversity, an additional information maximization loss Lim is added to regularize unlabeled target data.

where, N is the total number of image samples, K is the total number of image categories.

Therefore, the loss of specific visual modalities is:

where the value of β can be adjusted based on the influence of supervised source domain data.

3.4 Overall Training Objective

We train our method with the supervised loss and above regularizations in an end-to-end manner. As depicted in Figure 2, the pre-trained text and vision encoder are frozen. We optimize parameters of a counterfactual disentangled network and discrimination knowledge maintenance network denoted as θcau and θmod respectively. Combining Eq. (6), Eq. (9), Eq. (13), we define the following optimization problem:

The overall loss of the entire network is:

4 Experiments

4.1 Datasets and Implementation Details

Datasets. We evaluated our method on two prominent public datasets: Office-Home (Venkateswara et al., 2017), and VisDA-2017 (Peng et al., 2018). Office-Home comprises images across four distinct domains, encompassing 65 categories. VisDA-2017 features 152,000 synthetic images in the source domain and 55,000 real images in the target domain.

Training Configuration. We employed ResNet50 [He et al., 2016], and ViT-B/16 [Dosovitskiy et al., 2021] as visual encoders V, and utilized a pre-trained CLIP text encoder as the text encoder T . The parameters of both encoders were frozen during training. We set the length of the learnable text prompt L to 32. For optimization, we used the Adam optimizer [Kingma and Ba, 2017] with an initial learning rate of 3e - 3 and trained the model for 30 epochs with 32 batch sizes. All experiments were conducted on an NVIDIA RTX A6000 GPU.

4.2 Comparasion with State-of-the-Arts

Results on Office-Home dataset. For a fair comparison, we evaluate our approach against a comprehensive set of traditional UDA methods based on CNN and transformer and CLIP-based approaches. Initially, we employ ResNet-50 as the backbone for our visual encoder, with the results detailed in Table 1. Next, we use ViT-B/16 as the visual encoder, and the corresponding results are shown in Table 2. Overall, our method consistently demonstrates outperformed performance. Especially, our method achieved the best results in 8 out of the 12 tasks evaluated.

Results on VisDA-2017 dataset. Similarly, we conduct comparisons using ResNet-101 and ViT-B/16 as visual encoders.The results are shown in Table 3 and Table 4, respectively. Our method further outperforms the state-of-the-art method in terms of average accuracy, by 2.2% and 0.9% respectively. The recognition accuracy has been improved across specific categories, such as "bicycle", "knife", "train", etc. Collectively, these improvements underscore the signif-

• (a) Weight values on loss

(b) Different prompt lengths

Figure 4: The results of different weight values on losses.

icance of extracting causal factors and maintaining discriminative knowledge.

4.3 Ablation Study

Effectiveness of each module. We conduct ablation experiments to substantiate the efficacy of counterfactual disentanglement and discriminative knowledge maintenance. The experimental outcomes are detailed in Table 5. Utilizing DAPL

[Ge et al., 2023] as the baseline. A major observation is that the omission of any module invariably resulted in performance decrements of varying magnitudes, emphasizing the positive contribution of each module to the overall performance. Similarly, as our method is based on CLIP, we demonstrated the results of our ablation experiment using different backbone networks.

Parameter sensitivity analysis. We also conduct sensitivity analysis on key parameters, focusing on the weights of causal loss and knowledge maintenance loss, as well as the length of learnable tokens. According to Figure 4 (a), we can see that the weights of both losses affect the performance of the model to varying degrees. When γ1 is set to 8 and γ2 is set to 4, the experimental results are the best. According to Figure 4 (b), we can see that as the length of the learnable token increases, the accuracy of the model also increases. However, when the length of the learnable token exceeds 32, the accuracy of the model gradually begins to decline. Therefore, we set the length of the learnable token to 32 in the experiment.

Model Complexity. As shown in Figure 5, the result of our method requires fewer 4.23%, 17.82%, 18.12%, 18.27% than DANN [Ganin and Lempitsky, 2015], CLIP [Radford et al., 2021], DAPL [Ge et al., 2023], and AD-CLIP [Singha et al., 2023] computational resources than most others.

4.4 Analysis

Confusion matrix visualization. We generated confusion matrices using our method on the Office-Home dataset and the VisDA-2017 dataset. As depicted in (a) and (b) of Figure 6, both matrices exhibit a pronounced diagonal structure.

Feature map visualization. Figure 7 shows the visualization of the model's feature map on the Office-Home dataset. The left portion illustrates the adaptation from the "Clipart" domain to the "Product" domain, and the right portion depicts the adaptation from the "Art" domain to the "Real World" domain. The feature map reveals that our method facilitates the

(a) Visual embedding

(b) Text embedding

Figure 8: t-SNE visualizations of visual embedding and text embeddings from "Art" to "Real World" domains on Office-Home dataset.

model in primarily learning category-related causal features. By mitigating the interference from background information, our approach enhances the model's generalization capability.

t-SNE visualization. Figure 8 shows the t-SNE visualization of the "Art" to "Real World" task on the Office-Home dataset. The visualization reveals that our method successfully aligns visual and text embeddings, achieving both domain invariance and discriminability. The overlapping areas indicate the capability to learn domain invariant knowledge. Moreover, the text embeddings exhibit marked separation between distinct categories, which underscores the model's strong discriminative power in differentiating various categories.

5 Conclusion

In this paper, we introduce a novel UDA method, which employs counterfactual disentanglement and discriminative knowledge maintenance. By leveraging counterfactuals to disentangle domain-specific and semantics-related features, our approach mitigates the confounding effects of domainrelated features. To further address the challenge of classifying samples with ambiguous semantic information, we propose modal adaptive fusion to enhance the extraction of class-discriminative features. The proposed method lead to improved feature disentanglement and class recognizability. Extensive experiments demonstrate that our method consistently outperforms two strong baselines, offering a robust method for UDA to harness source code and pre-trained VLM knowledge. Future work will explore the application of UDA in open-world scenarios, where models need to adapt to dynamically changing environments.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 62272461, Grant 62172417, Grant 62276266 and Grant 62277046; in part by the Double First-Class Project of China University of Mining and Technology for Independent Innovation and Social Service under Grant 2022ZZCX06; and in part by the Six Talent Peaks Project in Jiangsu Province under Grant 2015-DZXX010 and Grant 2018-XYDXX-044; and in part by the Postgraduate Research & Practice Innovation Program of Jiangsu Prowince under Grant 123625314; and Funded by the Graduate Innovation Program of China University of Mining and Technology under Grant 123625314.