Domain Adaptation via Maximizing Surrogate Mutual Information

1 Introduction

Inspired by human beings' ability to transfer knowledge across domains and tasks, transfer learning is proposed to leverage knowledge from source domain and task to improve performance on target domain and task. However, in practice, labeled data are often limited on target domains. To address such situation, unsupervised domain adaptation (UDA), a category of transfer learning methods [Long et al., 2015; Long et al., 2017b; Ganin et al., 2016], attempts to enhance knowledge transfer from labeled source domain to target domain by leveraging unlabeled target domain data.

Most previous work is based on the data shift assumption, i.e., the label space maintains the same across domains, but the data distribution conditioned on labels varies. Under this hypothesis, domain alignment and class-level method are used to improve generalization across source and target feature distributions. Domain alignment minimizes the discrepancy between the feature distributions of two domains [Long et al., 2015; Ganin et al., 2016; Long et al., 2017b], while class-level methods work on conditional distributions. Conditional alignment aligns conditional distributions and use pseudo-labels to estimate conditional distribution on target domain [Long et al., 2018; Li et al., 2020c; Chen et al., 2020a]. However, the conditional distributions from different categories tend to mix together, leading to performance drop.

Contrastive learning based methods resolve this issue by discriminating features from different classes [Luo et al., 2020], but still face the problem of pseudo-label precision. In addition, most of the class-level methods lack solid theoretical explanations for the relationship between cross domain generalization and their objectives. Some works [Chen et al., 2019; Xie et al., 2018] yield some intuition for conditional alignment and contrastive learning, but the relation between their training objectives and cross-domain error remains unclear.

In this work, we aim to address the generalization problem in domain adaptation from an information theory perspective. In failed case of domain adaptation, as shown in Figure 1, features from the same class do not represent each other well and this inspires us to use mutual information to reduce this confusion. Our motivation is to find more representative features for both domains by maximizing mutual information between features of the same class (on both source and target domains). Therefore, if our classifier can accurately predict features on source domain, then it would also function well on target domains where features share enough information with the source features.

Based on the above motivation, we propose Surrogate Information Domain Adaptation (SIDA), a general domain adaptation framework with strong theoretical guarantees. SIDA achieves adaptation by maximizing the mutual information (MI) between features within the same class, which improves the generalization of the model to the target domain. Furthermore, a surrogate distribution is constructed to approximate the unlabeled target distribution, which improves flexibility for selecting data and assists MI estimation. Also, our theoretical analyses directly establish a bound between MI of features and target expected risk, giving a proof that our model can improve generalization across domain.

Our novelties and contributions are summarized as follows:

• We propose a novel framework to achieve domain adaptation by maximizing surrogate MI.
• We establish an expected risk upper bound based on feature MI and surrogate distribution bias for UDA. This provides theoretical guarantee for our framework.
• Experiment results on three challenging benchmarks demonstrate that our method performs favorably against state-of-art class-level UDA models.

2 RelatedWork

Domain Adaptation Prior works are based on two major assumptions: (1) the label shift hypothesis, where the label distribution changes, and (2) a more common data shift hypothesis where we only study the shift in conditional distribution under the premise that the label distribution is fixed. Our work focuses on the data shift hypothesis, and previous work following this line can be divided into two major categories: domain alignment methods which align marginal distributions, and class-level methods addressing the alignment of conditional distributions.

Domain alignment methods minimize the difference between feature distributions of source and target domains with various metrics, e.g. maximum mean discrepancy (MMD) [Long et al., 2015], JS divergence [Ganin et al., 2016] estimated by adversarial discriminator, Wasserstein metric and others. Maximum Mean Discrepancy (MMD) is applied to measure the discrepancy in marginal distributions [Long et al., 2015; Long et al., 2017b]. Adversarial domain adaptation plays a mini-max game to learn domain-invariant features [Ganin et al., 2016; Li et al., 2020b].

Class-level methods align the conditional distribution based on pseudo-labels [Li et al., 2020c; Chen et al., 2020a; Luo et al., 2020; Li et al., 2020a; Tang and Jia, 2020; Xu et al., 2019]. Conditional alignment methods [Xie et al., 2018; Long et al., 2018] minimize the discrepancy between conditional distributions. In class-level methods, conditional distributions are assigned by pseudo-labels. The accuracy of pseudo-labels greatly influences performance and later works construct more accurate pseudo-labels [Chen et al., 2020a]. However, the major problemwith this method is that error in conditional alignment leads to distribution overlap of features from different class, resulting in low discriminability on target domain. Contrastive learning addresses this problem by maximizing the discrepancy between different classes [Luo et al., 2020; Li et al., 2020a]. However, the performance of contrastive learning also relies on pseudo-labeling.

In addition, previous class-level works provide weak theoretical support for cross-domain generalization. Prior works mainly focus on domain alignment [Ben-David et al., 2007; Redko et al., 2020]. Some works [Chen et al., 2019; Xie et al., 2018] consider optimal classification on both domains, and yield some intuitive explanation for conditional alignment and contrastive learning, but the relation between their objective function and theoretical cross-domain error remains unclear.

Information Maximization Principle Recently, mutual information maximization (InfoMax) for representation learning has attracted lots of attention [Chen et al., 2020b; Hjelm et al., 2018; Khosla et al., 2020]. The intuition is that two features belonging to different classes should be discriminable while features of the same class should resemble each other. The InfoMax principle provides a general framework for learning informative representations, and provides consistent boosts in various downstream tasks.

We facilitate domain adaptation with MI maximization, i.e. maximizing the MI between features of the same class. Some works solve domain adaptation problem via information theoretical methods [Thota and Leontidis, 2021; Chen and Liu, 2020; Park et al., 2020], which maximize MI using InfoNCE estimation [Poole et al., 2019]. As far as we know, we are the first to provide theoretical guarantee for the target domain expected risk based on MI. Compared with InfoNCE, the variational lower bound of MI we use is tighter [Poole et al., 2019]. We also construct a surrogate distribution as a substitute for unlabeled target domain, which is more suitable for MI estimation.

3 Preliminaries

3.1 Notations and Problem Setting

Let X be the data space and Y be the label space. In UDA, there is a source distribution PS(X, Y ) and a target distribution PT (X, Y ) on X × Y. Note that distributions are also referred to as domains in UDA. Our work is based on the data shift hypothesis, which assumes PS(X, Y ) and PT (X, Y ) satisfy the following properties: PT (Y ) = PS(Y ) and PT (X|Y ) ̸= PS(X|Y ) .

In our work, we focus on classification tasks. Under this setting, an algorithm has access to nS labeled samples {(xiS , yiS )}nS i=1 ∼ PS(X, Y ) and nT unlabeled samples {(xiT )}nT i=1 ∼ PT (X), and outputs a hypothesis composed of an encoder G and a classifier F. Let Z be the feature space. The encoder maps data to feature space, denoted by G : X → Z. Then the classifier maps the feature to a corresponding class, F : Z → Y.

For brevity, given encoder G and data-label distribution P(X, Y ), denote the distribution of G-encoded feature and label by P G, i.e. P G(z, y) = P(x = G-1 (z), y).

Let F be a hypothesis, and P be the distribution of feature and label. The expected risk of a F w.r.t. P is denoted as

where δF(z)(y) equals to 1 if y = F(z) and equals 0 in else cases. Our objective is to minimize the expected risk of F on target feature distribution encoded by G,

We maximize MI between features on both source and target domain, regardless of which domain they come from. So we introduce mixture distribution S + T of both domain, which is

Note that because PS(y) = PT (y) = PS+T (y), PS+T (x|y) = 12 PS(x | y) + 1 2 PT (x | y). Define the distribution of features from the same class as

which means the feature z1 and z2 are sampled independently from the conditional distribution of the same class, with equal probability from source domain and target domain.

MI between features is maximized within the mixture distribution, as formalized bellow:

However, due to the lack of target domain labels, P G S+T is hard to model and thereby it is infeasible to estimate I G S+T directly. To address this problem, we propose a surrogate joint distribution Q(Z, Y ) as the substitute for target domain P G T . Then the mixture distribution becomes P G S+Q = 12 (P G S +Q), and the objective becomes maximizing I G S+Q(Z1;Z2). The construction and optimization of the surrogate joint distribution is explained in Section 4.3.

Theoretical Motivation for MI Maximization

We use theoretical bound to demonstrate the motivation for using MI maximization. Our theoretical results prove that minimizing the expected risk on the target domain can be naturally transformed into MI maximization and expected risk minimization on the source domain, which explains why MI maximization is pivotal to our framework. The proofs are in appendix.

Definition 1 (H∆H-Divergence). Let F1 ∈ H, F2 ∈ H be two hypotheses in hypothesis space H : Z → Y.

4 Methodology

4.1 Overview

In UDA task, the model needs to generalize across different domains with varying distributions; thus the encoder needs to extract appropriate features that are transferable across domains. The challenges of class-level adaptation are two folds: learning transferable features, and modeling P G T (Z|Y ) without label information.

To solve the first problem, we use MI based methods. Following the InfoMax principle, we maximize the mutual information between features from the same class on the target and source mixture distribution. This encourages the features of the source domain to carry more information about the features of the same class in target domain, and thus provides opportunities for transferring classifier across domains.

As for the second challenge, we first revisit the data shift hypothesis. The distribution of labels P (Y ) remains independent of domains; therefore the key is to model the conditional distribution P(Z|Y ) on the target domain. However, modeling P(Z|Y ) is intractable, since labels on the target domain are inaccessible. To tackle this problem, we model a surrogate distribution Q(Z|Y ) instead.

We introduce the goal of maximizing MI in section 4.2, and theoretically explain how MI affects domain adaptation risk. In Section 4.3, we will introduce the model in detail, including the variational estimation of MI, the modeling of the surrogate distribution, and the optimization of the loss function of the model.

4.2 Mutual InformationMaximization

MI measures the degree to which two variables can predict each other. Inspired by InfoMax principle [Hjelm et al., 2018], we maximize the MI between the features within the same class. It encourages features from different classes to be discriminable from each other.

Define ϵP (F1, F2) as the disagreement between hypotheses F1, F2 w.r.t. distribution P on Z, ϵP (F1, F2) ≜ Ez∼P ∣∣δF1(z) - δF2(z) ∣∣. H∆H-divergence, which is the discrepancy of two distributions P1, P2 w.r.t. any hypothesis F1 - F2 where F1, F2 ∈ H, is defined as dH∆H(P1, P2) ≜ 2 supF1,F2∈H |ϵP1 (F1, F2) - ϵP2 (F1, F2)|.

Theorem1 (Bound of Target Domain expected risk). The expected risk on target domain can be upper-bounded by the negative MI between features, and H∆H -divergence between features of two domains:

The proof is in appendix. We give an explanation of the conditions for the upper bound to be equal. I G S+T (Z1; Z2) is a lower bound of I G S+T (Z; Y ), and it measures how much uncertainty of Y is reduced by knowing the feature, and it's equal to H(Y ) if and only if P G S+T (Y |Z) is deterministic, i.e., P G S+T (Y |Z) is δ distribution, which means P G S (Y |Z) = P G T (Y |Z) = δY (Z). Thus if the H∆H -divergence is zero, i.e., P G S (Z) = P G T (Z), then it's ensured that P G S (Z, Y ) = P G T (Z, Y ), and ϵP G T (F) = ϵP G S (F).

This upper bound decomposes the cross-domain generalization error into the divergence of feature marginal distribution and MI of features. It emphasizes that in addition to the divergence of the feature marginal distributions, only a MI term is enough for knowledge transfer across domains.

In this work, we minimize the expected risk on the source domain and maximize MI, for minimizing the upper bound of expected risk on target domain. Due to the lack of labels on target domain, we estimate MI based on surrogate distribution Q. The expected risk upper bound based on surrogate MI is further derived as follows.

Definition 2 (L1-distance). Define L1-distance of P1, P2 as d1 (P1, P2) ≜ 2 supB∈B |PrP1 [B] - PrP2 [B]| where B is the set of measurable subsets under P1 and P2.

Theorem 2 (Bound Estimation with Surrogate Distribution). Let B ≜ d1(P G T (Z),Q(Z)) + ϵP G T (Q(Y |Z)) be the bias of surrogate distribution Q w.r.t target distribution. The expected risk on target domain can be upper-bounded by the negative surrogate MI between features, H∆H -divergence between source and target domain, and additional bias of surrogate domain:

The proof is in appendix. This theorem supports the feasibility of domain adaptation via maximizing surrogate MI I G S+Q(Z1; Z2). The bias of surrogate distribution is expressed in terms d1(P G T (Z), Q(Z)) + ϵP G T (Q(Y |Z)), where the first term is the distance between the surrogate and target feature marginal distribution, and the second term is the risk of conditional label surrogate distribution. To minimize the upper bound, the bias of the surrogate distribution should be small.

Bias equal to zero if and only if surrogate feature distribution and conditional label distribution are the same as target distribution, i.e., P G T = Q, where surrogate distribution does not introduce errors.

4.3 SIDA Framework

We employ MI maximization and surrogate distribution in our SIDA framework, as shown in Figure 2. During training, a surrogate distribution is first built from target and source data via optimizing w.r.t. Laplacian and MI. Then a mixture data distribution is created by encoding source data to features and sampling target features from the surrogate distribution. The encoder is optimized by maximizing MI, and minimizing classification error. The overall loss is:

Lmodel = LClassify+α1LMI+α2LAuxiliary+LLaplacian. (8) We elaborate each module in the following sections, and introduce the optimization of surrogate distribution in the last sections.

Mutual Information Estimation

Several MI estimation and optimization methods are proposed in deep learning [Poole et al., 2019]. In this work, we use the following variational lower bound of MI as proposed in [Nguyen et al., 2010]:

where f is a score function in Z × Z → R. The equality holds when ef(z1,z2) EP(z1)ef(z1,z2) = P(z1|z2) P(z1) and EP(z1)EP(z2)ef(z1,z2) = e. The proof is in appendix. Therefore maximizing MI can be transformed into maximizing its lower bound, and the loss is:

where f(z1, z2) is constructed as T m2 m1 (|z1 -z2|2). T m2 m1 is a threshold function, i.e., T m2 m1 (a) = max(m1, min(m2, a)).

Surrogate Distribution Construction

We decompose the surrogate distribution Q(Z, Y ) into two factors Q(Z, Y ) = Q(Y )Q(Z|Y ), and describe the construction of two factors individually.

According to the data shift assumption, PT (Y ) is similar to PS(Y ), thus Q(Y ) should be similar to PS(Y ). However, source distribution may suffer from the class imbalance problem, which will harm the performance on classes with fewer data. A common solution to this problem is class-balanced sampling, which samples data on each class uniformly. In this work, for the balance across different classes, the marginal distribution PS(Y ) and Q(Y ) are both considered as uniform distribution.

As for the second term, the conditional surrogate distribution Q(Z|Y ) is constructed by weighted sampling method. We need to construct the Q(Z|Y ) to calculate Eq. 10, which takes the form of expectation, and only needs samples from Q(Z|Y ) to estimate. Instead of explicitly modeling Q(Y |Z), we use the ideas of importance sampling. For each class, the surrogate conditional distribution Q(Z|yj) is constructed by weighted sampling from target features. Thus Q(Z|Y ) is a distribution on target features{G(xiT )}nT i=1 , and parameterized by W ∈ RnT×nY , where nY is the number of labels:

Compared with pseudo-labeling, our estimation method has the following advantages: (1) The surrogate marginal distribution of feature Q(Z) = ∑Y Q(Z|Y )P(Y ) is not fixed, which enables us to select features more flexibly. (2)The construction process of the surrogate distribution makes MI estimation I(Z1, Z2) more convenient. Our surrogate distribution Q(Z|Y ) provides weights so that weighted sampling can be performed directly.

The challenge is to optimize the sampling probability weights Wij so as to minimize the bias of the surrogate distribution. We propose to optimize this distribution via Laplacian regularization as well as MI, which is explained in details in the following section.

Surrogate Distribution Loss

Inspired by semi-supervised learning, we expect that the surrogate distribution is consistent with the clustering structure of the feature distribution, based on the assumption that the feature is well-structured and clustered according to class, regardless of domains. We employ Laplacian regularization to capture the manifold clustering structure of feature distribution.

Let A ∈ RnT×nT be the adjacent matrix of target features, where the entry Aij measures how similar G(xiT ) and G(xjT ) are, and D = Diag(A1) is the degree matrix, i.e. Dii = ∑j Aij and Dij = 0, ∀i ̸= j. We construct A as K-nearest graph on target features, and the Laplacian regularization of W is defined as

where L is the normalized Laplacian matrix L = I - D- 12AD- 12 . This regularization encourages Wik and Wjk to be similar if feature G(xiT ) is similar to G(xjT ). It also enables the conditional surrogate distribution to spread uniformly on a connected region.

Classification and Auxiliary Loss

The model is optimized in supervised manner on the source domain. The classification loss is the standard cross-entropy loss via class-balanced sampling.

And we use auxiliary classification loss on pseudo-labels from the surrogate distribution, as the classifier will benefit from label information of the surrogate distribution. We use mean square error (MSE) for pseudo-labels, which is more robust to noise than cross entropy loss.

Optimization of Surrogate Distribution

We optimize both LLaplacian and LMI w.r.t. W for a structured and informative surrogate distribution. At the beginning of each epoch, W is initialized by K-means clustering and filtered by the distance to the clustering centers, i.e. ˜ Wi,j = 1µj nearest to G(xi)∆1d(G(xi),µj)<θ, where µj is the j-th clustering center during clustering, and normalized as Wi,j = ˜ Wi,j .

To minimize two losses w.r.t W, the gradients are derived analytically. The derivation is in appendix.

Based on the gradient of these two losses, we perform Tstep descent update of W with learning rate η1 and η2 respectively, and each step we project W back to the probability simplex. See appendix for details.

5 Experiments

In this section, We evaluate the proposed method on three public domain adaptation benchmarks, compared with recent state-of-the-art UDA methods. We conduct extensive ablation study to discuss our method.

5.1 Datasets

VisDA-2017 [Peng et al., 2017] is a challenging benchmark for UDA with the domain shift from synthetic data to real imagery. It contains 152,397 training images and 55,388 validation images across 12 classes. Following the training and testing protocol in [Long et al., 2017a], the model is trained on labeled training and unlabeled validation set and tested on the validation set.

Office-31 [Saenko et al., 2010] is a commonly used dataset for UDA, where images are collected from three distinct domains: Amazon (A), Webcam (W) and DSLR (D). The dataset consists of 4,110 images belonging to 31 classes, and is imbalanced across domains, with 2,817 images in A domain, 795 images in W domain, and 498 images in D domain. Our method is evaluated on all six transfer tasks. We follow the standard protocol for UDA [Long et al., 2017b] to use all labeled source samples and all unlabeled target samples as the training data.

Office-Home [Venkateswara et al., 2017] is another classical dataset with 15,500 images of 65 categories in office and home settings, consisting of 4 domains including Artistic images (A), Clip Art images (C), Product images (P) and Real-World images (R). Following the common protocol, all 65 categories from the four domains are used for evaluation of UDA, forming 12 transfer tasks.

5.2 Implementation details

For each transfer task, mean (±std) over 5 runs of the test accuracy are reported. We use the ImageNet pre-trained ResNet-50 [He et al., 2016] without final classifier layer as the encoder network G for Office-31 and Office-Home, and ResNet-101 for VisDA-2017. The details of experiments are in appendix. The code is available at https://github.com/zhaoht/SIDA.

5.3 Baselines

We compare our approach with the state of the arts. Domain alignment methods include DAN [Long et al., 2015], DANN [Ganin et al., 2016], JAN [Long et al., 2017b]. Class-level methods include conditional alignment methods (CDAN [Long et al., 2018], DCAN [Li et al., 2020c], ALDA [Chen et al., 2020a]), and contrastive methods (DRMEA [Luo et al., 2020], ETD [Li et al., 2020a], DADA [Tang and Jia, 2020], SAFN [Xu et al., 2019]). We only report available results in each baseline. We use NA, DA, CA, CT to note no adaptation method, domain alignment methods, conditional alignment methods and contrastive methods respectively.

5.4 Results and Comparative Analysis

In this section we will present our results and compare with other methods for evaluation on three standard benchmarks mentioned earlier. We report average classification accuracies with standard deviations. Results of other methods are collected from original papers or the follow-up work. We provide visualizations of the features learned by the model in the appendix.

VisDA-2017 Table 1 summarizes our experimental results on the challenging VisDA-2017 dataset. For fair comparison, all methods listed here use ResNet-101 as the backbone network. Note that SIDA outperforms baseline models with an average accuracy of 84.0, surpassing the previous best result reported by +4%.

Office-31 The unsupervised adaptation results on six Office-31 transfer tasks based on ResNet-50 are reported in Table 2. As the data reveals, the average accuracy of SIDA is 90.4, the best among all compared methods. It is noteworthy that our proposed method substantially improves the classification accuracy on hard transfer tasks, e.g. W→A, A→D, and D→A, where source and target data are not similar. Our model also achieves comparable classification performance on easy transfer tasks, e.g. D→W, W→D, and A→W. Our improvements are mainly on hard settings.

Office-Home Results on Office-Home using ResNet-50 backbone are reported in Table 3. It can be observed that SIDA exceeds all compared methods on most transfer tasks with an average accuracy of 71.2. The performance reveals the importance of maximizing MI between feature in difficult domain-adaptation tasks which contain more categories.

In summary, our surrogate MI maximization approach achieves competitive performance compared to traditional alignment based methods and recent pseudo-label based methods for UDA. It underlines the validity of using information theory methods for UDA via MI maximization.

5.5 Ablation Study

In this section, to evaluate how different components of our work contribute to the final performance, we conduct ablation study for SIDA on Office-31. We mainly focus on harder transfer tasks, e.g. A→W , A→D, D→A and W→A. We investigate different combinations of two components:MI maximization and surrogate distribution (SD). Note that without surrogate distribution, we use pseudo label computed by the same method as surrogate distribution initialization to estimate MI. The average classification accuracy on four tasks are in Table 4.

From the results, we can observe that the model with MI maximization outperforms the base model without the two components by about 2.5% on average, which demonstrates the effectiveness of the maximization strategy. The surrogate distribution also improves the average performance by 1.1% compared to base model, confirming that the surrogate distribution improves the estimation quality of target domain compared to pseudo label method. The combination of two components yields the highest improvement.

6 Conclusion and FutureWork

In this work, we introduce a novel framework of unsupervised domain adaptation and provide theoretical analysis to validate our optimization objectives. Experiments show that our approach gives competitive results compared to state-ofthe-art unsupervised adaptation methods on standard domain adaptation tasks. One unresolved problem is to integrate the domain discrepancy in target risk upper bound into mutual information framework. This problem is left for future work.