Multi-level Consistency Learning for Semi-supervised Domain Adaptation

1 Introduction

Semi-supervised Domain Adaptation (SSDA) has attracted lots of attention due to its promising performance compared with Unsupervised Domain Adaptation (UDA). Since only a few labeled target samples are available in SSDA, it can be viewed as a combination of UDA and Semi-supervised Learning (SSL) subtasks. Thus, it is straightforward to borrow ideas from SSL to address SSDA. For example, [Yang et al., 2021] use mixup and co-training to boost the performance of SSDA. More recently, [Li et al., 2021a] utilize consistency regularization [Xie et al., 2019; Sohn et al., 2020] and achieve state-of-the-art performance on several SSDA benchmarks.

∗Equal contribution

†Corresponding author

Although [Li et al., 2021a] show that consistency regularization benefits SSDA, it has only been naively applied at the sample level, i.e., enforcing the prediction of an unlabeled target sample to be consistent with those of its different views (obtained via augmentations). However, in SSDA, it is also important to attend the inter-domain knowledge transfer to better leverage the source supervision and intra-domain representation learning for both compact and discriminative class clusters. Therefore, to fully utilize consistency regularization in SSDA, we propose a Multi-level Consistency Learning (MCL) framework, as shown in Fig. 1, which regularizes the model with (i) inter-domain level alignment consistency, (ii) intra-domain level class-wise assignment consistency, and (iii) sample level prediction consistency.

At the inter-domain level, we use consistency regularization to robustly and accurately align the source and target domain. We regularize the consistency between one view (weakly augmented target samples) to source mapping and another view (strongly augmented target samples) to source mapping. Specifically, we first solve the optimal transport mapping between one view of the target samples and the source prototypes, and leverage the solved mapping to cluster another view to the source prototypes. At the intradomain level, we propose a novel class-wise contrastive clustering loss function to help the network learn both discriminative and compact feature representations in the target domain. This is implemented by computing the class-wise cross-correlation between the predictions from two views (target samples) in a mini-batch, and increasing the correlation of two views in the same class while reducing those from different classes. At the sample level, we follow standard practice and enforce the predictions of the weakly-augmented and the strongly-augmented views to be consistent. In this way, the sample-level information can be fully probed to learn better feature representations for each single unlabeled target sample. With our multi-level consistency learning framework, the model is encouraged to capture complementary and comprehensive information at different levels, thereby learning more well-aligned and discriminative target features.

In summary, our work has the following contributions: (1) We propose a novel Multi-level Consistency Learning (MCL) framework for SSDA tasks. Our framework benefits SSDA by fully utilizing the potential of consistency regularization at three levels: the inter-domain level, the intra-domain level, and the sample level. (2) Our inter-domain level consistency learning method consists of a novel mapping and clustering strategy, and a novel prototype-based optimal transport method, which yields more robust and accurate domain alignment. (3) Our intra-domain level consistency learning method consists of a novel class-wise contrastive clustering loss, which helps the network learn more compact intraclass and discriminative inter-class target features. (4) Extensive experiments on SSDA benchmarks demonstrate that our MCL framework achieves a new state-of-the-art result.

2 RelatedWork

Semi-supervised Domain Adaptation (SSDA) Unlike UDA, SSDA assumes that there exist several labeled samples in the target domain, e.g. 3 shots per class. Many works have recently been proposed to tackle the SSDA problem [Saito et al., 2019; Yang et al., 2021; Qin et al., 2021; Li et al., 2021b; Kim and Kim, 2020; Li et al., 2021a; Jiang et al., 2020; Yao et al., 2022; Singh, 2021]. [Saito et al., 2019] first proposes to solve the SSDA problem by min-maximizing the entropy of target unlabeled data. [Jiang et al., 2020] train the network with virtual adversarial perturbations so that the network can generate robust features. [Yang et al., 2021] decompose SSDA into an inter-domain UDA subtask and an intradomain SSL subtask, and utilize co-training to make the two networks teach each other and learn complementary information. [Qin et al., 2021] use two classifiers to learn scattered source features and compact target features, thereby enclosing target features by the expanded boundary of source features. [Li et al., 2021a] propose to reduce the intra-domain gap by gradually clustering similar target samples together while pushing dissimilar samples away. In this work, instead of focusing on either inter-domain level or intra-domain level, we address both of them simultaneously so that the learning of both levels will benefit each other to learn more representative target features.

Consistency Regularization Consistency regularization is a widely used technique in semi-supervised learning [Sohn et al., 2020] and self-supervised learning [Chen et al., 2020]. In semi-supervised learning [Sohn et al., 2020], enforcing the consistency among different augmentations of unlabeled samples can effectively boost the model performance. Recent works [Chen et al., 2020] have shown that the consistency regularization objective can guide the model to learn meaningful representations in an unsupervised manner without any annotations, which inspires its application in label-scarce learning scenarios, e.g. domain adaptation, semi-supervised learning. Recently, some works have demonstrated that consistency regularization can also be utilized in domain adaptation tasks [French et al., 2018]. Although our work is also built upon consistency regularization, rather than regularizing the sample-wise predictions only, we additionally regularize the consistency at both inter-domain and intra-domain levels.

3 Methodology

In this section, we first present the preliminaries including the definition of SSDA and associated notations, and then introduce the proposed Multi-level Consistency Learning (MCL) framework in detail. An outline of our MCL framework is illustrated in Fig. 2.

3.1 Preliminaries

In the semi-supervised domain adaptation (SSDA) problem, the source domain Ds = {Xs, Ys} is fully labeled, and the target domain Dt = {Xt, Ytl} contains a few labels Ytl for partial samples Xtl, so Dt = {Xtl, Ytl}∪{Xtu}, where {Xtu} is the unlabeled target sample set. SSDA methods aim to learn a classifier with high accuracy on the target domain using the rich data-label pairs {Xs, Ys} and few ones {Xtl, Ytl}. We assume that (i) the source and target domains share the same label space of C classes; (ii) there are only a few labels for each class in the target domain, e.g. one label (1-shot) or three labels (3-shot).

SSDA model usually consists of two components, a feature extractor G and a classifier F, parameterized by θG and θF, respectively. Following [Sohn et al., 2020], we first generate two views for each target sample xt ∈ Xt, denoted as xAt and xBt , through the weak (view A) and strong (view B) augmentation, respectively. The two views are then fed into G to obtain their feature representations f A t , f B t ∈ Rd , which finally go through F to get the probabilistic prediction pAt , pBt ∈ RC . This process can be formulated as:

where σ(·) is the softmax function. Similarly, in the source branch, each source sample xs is fed into the same feature extractor G to generate fs ∈ Rd , and then into the classifier F to obtain the prediction ps ∈ RC .

To train the model, each sample-label pair (x, y) (from either the source or target domain) can be used to minimize a standard cross-entropy (CE) classification loss as follows:

where p(x) is the prediction of sample x through G and F. As detailed below, to fully utilize the unlabeled target data Xtu, our MCL framework regularizes the consistencies in SSDA at three levels.

3.2 Inter-domain Alignment Consistency

At the inter-domain level, we propose a novel consistency regularization method to robustly and accurately align samples from the target domain to those of the source domain, so that the knowledge in the source domain can be better transferred into the target domain. To obtain such a high-quality alignment, we compute the mapping plan from two views of the target sample to the source sample, and leverage the consistency between the two mappings to guide the learning process. Specifically, in each iteration, we first compute the mapping from the view A of a target sample xAt to a source sample xs, according to which the representations of its view B, xBt and xs are clustered closer. Accordingly, our method alternates the following two steps throughout the training:

Step 1. View-A to SourceMapping. We formulate the domain alignment as an optimal transport problem [Courty et al., 2016], which aims to find the optimal sample-level coupling plan γ ∗ between source samples and view-A target samples by minimizing the total transport cost:

where γ ∗ i,j denotes the transport plan between the ith and jth sample in the source and target domain, respectively, ⟨·, ·⟩F denotes the Frobenius inner product, 1d is a d-dimensional vector of 1's, and µs ∈ Rns , µt ∈ Rnt are the empirical distributions of source and target samples, respectively, which are usually set as uniform distributions. CA ∈ Rn×n is the cost function matrix to indicate the transport cost between each source-target sample pair, with each element CAi,j in the matrix is inversely proportional to the cosine distance between the i-th source feature fsi and the j-th view-A target feature f A tj , computed as:

where more similar a feature vector pair is, a smaller transport cost can be obtained. With the cost matrix, CA computed, the total transport cost can be minimized to obtain the optimal coupling plan γ ∗ , of which each element indicates a correspondence score for alignment.

Step 2. View-B to Source Clustering. With the guidance of optimal coupling γ ∗ , the feature representation of each strongly-augmented view-B sample can be clustered toward the source prototype. In this way, the learned view-B features can have a consistent mapping plan with the weaklyaugmented view A. Hence the objective of the inter-domain alignment consistency learning is to minimize the loss function as below:min

so that G can generate well-aligned view-B features that are close to the corresponding source sample feature.

Prototype-based OT. To further improve the accuracy and robustness, we propose to replace the source samples used in our OT method with source prototypes. Our source prototypes bring two benefits: (i) their features are more representative; (ii) they always cover all classes and avoid the misalignment error caused by missing classes in a mini-batch of source samples. To this end, for each class c ∈ {1, 2, · · · , C}, we compute its prototype hc by averaging the feature representations of all source samples in c and substitute all fsi with hi, i ∈ {1, 2, · · · ,C} in Eq. 3 and 5. In Step 2 of our method, after the update of G, each hi in {hc} is updated with the batch mean ˆhi and a momentum m after each iteration:

Thus, we alternate between these two steps during training: (i) with fixed neural network parameters, solve the optimal coupling γ ∗ for view A (Eq. 7), and (ii) with the optimal coupling fixed, optimize the alignment loss for view B (Eq. 6).

3.3 Intra-domain Class-wise Consistency

To facilitate representation learning in the target domain, we incorporate consistency regularization into the intra-domain level. Specifically, we aim to learn both compact and discriminative class-wise clusters in the target domain, i.e., the clusters of the same class from two views should be grouped, while those of different classes should be pushed away from each other. Existing unsupervised contrastive learning methods focus on sample-level representation learning, which still hardly learn highly discriminative clusters for each class [Khosla et al., 2020].

Instead of the sample-wise contrastive learning, we propose a novel class-wise contrastive clustering loss function, by modeling the cluster of each class as all samples' classification confidence of this class in a batch, named as batchwise class assignment (illustrated as the column bounded by a red rectangle in Fig .1). The class-wise contrastive clustering has two purposes, (i) keeping the consistency of two views for the same batch-wise class assignment (positive pairs), and (ii) enlarging the distance between the batch-wise assignment of different classes (negative pairs).

Formally, given a batch of target samples' predictions P ∈ Rn×C , where n is the mini-batch size and Pi,j represents the confidence of classifying the ith sample into the jth class. With the cluster assumption that each feature of samples in the same class should form a high-density cluster, we regard each column of P as the batch-wise assignment of one class. Although the columns of P will dynamically change with the iteration of mini-batches during training, the class assignment in different views of the same image should be similar, while dissimilar to other images. Therefore, we compute the crosscorrelation matrix Λ ∈ RC×C between two views A and B to measure the class-wise similarity, formulated as:

where PA and PB are the batch prediction from view A and B, respectively. Λ is an asymmetry matrix with each element Λi,j measuring the similarity between PA's ith column and PB 's jth column. The objective is to increase the correlation values between two views of the same class assignment (diagonal values) while reducing ones for different classes' views (off-diagonal values), which yields the loss function of intra-domain prediction consistency learning:

where ϕ(·) is a normalization function to keep the row summation as 1, i.e. divide the elements with its row summation, I ∈ RC×C is an identity matrix, and ∥ · ∥1 denotes the summation of the absolute values of the matrix. Note that the loss function is the summation of two terms for respectively normalizing the rows and columns of the correlation matrix Λ. Such two normalization operations can alleviate the influence of class imbalance in a batch, i.e., the values in the class assignment may be too large or small according to the number of samples that belong to this class. Intuitively, the normalized row of correlation values can be regarded as the scores of matching one class assignment with each column in the other view to be a positive pair. Thus the dual design helps bi-directional matching score computation, i.e., the ith row in ϕ(Λ) represents the matching scores from PA's ith column to PB 's each column, while the one in ϕ(Λ⊤) represents matching in the inverse direction.

Note that minimizing Eq. 11 has the following properties: i) The rows of PA and PB are encouraged to be sharper, which implements entropy minimization [Grandvalet et al., 2005] in label-scarce scenarios. ii) For each sample, the prediction of view A and B are encouraged to become similar, which coincides with the consistency regularization [Chen et al., 2020] that helps the network to learn better representations. iii) It avoids a trivial solution that predicts all samples as the same class: the diagonal values of the normalized cross-correlation matrix are enforced to be close to 1, which punishes the trivial solution by a large loss.

3.4 Sample Prediction Consistency

Similar to [Li et al., 2021a], we regularize the consistency at the sample level via pseudo labeling: the prediction of view A of a target sample is set as the pseudo label of view B of the target sample. Specifically, for each sample xtu, we compute the cross-entropy loss as:

where pAi and pBi are the ith elements in the prediction of xtu from view A and view B, respectively, 1(·) is the indication function, and τ is the threshold of confidence score.

3.5 Overall Objective Function

Therefore, the overall objective can be formulated as:

where λ1 and λ2 are the hyper-paramters to balance Linter and Lintra, respectively.

4 Experiments

4.1 Experiment Setups

Datasets. We evaluate our proposed MCL on several popular benchmark datasets, including VisDA2017 [Peng et al., 2017], DomainNet [Peng et al., 2019], and OfficeHome [Venkateswara et al., 2017]. DomainNet was first introduced as the multi-source domain adaptation benchmark, and it consists of 6 domains with 345 categories. Following recent works [Saito et al., 2019; Li et al., 2021a], the Real, Clipart, Painting, Sketch domains with 126 categories are selected for evaluations. VisDA2017 is a popular synthetic-toreal domain adaptation benchmark, which consists of 150k synthetic and 55k real images from 12 categories. OfficeHome is a popular domain adaptation benchmark, which consists of 4 domains (Real, Clipart, Product, Art) and 65 categories. Following most recent works [Yang et al., 2021;

Li et al., 2021a], we conduct 1-shot and 3-shot experiments on all datasets. Note that we follow most UDA works [Long et al., 2013] to report the Mean Class-wise Accuracy (MCA) as the evaluation metric for VisDA2017 and overall accuracy for DomainNet and Office-Home datasets.

Implementation details. Similar to [Saito et al., 2019; Li et al., 2021a], we use a ResNet34 as the backbone for most experiments. The backbone is pre-trained on Imagenet [Deng et al., 2009]. The batch size, optimizer, feature size, learning rate, etc. are set as same as in [Saito et al., 2019]. Similar to [Li et al., 2021a], the threshold τ (Eq. 12) is set as 0.95, and we use the softmax temperature T to control the sharpness of the prediction for the thresholding operation (1 for Domainnet, 1.25 for Office-Home and VisDA). The loss weight balancing hyperparameters λ1 is set as 1, and λ2 is set to 1 for DomainNet, 0.2 for Office-Home , and 0.1 for VisDA. We use RandomFlip and RandomCrop as the augmentation methods for view A and RandAugment [Cubuk et al., 2020] for view B. For the OT solver, we introduce soft regularizations to the OT objective to make it a convex optimization problem that can be efficiently solved by Sinkhorn-Knopp iterations [Fatras et al., 2021]. Moreover, the momentumm used to update source prototypes is set to 0.9. We use Pytorch and POT 1 to implement our MCL framework.

4.2 Comparative Results

We compare our MCL framework with i) two baseline methods S+T and ENT [Grandvalet et al., 2005]; and ii)

1 https://pythonot.github.io

several existing methods, including DANN [Ganin et al., 2016], MME [Saito et al., 2019], UODA [Qin et al., 2021], BiAT [Jiang et al., 2020], MetaMME [Li and Hospedales, 2020], APE [Kim and Kim, 2020], ATDOC [Liang et al., 2021], STar [Singh et al., 2021], CDAC [Li et al., 2021a], and DECOTA [Yang et al., 2021]. Between the two baselines, "S+T" uses only source samples and labeled target samples in the training; "ENT" uses entropy minimization for unlabeled samples [Grandvalet et al., 2005]. Note that all methods are implemented on a ResNet34 backbone for a fair comparison.

Domainnet. Results on the DomainNet dataset are shown in Tab. 1. To our knowledge, the proposed MCL achieves a new state-of-the-art (SOTA) performance, 74.4% and 76.5% mean accuracy over 7 adaptation scenarios, under the 1-shot and 3-shot setting, respectively.

Office-Home. Results on the Office-Home dataset further validate the effectiveness of MCL. As shown in Tab. 2, MCL consistently outperforms other methods on both the 1-shot and 3-shot settings, and on all 12 adaptation scenarios. The mean accuracy of MCL reaches 74.0% and 77.1%, which outperform the SOTA performance by around 3% and 1.4%, under the 1-shot and 3-shot settings, respectively.

VisDA. As shown in Tab. 3, MCL outperforms all existing SOTA methods in both 1-shot and 3-shot settings. Our MCL achieves 86.3%MCA under the 1-shot setting, and 87.3% for the 3-shot setting, significantly surpassing existing methods.

4.3 Analysis

Ablation studies. We conduct ablation studies on the Office-Home of A → R and C → P, and under both 1-shot and 3-shot settings, as shown in Tab. 4. Row (b)(c)(d) show that each component can yeild significant improvement. Row (e)(f)(g) show that each combination can still improve the performance, which indicates the versatility of the proposed modules. The best performance is achieved with all components activated.

Prototype-based OT. In the proposed MCL, prototypebased OT is adopted to compute the mapping plan between target samples and source prototypes, which generates more representative source features for transport to learn robust inter-domain alignment. In comparison, we also implement sample-wise OT on the DomainNet dataset under the 3-shot setting, with other hyper-parameters the same as MCL. Evaluation results on two adaptation scenarios are listed in Tab. 5, sample-wise OT ("Standard") results in a drop of 3.5% (C → S) and 2.1% (P → R), which verify our claim.

Class-wise contrastive clustering. In the design of classwise contrastive clustering loss, an alternative is to compute the sample-wise correlation matrix Λ′ = PAPB⊤ ∈ Rn×n instead, where n is the batch size. We claim that such a sample-wise operation is inferior, and experimental results on the DomainNet dataset under the 3-shot setting support our argument, as listed in Tab. 5. One potential reason might be that the class-wise inner product focuses on clustering each class for higher compactness, whereas the sample-wise one pays more attention to the correlation between individual samples, which is less effective when labels are scarce and compact representations are difficult to generate.

5 Conclusion

In this paper, we presented a novel Multi-level Consistency Learning (MCL) framework for the SSDA problem. MCL consists of three levels of consistency regularization, (i) inter-domain alignment consistency for learning robust inter-domain alignment via a prototype-based optimal transport method, (ii) intra-domain class-wise consistency for learning both compact and discriminative class clusters, and (iii) sample-level prediction consistency to better leverage instance-wise information. Our MCL approach well leverages information at multiple levels, with extensive experiments and ablation studies solidly proving its superiority.

Acknowledgments

The work was supported in part by Chinese KeyArea Research and Development program of Guangdong Province (2020B0101350001), by the Basic Research Project No. HZQB-KCZYZ-2021067 of Hetao Shenzhen-HK S&T Cooperation Zone, by National Key R&D Program of China with grant No. 2018YFB1800800, by Shenzhen Outstanding Talents Training Fund 202002, by Guangdong Research Projects No. 2017ZT07X152 and No. 2019CX01X104, by the National Natural Science Foundation of China under Grant No.61976250 and No.U1811463, by the Guangzhou Science and technology project under Grant No.202102020633 and by Shenzhen Research Institute of Big Data. Thanks to the ITSO in CUHKSZ for their HighPerformance Computing Services.