Semi-supervised Domain Adaptation via Prototype-based Multi-level Learning

1 Introduction

The deep neural network has achieved great success in various visual tasks. However, due to the high cost of labeled data it requires and the degradation of model performance when deploying models in a new domain (target), unsupervised domain adaptation (UDA) has been proposed. To further improve the model performance, many semi-supervised domain adaptation (SSDA) works are proposed by adding a few (e.g., one sample per class) labeled target samples based on UDA. Compared to UDA, the key to SSDA is whether the model can make better use of the additional limited target samples to help the model better fit the learned features from the source domain to the target domain.

∗Corresponding Author

In recent years, to help model knowledge transfer, some existing SSDA studies make better use of unlabeled samples in target domains by combining self-supervised learning [PérezCarrasco et al., 2022] and semi-supervised learning [Li et al., 2021a; Yan et al., 2022]. However, they ignore the potential of labeled samples in the target domain. Although [Li et al., 2021b; Singh, 2021] use few target samples by building a prototype, they only use the prototype from one level unilaterally and ignore the additional knowledge learned from other levels (e.g. intra-domain level). At the same time, it is not robust to use only a very limited number of label or pseudo-label samples to build prototypes. To fill this gap, we propose a robust prototype-based multi-level learning (ProML) framework. Our prototype is updated by both labeled and unlabeled target samples which helps the model to transfer knowledge across domains more robustly. Our ProML makes full use of the target prototype from three levels: intra-domain, inter-domain, and batch-wise to help the model better transfer knowledge through target samples, as shown in Figure 1.

Specifically, at the intra-domain level, we propose a pseudo-labeling strategy based on the optimal transport to help the model align the data distribution of labeled and unlabeled samples in the target domain, obtaining a more compact intra-domain feature distribution and robust pseudo-labels. This method improves the pseudo-labels of target samples by resolving the optimal transport plan between the weakly augmented target samples and the target prototype. At the same time, the transport plan is also applied to the strongly augment view to form a consistency constraint. At the inter-domain level, we cross-align the source samples with the same class of the target prototype. Inter-domain prototype alignment helps the model learn better about cross-domain knowledge transfer and category alignment. At the batch level, different from work [Yan et al., 2022], we consider a mini-batch of samples to calculate class correlation matrices between predictions with different augments from two perspectives of prediction probability and prototype similarity, increasing the correlation in the same class and reducing the correlation from different classes. Classifiers from different perspectives represent features comprehensive to learn the relationship between the batch-wise target samples. Combining the above three, through our ProML framework, the model can capture the knowledge of target domain samples from different levels, to better enable the model to learn more comprehensive and complementary domain-adaptive knowledge.

Our main contributions can be summarized as follows:

1. We propose a novel Prototype-based Multi-level Learning (ProML) framework, which exploits the potential of labeled target samples by making full use of the target prototype frommultiple levels.
2. We propose a pseudo-label aggregation based on intradomain optimal transport to SSDA, which helps the model form a more compact target domain robustly.
3. We propose a batch-wise dual consistency, which helps the model learn more distinctive target representations from different perspectives.
4. Experiments have shown that our ProML implements a new state-of-the-art result in most SSDA problems.

2 RelatedWorks

Unsupervised domain adaptation. Unsupervised domain adaptation (UDA) [Pan et al., 2010] has made some exciting achievements. It aims to transfer the knowledge learned from the labeled source domain to the unlabeled target domain. The methods based on feature space alignment [Long et al., 2017; Sun and Saenko, 2016] usually use joint distribution to make the two domains as close as possible in the feature space, to reduce the differences between the two domains. Some methods based on GAN [Chen et al., 2019; Long et al., 2018; Pei et al., 2018] are also popular. They calibrate the distribution of the source domain and target domain by generating codes that cannot be distinguished from the perspective of the discriminator, which is trained to classify the target domain. Despite significant advances in UDA, UDA methods do not perform well in SSDA [Saito et al., 2019], which is our main reason for focusing on SSDA.

Semi-supervised domain adaptation. In SSDA, the problem assumes that there are a few labeled samples in the target domain. It can be thought of as a combination of semisupervised learning (SSL) and DA. Some methods [Saito et al., 2019; Li et al., 2021a] propose using Adversarial training to adjust source and target distributions, clustering similar target samples together through different clustering strategies, and separating different samples to reduce intra-domain gaps.

Some methods [Li et al., 2021b; Singh, 2021] use the idea of self-supervised learning to build target prototypes and help model transformation across domains by comparing learning methods. They only build the prototype with very limited labels or high-noise pseudo-labels, ignoring the importance of different sample relationships. The methods based on SSL [Pérez-Carrasco et al., 2022] use different losses to enhance the consistency between the feature representations of unlabeled samples and [Yan et al., 2022] standardizes the consistency of different views of target domain samples at three levels, which facilitates learning more representative target features from each other. However, they all ignore the importance of making further use of the labeled target samples. In this work, we use the prototype to tap the potential of labeled target samples from multiple levels to help the model learn comprehensive and complementary feature representations.

3 Methodology

In this section, we first specify the definition and notation of SSDA and then introduce the proposed Prototype-based Multi-level Learning (ProML) framework from three levels.

3.1 Framework

In SSDA, we sample datasets from two different distributions. In this setting, we can access labeled source samples Ds = {(xsi, ysi)}Ns i=1 sampled from the source distribution Ps(X, Y ). We also have a limited number of labeled target samples Dlt = {(xlti , ylti )}N l t i=1 and a large number of unlabeled samples Du t = {(xuti )}N u t i=1 both from the target distribution Pt(X, Y ). This two distribution satisfy: Ps(Y ) = Pt(Y ) and Ps(X|Y ) ̸= Pt(X|Y ). The framework is composed of a feature extractor G and a linear classifier F. An outline of our ProML framework is illustrated in Figure 2.

Following [Sohn et al., 2020], we feed the labeled source samples and the labeled target samples into G to obtain their feature representations fs, f l t ∈ Rd , and then get the probability prediction ps, plt ∈ Rc through F . Similarly, we generate two different views, for each unlabeled target sample xuti by weak and strong augment, represented as xuw ti and xus ti . Then the target samples of these two views are fed to the same feature extractor G to generate representations f uw ti and f us ti . Finally, the probability prediction puw ti , pus ti are obtained through the same classifier F. We can calculate the standard cross-entropy loss by using labeled sample pairs from two domains and consider a simple pseudo-label regularization with different views of unlabeled samples. Specifically, we optimize the following baseline:

where 1 is an indicator function, y is the label of labeled samples x, and τ1 is the pseudo-label threshold of the regularization. In the following, we will further introduce our framework from three different levels.

3.2 Intra-domain Pseudo-label Aggregation

A compact target domain can help the model make better use of pseudo-labels and cross-domain knowledge transfer. We propose a novel pseudo-label aggregation strategy to align the unlabeled and labeled target data robustly and accurately.

To make full use of the labeled target samples, we initialize the target prototype:

where ylti is the label of the i-th labeled target sample. The optimal transport (OT) is used to align the inter-domain level representation for domain adaptation [Courty et al., 2017; Yan et al., 2022]. It finds the optimal coupling plan γ0 according to the given transport function to minimize the total transport cost. To make better use of the additional data provided by SSDA, we propose to apply OT to the intra-domain feature aggregation instead of the inter-domain. For labeled and weakly augment unlabeled samples, we assume that:

where γ0i,j means the transport plan between the i-th labeled sample and the j-th unlabeled sample in the target domain, ⟨·, ·⟩F represents Frobenius inner product, 1d is 1s' ddimensional vector, µs ∈ RNs , µt ∈ RN u t are the empirical distributions of labeled and unlabeled samples respectively, and the default is a uniform distribution. Cw ∈ RNs×N u t is a cost matrix for indicating each transportation. In consideration of the robustness and accuracy of the transportation, we propose to use the target prototype to replace the labeled samples in the transport plan. This not only makes the feature representation more representative but also alleviates the negative alignment caused by the lack of some classes in the minibatch of samples. The cost matrix is as follows:

where each element in the cost matrix Cwi,j represents the degree of difference between the i-th target prototype cit and the j-th weak augment target feature f uw tj . The resulting OT plan γ0 helps the model better align the distribution of different samples in the target domain.

To further improve the robustness of the prototype, we add unlabeled target samples to the update of the prototype and regard the OT plan as a supplement to the pseudo-label strategy. Specifically, we consider the following pseudo-label strategy:

where τ2 is the confidence threshold for using a pseudolabel supplementation strategy, ˜ puw ti is the maximum prediction probability of the i-th unlabeled weak sample in all categories, and argmax(·) represents the category corresponding to the prediction probability. When the distribution of unlabeled data is close to that of labeled data, this module can better use the prototype initialized by labeled data to assign pseudo-labels to unlabeled data through the proposed strategy. We make better use of labeled data to improve the robustness of prototypes through the pseudo-label strategy.

For each minibatch, we calculate the feature average of labeled and pseudo-labeled target samples for clustering, and then use the exponential moving average to update the target prototype during training:

where ˜ckt is the target prototype clustered by target samples with labels and pseudo-labels in this minibatch. Under the guidance of the optimal coupling γ0, the feature representation of each strongly augment view target sample can have a consistent mapping plan with the weak augment view, forming a consistency constraint:

where Cs is the similarity matrix between the strong augment target sample and the target prototype similar to Equation 5. As mentioned above, the model can effectively align the distribution of the target prototype and unlabeled samples and pay more attention to the details of the intra-domain level.

3.3 Inter-domain Alignment

Knowledge transfer is an essential capability of models in SSDA. For the cross-domain level, we can use the relationship between the target prototype and the source sample to naturally transfer the knowledge of the model in the source domain to the target domain.

Specifically, for each category of target prototype, we can calculate the softmax of the similarity of the corresponding target prototype in the feature space according to the category of the source sample:

where sim(·, ·) means cosine similarity, fsi is the feature of i-th source samples, T1 is a scale temperature, and Ns is the number of source samples. Then, we can calculate the crossdomain prototype alignment loss for each source sample:

where C is the number of classes. Our prototype is not only composed of labeled targets but also adopts a more robust pseudo-label update strategy to alleviate negative transfer caused by the dispersion of target samples.

3.4 Batch-wise Dual Consistency

To make the model more comprehensively learn the representation in the target domain, different from [Yan et al., 2022], we consider the dual relationship between target features at the batch level. We increase the confidence difference between different views by sharpening the confidence and then model the clustering of each class as the classification confidence and prototype similarity for all samples of that class in the batch. Maintain the consistency of the strong and weak views of the class allocation as the positive pair in the same batch, and reduce the similarity between different classes as the negative pair, instead of the sample-wise contrastive learning.

Given the prediction Puw ∈ RNt×C of the target sample of a batch, we use the sharpening function to reduce the entropy of label distribution, widening the gap between different confidence, and enhancing the contrast between different views:

where T2 is a temperature hyperparameter, C is the number of classes. However, it is one-sided to optimize only from the perspective of linear classifiers. Due to the existence of the target prototype, we propose that batch level relationships be learned dually not only from the perspective of linear classifier but also from the perspective of prototype similarity. Since linear classifier can assign learnable parameters to each class, while prototype-based classifier only relies on excellent feature representation, we calculate the cross-correlation matrix of strong and weak views from the perspective of linear classifier and prototype-based classifier respectively:

where ˆPuw , ˆPus are sharpening batch probability matrices and Suw , Sus are batch similarity matrices with the prototype from weak and strong views similar to Equation 9. Rws l , Rws p are asymmetric matrices, and each element represents two similarities of different views at the batch level. From this, the dual contrast loss of batch class can be calculated:

where ϕ(·) is a normalized function that keeps the row total as 1. I ∈ RC×C is the identity matrix, and || · ||1 represents the sum of the absolute values of the matrix.

Dual consistency can benefit the framework in learning cross-domain knowledge from two perspectives. Linear classifiers can use learnable parameters to focus more on the distinguishing dimensions represented by features while suppressing unrelated feature dimensions by assigning higher or lower weights to different dimensions. In contrast, prototypebased classifiers cannot take advantage of this which requires more discriminative feature representation [Xu et al., 2022].

At the same time, due to the setting of DA, the prediction of the linear classifier may be biased toward the source domain. Comparing with target prototype similarity can alleviate the inaccurate contrast caused by over-fitting the source domain. We combine the two to encourage the model to learn a more discriminative and accurate relationship between batch level objectives from different perspectives.

3.5 Overall Framework and Training Objective

To sum up, the overall training objectives of the framework are as follows:

where λintra, λinter and λbatch are the hyper-parameters that balance different levels. We train the model in our framework by employing the overall training loss described in Equation 14.

4 Experiments

4.1 Experimental Setup

Datasets. We verified our ProML framework on three popular SSDA datasets. DomainNet is the latest large-scale multi-source domain adaptive dataset, with 6 domains and 345 categories [Peng et al., 2019]. According to [Saito et al., 2019], four fields (Real, Clipart, Painting, Sketch) and 126 categories were selected for SSDA evaluation. VisDA2017 consists of 150k synthetic images and 55k real images, including two domains and 12 categories [Peng et al., 2017]. For each category of each dataset, we randomly select one or three labeled samples (1-shot or 3-shot) as labeled target samples. Office-Home is also a mainstream domain adaptive dataset, including 4 domains (Real, Clipart, Art, Product) and 65 classes [Venkateswara et al., 2017]. We follow the standard of most SSDA work [Saito et al., 2019], and report the overall accuracy as an indicator of DomainNet and OfficeHome, and report the average class accuracy (MCA) as an evaluation indicator of VisDA2017.

Implementation details. 1 To ensure the fairness of the experiment, similar to [Saito et al., 2019; Li et al., 2021a], we use ResNet34 [He et al., 2016] pre-trained on Imagenet as the backbone of the model. The settings of batch size, optimizer, feature size, and learning rate are also consistent with [Saito et al., 2019]. Similar to [Yan et al., 2022], RandomFlip and RandomCrop are used as weak image augment methods, and RandAugment [Cubuk et al., 2020] is used as strong augment methods. The momentum α used to update prototypes is set to 0.9, and the threshold τ1 is set to 0.95 and τ2 is set to 0.4 for VisDA2017, 0.3 for DomainNet and 0.1 for Office-Home. For the OT solver, we solve it through Sinkhorn-Knopp iteration [Fatras et al., 2021]. The temperature T1 and T2 in the similarity function are set to 0.05 and 0.1. Balance hyparameter λintra and λinter is set to 1. λbatch is set to 1 for DomainNet, 1 for Office-Home, and 0.1 for VisDA2017.

4.2 Analysis of Experimental Results

In the experiments, we compare our ProML with two baselines: S+T uses only source and labeled target data, and ENT [Grandvalet and Bengio, 2004] uses entropy minimization for unlabeled samples; and several popular SSDA methods, i.e., DANN [Ganin et al., 2016], MME [Saito et al., 2019], BiAT [Jiang et al., 2020], APE [Kim and Kim, 2020], Con2DA [Pérez-Carrasco et al., 2022], CDAC [Li et al., 2021a], DECOTA [Yang et al., 2021], CLDA [Singh, 2021], ECACL [Li et al., 2021b] and MCL [Yan et al., 2022].

DomainNet. As shown in Tab. 1, our proposed ProML achieves 76.1% and 77.4% average accuracy and SOTA per-

formance in 7 scenarios of 1-shot and 3-shot, respectively. VisDA2017. VisDA2017 demonstrates the validity of ProML in Tab. 2, our ProML achieves 87.6% MCA in 1-shot and 88.4% in 3-shot, which are superior to the SOTA methods.

Office-Home. As shown in Tab. 3, ProML outperforms the existing SOTA methods in both 1-shot and 3-shot scenarios, with accuracy reaching 74.6% and 77.8%, respectively.

1 https://bupt-ai-cz.github.io/ProML/

4.3 Additional Analysis

Ablation studies. We performed ablation experiments on VisDA2017 at the settings of 1-shot and 3-shot, as shown in Tab. 4. Row ② ③ ④ show that each component can be significantly improved. Row ⑤ ⑥ ⑦ show that each combination still improves performance, indicating the universality of the proposed module. Due to the integrity of our framework, one aspect of the lack of consideration may not be optimal in performance, but the best performance can be achieved when all components are activated.

Prototype-based contrast and update. Tab. 5 shows ablation studies of prototype-based classifier and updated procedure. Row ① shows only linear prediction, and its performance degrades a lot, which proves that the knowledge learned only by using a linear classifier is limited. The comparison with row ③ proves that the model will benefit from the dynamic update of the prototype. The global static prototype that only uses labeled data is not robust, so its generalization ability is limited. Row ② shows only prototypebased predictions and dynamic updates during training. Its performance is much lower, which means that without the help of a linear classifier, the prototype-based classifier cannot learn consistency under limited label data settings, so it cannot make full use of the additional knowledge brought by prototypes. Row ④ only adds the prototype-based classifier on row ①, but its performance does not increase but decreases, which indicates that biased prototypes will bring negative knowledge transfer. Row ⑤ achieves the optimal performance after integrating all methods, which shows that our method can complement knowledge from multi-level.

Intra-domain OT pseudo-labels. Tab. 6 shows the pseudolabel accuracy of different methods. By making the target domain more compact, our strategy gives more accurate pseudolabels at different training stages than simple methods only using maximum linear classifier prediction (Linear) or maximum prototype similarity (Proto.) under the same confidence threshold τ2 in Equation 6.

Convergence analysis. To further analyze the convergence of our method, we describe the feature t-SNE [Van der Maaten and Hinton, 2008] visualization of source and target domains during different training times in Figure 3. We randomly selected seven categories of the feature on Domain- Net R→S for clearer visualization. Figure 3(a) clearly shows the initial domain difference between the source and the target domain. Early feature descriptions often show many misaligned source and target clusters, so the model initially performs poorly in the target domain. As the training progresses, it can be seen from Figure 3(b) and Figure 3(c) that our method converges and aggregates the feature of the target domain. Accumulated good target feature is obtained and it proves that our method can obtain a compact target domain distribution, as shown in Figure 3(d).

Parameter sensitivity. We will use the pseudo-labels threshold τ2 of intra-domain OT strategy in Equation 6 as a variable, analyze changes in MCA with VisDA2017 under the settings of 1-shot and 3-shot, and experimental results support our argument. As shown in Figure 4, with the change of the threshold, MCA floats within 1%, which demonstrates the robustness of the strategy. When this strategy is not used, i.e. τ2 = 0.95, MCA decreased significantly. This result is reasonable and shows that this strategy can help the model make use of unlabeled target samples.

Impact of the number of target labels. We studied the performance impact of different target label numbers on Domain R→S. From Figure 5, all methods can improve performance by labeling more target samples. By contrast, our ProML always achieves optimal performance in all cases. This confirms that our method can make use of a flexible number of labeled target samples to help model knowledge transfer.

5 Conclusion

In this paper, we propose a novel prototype-based multi-level learning (ProML) framework to make better use of target samples. ProML leverages prototypes constructed from target samples at three levels, (i) intra-domain level, aligns labeled and unlabeled sample distributions within the target domain using pseudo-label aggregation help models based on optimal transport, (ii) inter-domain level, it aligns source and target domain, and (iii) batch level, it learns compact classes and partition clusters from two dual perspectives. Extensive experimental studies have demonstrated its advantages.

Acknowledgements

This work was supported by the National Key R&D Program of China (2021ZD0109802), by the National Natural Science Foundation of China (81972248).