Exploring Safety Supervision for Continual Test-time Domain Adaptation

1 Introduction

Deep neural networks have achieved remarkable success in visual tasks when training and test data follow the same distribution. These networks, however, suffer from the generalization problem in the presence of domain shift. For example, a classification network pre-trained in the normal, natural images domain may not recognize the corrupted images due to domain shift [Dong et al., 2020; Yang et al., 2022]. Thus, an adaptation is necessary to transfer knowledge from the source domain to the target one by reducing the domain shift. However, the label of the target domain is usually unavailable, so the problem is known as Unsupervised Domain Adaptation (UDA) [Ganin and Lempitsky, 2015; Prabhu et al., 2021; Sun et al., 2019; Dong et al., 2021]. In addition, the source data is often inaccessible during inference time due to privacy or business problems, making the adaptation problem more challenging but more realistic. Therefore, such adaptation problem becomes SourceFree/Test-Time domain Adaptation (TTA) [Chen et al., 2022; Yang et al., 2021; Kurmi et al., 2021; Kundu et al., 2020; Liu et al., 2021] where only the source model and unlabeled target data are available in the adaptation process.

∗Corresponding author.

Existing TTA methods usually solve the domain shift problem by updating the adapted model parameters using the generated pseudo-labels or entropy regularization. These selftraining-based methods are effective when the distribution of data in the target domain is fixed, but when the distribution of the target domain is constantly changing [Wang et al., 2022; Prabhu et al., 2021], these methods become unstable. Noise problem [Wang et al., 2022] caused by the change of target domain distribution seriously affects the adaptation process. CoTTA [Wang et al., 2022] first defines this kind of problem as Continual Test-Time Domain Adaptation, where a source pre-trained model needs to adapt to a stream of continually changing target test data without using any source data. It uses a weight-average teacher network to improve the quality of generated pseudo-labels. However, the accurate optimization of a network requires the joint action of label and loss function. The pseudo-labels generated by existing methods become noisier in the changing target domain environment because of domain shifts in the adaptation process. Thus, refining pseudo-label learning requires high-quality pseudolabels and a reliable label selection strategy.

In this paper, we propose a simple yet effective framework for continual test-time domain adaptation, which refines the pseudo-label learning process for exploring safety supervision from three perspectives: Label Safety, Sample Safety, and Parameter Safety, to alleviate the instantaneous and long-term impact of noisy pseudo-labels. In terms of label safety, noisy pseudo-labels will immediately affect the loss calculation in network optimization. Thus, we define and adjust the confidence threshold in a self-adaptive manner to restrain the noisy pseudo-labels in the test-time learning status. Notably, instead of simply presenting a constant hyper-parameter and treating all classes equally, we select an independent threshold for each class through global and local strategies to choose more reliable pseudo-labels as supervision information. Based on such thresholds, we construct a soft-weighted contrastive learning module for the sample safety of contrastive learning module, which pulls the reliable same-class samples closer and discriminates against uncorrelated samples. Moreover, the Fourier transform is introduced into the contrastive learning process as a strong augmentation to explore domain-invariant predictions. In this way, we can alleviate instantaneous negative learning and improve learning efficiency.

From the perspective of long-term impact, as the model continually adapts to the target domain with changing distributions, the knowledge from the pre-trained model is constantly forgotten due to error accumulation caused by the noisy pseudo-labels. To slow down the accumulation of errors and combat knowledge forgetting, we propose a Soft Weight Alignment (SWA) strategy, which continuously distills knowledge from the source pre-trained model. In detail, we normalize the distance between the parameters of the adapted model and the pre-trained model. In doing so, we continuously distill knowledge from the source pre-trained model and can effectively alleviate the long-term knowledgeforgetting problem. Extensive experimental results demonstrate that our method achieves state-of-the-art performance on several datasets.

To sum up, our contributions are as follows:

• We design a novel yet efficient continual test-time domain adaptation method for exploring safety supervision. The learnable thresholds are tuned in a local and global manner according to the test-time learning status to enhance label safety, sample safety, and parameter safety, further alleviating instantaneous negative learning and long-term knowledge-forgetting problems.
• We introduce a soft-weighted contrastive learning module, which pulls the reliable same-class samples closer and alleviates the false negative samples. The learnable thresholds and pseudo-labels are utilized to select more discriminative positive and precise negative pairs for contrastive learning. Moreover, we employ the Fourier transform to augment the positives, exploring domain-invariant predictions.
• We propose a Soft Weight Alignment strategy, which continuously distills knowledge from the source pretrained model. We normalize the distance between the adapted and pre-trained models' parameters. In doing so, we continuously distill knowledge from the source pre-trained model and can effectively alleviate the longterm knowledge-forgetting problem.

2 RelatedWork

2.1 Domain Adaptation

Domain adaptation refers to the goal of learning a concept from labeled data in the source domain that performs well on different but related target domains. The critical problem of domain adaptation lies in the misalignment between the feature and label spaces of the source and target domains. To solve this problem, some domain adaptation methods guide the deep model to learn domain invariant representation and classifiers. Specifically, some works [Ganin and Lempitsky, 2015; Tzeng et al., 2017; Ganin et al., 2016] utilize adversarial training to align feature distribution with a domain discriminator, and some works constrain the cross-domain feature space by entropy constraint [Grandvalet and Bengio, 2004; Saito et al., 2019] or maximum prediction rank [Cui et al., 2020; Yang et al., 2019].

2.2 Test-Time Domain Adaptation

Recently, some works on test-time domain adaptation focus on a more challenging setting where only the source model and unlabeled target data are available. Some testtime domain adaptation methods [Kurmi et al., 2021; Li et al., 2020] utilize generative models to achieve the feature alignment between the source and target domain in the absence of source data. Kurmi et al. adopt the trained classifier [Kurmi et al., 2021] to generate samples from the source classes. This method learns the joint distribution of data by using the energy-based modeling of the trained classifier. Yeh et al. propose the method of Source-data-free Feature Alignment (SoFA) [Yeh et al., 2021] to extract features with class semantics, thus realizing domain adaptation in the absence of the source data. In addition, some methods achieve test-time domain adaptation by finetuning the source model with the help of target data and do not require explicit domain alignment. Test entropy minimization (TENT) [Wang et al., 2020] introduces entropy minimization as a test-time optimization objective, which estimates normalization statistics and optimizes channel-wise affine transformations to update online on each batch. Source HypOthesis Transfer (SHOT) [Liang et al., 2020] aims to learn the optimal target-specific feature learning module to fit the source hypothesis. It is worth mentioning that SHOT requires using source data to train a specialized source model, so it cannot support an arbitrary source pre-trained model. Zhou et al. present Bayesian Adaptation for Covariate Shift (BACS) [Zhou and Levine, 2021] to obtain both improved accuracy and well-calibrated uncertainty estimates when faced with domain shift.

Most test-time adaptation methods only consider the offline scenario, where the full set of test data is provided during the training process. Further, CoTTA [Wang et al., 2022] extends test-time adaptation from offline scenario to online continual scenario. It considers a more challenging but more realistic problem named Continual Test-Time Domain Adaptation, where a source pre-trained model needs to adapt to a stream of continually changing target test data without using any source data. In this paper, we also focus on such a more realistic setting than standard test-time adaptation.

3 ProposedMethod

3.1 ProblemDefinition

Following [Wang et al., 2022], we consider a continual testtime domain adaptation setting, where a pre-trained model needs to adapt to a continually changing target domain in an online fashion when the absence of source data. Consider a pre-trained model Fθ(x) with parameter θ trained on the source data (X S , Y S ). Unlabeled target domain data X T is provided sequentially and the data distribution of X T is continually changing. At inference time t, when the unlabeled target data XT (t) = [xT1 (t), ..., xTB (t)] is sent to the model Fθt , where B is the number of samples. The model Fθt needs to make the prediction Fθt (XT (t)) and adapts itself accordingly for the next input (θt → θt+1). It is worth noting that the total evaluation process is online, and the model only has access to the data XT (t) of the current time step t. The framework is shown in Fig. 1.

3.2 Self-adaptive Thresholding Cross-Entropy Loss

In this section, we detail the self-adaptive thresholding crossentropy loss. In the process of adapting to a continually changing target domain, the quality of pseudo-labels decreases significantly because of the distribution shift. To improve the quality of pseudo-labels, following [Wang et al., 2022], we use a weight-averaged teacher model Fˆ θ to generate the pseudo-labels. At the time t = 0, the teacher network is initialized by the pre-trained source model. Then, at time t, the teacher model Fˆ θ generates the pseudo-label ˆ yT (t) to help the learning process of the student model (adapted model) Fθt . Specifically, we compute the cross-entropy loss between the output of the student model and the pseudo-label ˆ yT (t). The learning process can be denoted as follows:

ˆ yT bk (t) is the soft predication of the teacher model of class k ∈ K, and yT bk (t) is the prediction of the student model.

During this process, the accumulation of erroneous pseudo-labels can severely disturb the model's predictive performance. This work focuses on pseudo-labeling using crossentropy loss with a confidence threshold. Instead of simply presenting a constant hyperparameter and treating all classes equally, we advocate that the key to determining thresholds is that thresholds should reflect the test learning status. Thus, we present self-adaptive thresholding that automatically defines and adaptively adjusts the confidence threshold for each class by leveraging the current predictions during test-time training.

The global threshold should represent the model's confidence in the test data, reflecting the overall learning status. We set the global threshold τt as the average confidence from the model on test data, and estimate the global confidence as the exponential moving average (EMA) at each time step, where t represents the t-th time step. The global threshold τt is defined and adjusted as:

In addition to the global threshold, the local threshold is utilized to modulate the global threshold in a class-specific fashion to account for the intra-class diversity and the possible class adjacency. We compute the expectation of the model's predictions on each class k to estimate the classspecific learning status:

After integrating the global and local thresholds, we can obtain the final self-adaptive threshold of each class k.

Finally, we adopt the self-adaptive threshold to select reliable samples for supervision, and the training objective of Eq. 1 can be denoted as follows:

3.3 Soft-weighted Contrastive Learning

Such pseudo-labels can be further utilized to promise the model more substantial representation power, while previous methods have not achieved it. Under the context of contrastive learning, in particular, these semantic class structures can give helpful guidance in selecting contrastive pairs with similar semantics to improve training efficiency. Specifically, the similarity of the samples b and q can be calculated with cosine similarity using pseudo-labels.

We adopt the weighted similarity matrix wt to guide the traditional contrastive loss, which can be written as follows:

where zT b (t) = Fθ(xTb (t)). We then introduce the components of the objective in detail.

Fourier Augmentation. Some work [Xu et al., 2021; Lu et al., 2022] have demonstrated that Fourier phase information contained high-level semantics and was not easily affected by domain shifts, which may be a kind of universal domain-invariant features. Thus, we introduce the Fourier transform as a new augmentation and attempt to learn domain-invariant representations. The Fourier transformation FFT(x) for a single-channel two-dimensional data x is formulated as:

where u and v are indices, H and W are the height and the width, respectively. For data xTb (t) with several channels, the Fourier transformation for each channel is computed independently to obtain the corresponding phase information ˜xTb (t).

Positives. Except for introducing Fourier transform as a positive sample, we attempt to present more positive samples by utilizing the correlation between samples during the instantaneous learning process. According to the pseudo-labels output by the teacher model, we select reliable samples of the same class with b as the positive sample set through the learned threshold.

Negtatives. The traditional contrastive loss strives to maximize the cosine distances between b and every q in the batch. Instead, we argue that not pushing away same-class pairs helps learn better semantically meaningful clusters. Specifically, we adopt the labels to exclude same-class pairs from all negative pairs:

The objective of the contrastive loss can be written as:

Similar to [Wang et al., 2022], we also use the data augmentation strategy to refine the pseudo-label learning process. Specifically, we combine the original images with its N augmented images into a batch and send it to the student network. Large batch size is conducive to the update of the batch normalization layer and the convergence of the network, especially for the student model in an online update fashion. We use the average predictions for all augmented samples as the pseudo-labels to supervise the learning process of the student model. After the update of student model (θt → θt+1). The teacher model is updated by exponential moving average [Polyak and Juditsky, 1992; Tarvainen and Valpola, 2017] using the weights of the student model:

where α represents the smoothing factor.

3.4 Soft Weight Alignment

With the help of the pseudo-labels, the pre-trained model can quickly adapt to the target domain and make a more accurate prediction of the target data. However, due to the difference in the data distribution of the source and target domains, there are noisy pseudo-labels in the adaptation process, which leads to instantaneous negative learning. Furthermore, as the model continually adapts to the target domain with changing distributions, the instantaneous errors accumulate, and the knowledge from the pre-trained model is constantly forgotten. For example, the model may not recover after encountering some hard samples, which is detrimental to the test of subsequent data, even if the latter data is not severely shifted. To slow down the accumulation of errors and combat knowledge forgetting, we propose a Soft Weight Alignment (SWA) strategy, which continuously distills knowledge from the source pre-trained model.

Given a pre-trained model Fθ and its two derivative models: anchor model Fθa and student model Fθt , and both Fθt and Fθa are initialized by Fθ. In the continual test-time domain adaptation process, the parameters of the anchor model are fixed, and the parameters of the student network are constantly updated. In fact, the anchor model Fθa is the pretrained model Fθ0 at time t = 0. To maintain the knowledge gained from the pre-training model, we soft-align the parameters of the student network Fθt and the anchor network Fθa :

where θla and θlt are parameters of anchor model Fθa and student model Fθt at layer l, respectively. BN represents the Batch Normalization (BN) layer. SWA forces the parameters of the student model to be soft consistent with those of the anchor model, thereby enhancing the anti-forgetting ability of the student model. Specifically, the indicator function l /∈ BN equals 1 if layer l of the anchor or student network is not the BN layer, and 0 otherwise.

Therefore, we use SWA to constrain the update of the student network. The total loss function LT can be formulated as follows:

where β1 and β2 are the balance hyper-parameters of the total loss function.

4 Experiments

In this section, we review the proposed method on several benchmark tasks: CIFAR10-to-CIFAR10C (Standard and Gradual), CIFAR100-to-CIFAR100C, and ImageNetto-ImageNet-C. We first introduce several commonly used datasets and our method's implementation details; then, we present several commonly used comparison methods; finally, we report and analyze the results to validate the effectiveness of our approach.

4.1 Datasets

We use CIFAR10, CIFAR100, and ImageNet as the source domain datasets, and CIFAR10C, CIFAR100C, and ImageNet-C as the corresponding target domain datasets, respectively. The target domain datasets were originally created to evaluate the robustness of classification networks [Hendrycks and Dietterich, 2019]. Each target domain dataset contains 15 types of corruption with 5 levels of severity. Following [Wang et al., 2022], for each corruption, we use 10000 images for both CIFAR10C and CIFAR100C datasets and use 5000 images for ImageNet-C.

4.2 Implementation Detail

Following [Wang et al., 2022], the corrupted images are provided to the network online, which means these images can be used to update the model only once in the adaptation process. In addition, different from traditional test-time adaptation methods, which adapt to each corruption type data individually, we adjust the source model to each corruption type sequentially. We evaluate the adaptation performance immediately after encountering each corruption type data. The total type of corruption is set as 15, and the corruption level is set to the highest level of 5 (except for the gradual experiments on CIFAR10-to-CIFAR10C). For CIFAR10-to-CIFAR10C, we use a pre-trained WideResNet28 [Zagoruyko and Komodakis, 2016] model from the RobustBench benchmark[Croce et al., 2020]. We use Adam to optimize the network and set the learning rate to 1e-3. The data augmentation strategy is the same as [Wang et al., 2022], including color jitter, gaussian blur, gaussian noise, random affine, and random horizontal flip, N = 8. CIFAR100-toCIFAR100C, we use a pre-trained ResNeXt-29 [Xie et al., 2017] from [Hendrycks et al., 2019], N = 4. For ImageNetto-ImageNet-C, we use the standard pre-trained ResNet-50 from RobustBench [Croce et al., 2020], N = 4. The experiments on ImageNet-to-ImageNet-C are performed under ten diverse corruption orders. The smoothing factor α is set as 0.99.

4.3 Baselines

We compare our method with several state-of-the-art continual test-time adaptation algorithms, the details of these methods are as follows: 1) Source: It directly uses the pre-trained model for adaptation without any specific method for domain adaptation; 2) BN Stats Adapt: Batch Normalization Statistics Adaptation method keeps the pre-trained model weights and uses the Batch Normalization statistics from the input data of the input batch for the prediction [Li et al., 2016; Schneider et al., 2020]; 3) Pseudo-Label [Lee and others, 2013]: This method picks up the class which has the maximum predicted probability as the pseudo-labels to update the model; 4) TENT [Wang et al., 2020]: Test entropy minimization, a test-time entropy minimization scheme to reduce generalization error by reducing the entropy of model predictions on test data; 5) TENT-continual is a continual learning version of TENT; 6) CoTTA [Wang et al., 2022]: Continual Test-Time Adaptation, which reduces the error accumulation by using weight-averaged and augmentation-averaged predictions and avoids catastrophic forgetting by stochastically restoring a small part of the source pre-trained weights.

4.4 CIFAR10-to-CIFAR10C

Performance Evaluation. Table 1 shows the classification error rate for the standard CIFAR10-to-CIFAR10 task. Source method shows the highest average error rate. It depends mainly on the distance between the current corruptiontype data distribution and the source domain distribution. BN Stats Adapt method is simple and effective, dramatically reducing the evaluation error compared with the Source method. Compared with BN Stats Adapt and Pseudo-Label, the performance of TENT-continual is not improved or even decreased. It is mentioned that TENT-continual outperforms the BN Stats Adapt in earlier stages of the adaptation. However, after adapting to several types of corruption, the performance of TENT-continual decreases rapidly. This shows that the TENT-continual can not deal with the error accumulation or forgetting problem in continuous adaptation, and may be unstable under long-term continuous adaptation. CoTTA takes the error accumulation into account to further improve the performance. Our method achieves the best results in the average error value and most of the corruption-type data.

Compared with the strongest baseline CoTTA, the average error value of our method is reduced from 16.5% to 15.2%.

Ablation Studies. In addition, we also conducted ablation experiments to prove the effectiveness of each module of our method. The results are shown in Table 2. All modules we propose are helpful for performance gains.

Following [Wang et al., 2022], baseline is a combination of BN Stats Adapt and Pseudo-Label. Then we add the 'AugTeach' strategy to the baseline model, and the average error value dropped from 19.0% to 18.4%. Further, the addition of the 'STA' strategy reduced the model's average error rate from 18.4% to 16.5%, and 'SWC' loss further reduces the average error to 15.5%. It is worth mentioning that after adding FFT augmentations, the test model achieved better results. Next, the addition of 'SWA' has led to an increase in overall performance and a significant reduction in the error of subsequent adaptations. Inevitably, adding this additional constraint will slow down the adaptation, for example, the addition of 'SWA' makes the performance of the first three corrupted data worse than not adding. Finally, We randomly selected a batch of data in the middle and late stages, and the visualization results are shown in Fig. 3. The results demonstrate that the proposed label selection strategy is effective for instantaneous parameter learning.

Parameters Analysis. We also investigate the parameter sensitivity in the proposed method. Fig. 2 represents the change of accuracies with different loss weights, which indicates that our method is insensitive to the parameters λ1 and λ2 in the range of [0.1,2]. Gradually Changing Setup. Following [Wang et al., 2022], we also consider a gradually changing setup. For the standard setup, corruption types change abruptly in the highest severity. For the gradually changing setup, the corruption types change is gradual.

Specifically, the change process can be expressed as follows:

. . .2-→1

︸ ︷︷ ︸

change

----→

t-1 and before

︸

︷︷

︸

type 1-→2-→3-→4-→5-→4-→3-→2-→1

corruption type t, gradually changing severity

︸ ︷︷ ︸

type 1-→2. . .

t+1 and after

• , where the number represents the corruption severity. The corruption type changes when the severity level is the lowest. In addition, the severity level changes within each type are also gradual.

4.5 CIFAR100-to-CIFAR100C

As shown in Table 4, TENT-continual shows the highest average error rate, and its performance is even worse than the Source model of doing nothing. Pseudo-Label also offers a similar performance degradation curve. In the early stage of adaptation, the performance of Pseudo-Label is better than BN stats adapt and Source, but lower than these two methods in the later stage. This phenomenon is also caused by error accumulation. CoTTA considers the problem of error accumulation and reduces the error to 32.5%. Further, we propose a self-adaptive threshold loss to mitigate the instan-

change

----→

taneous impact of noisy pseudo-labels on optimization objective, and offer a SWA strategy to alleviate the long-term knowledge forgetting problem in continuous adaptation. The performance of our method is better than CoTTA on all corruption types of data, and the average error value is reduced to 30.0%.

4.6 ImageNet-to-ImageNet-C

We also make experiments on ImageNet dataset. Following [Wang et al., 2022], we conduct ImageNet-to-ImageNet-C experiments over ten diverse corruption type sequences in severity level 5. The average result of ten experiments is shown in Table 5. ImageNet is more complex than CIFAR100 and CIFAR-10, and the overall average test error is also greater. Our method outperforms other competing methods and reduces the average test error to 62.1%.

5 Conclusion

In this paper, we propose a simple yet effective framework for continual test-time domain adaptation, which refines the pseudo-label learning process from the perspective of the instantaneous and long-term impact of noisy pseudo-labels. Firstly, we propose a self-adaptive thresholding Cross-Entropy loss to optimize the adaptation process, which facilitates learning the adapted model. Secondly, the learned thresholds and pseudo-labels are utilized to select more discriminative positive and precise negative pairs for contrastive learning. Finally, we propose a Soft Weight Alignment strategy to normalize the distance between the parameters of the adapted model and the source pre-trained model, which improves the classification accuracy in the late stage of adaptation. Extensive experimental results demonstrate that our method achieves state-of-the-art performance on various benchmark datasets.

Acknowledgements

Our work was supported by Joint Fund of Ministry of Education of China (8091B022149), Key Research and Development Program of Shaanxi (2021ZDLGY01-03), National Natural Science Foundation of China (62132016, 62171343, 62071361 and 62201436), and Fundamental Research Funds for the Central Universities (ZDRC2102 and ZYTS23135).

Contribution Statements

Xu Yang and Yanan Gu contribute equally. They jointly designed the experiments and wrote the manuscript. Kun Wei and Cheng Deng contributed to the writing of the manuscript. Cheng Deng supervise the project.