Continuous Test-Time Adaptation via Dual Alignment

1 Introduction

Deep neural networks achieve strong performance under matched training and test distributions, however, their robustness degrades under distribution shifts [Zhou et al., 2023]. Test-time adaptation (TTA) mitigates this issue by updating models online with unlabeled test data [Wang et al., 2021], but suffers from error accumulation in continually evolving domains. Continuous test-time adaptation (CTTA) extends TTA to streaming scenarios by enabling continual adaptation across evolving target distributions [Wang et al., 2022].

Existing CTTA methods can be broadly categorized into two paradigms: entropy minimization and teacher-student schemes. Entropy minimization updates either Batch Normalization statistics [Niu et al., 2022] or a small subset of model parameters [Liu et al., 2024b], yielding immediate performance gains with minimal computational cost. However, under continual distribution shifts, this strategy is highly sensitive to miscalibrated predictions, which leads to uncontrolled error accumulation over time. In contrast, teacher-student frameworks generate pseudo labels for target samples and can provide more stable supervision. Nevertheless, conventional mean-teacher approaches [Tarvainen and Valpola, 2017] become increasingly noisy in dynamic environments, amplifying distributional drift. Although recent works [D¨obler et al., 2023; Park et al., 2025] employ testtime augmentation to improve pseudo-label quality, the required multiple forward passes incur substantial latency and computational overhead, limiting their practicality in CTTA.

∗Corresponding author.

As a result, existing methods face an inherent tradeoff between computational efficiency and adaptation stability [Wang et al., 2022; Liu et al., 2024a]. This trade-off arises because current approaches lack reliable and adaptive mechanisms to model continuously evolving target distributions in online streaming settings, where non-stationary

and pseudo-label supervision. Consequently, achieving stable and efficient adaptation under continuously shifting target distributions remains a fundamental challenge in CTTA.

Motivated by the aforementioned issue, we propose Dual Alignment (D-Align), a stable and efficient CTTA method that synergistically combines correlation alignment and masked consistency alignment. Specifically, Correlation Alignment (CorA) performs feature distribution matching by constructing a pseudo-source domain. However, in the CTTA setting, where target data arrive sequentially and exhibit nonstationary shifts, constructing a reliable pseudo-source domain is non-trivial. To tackle this, we first introduce a Feature Resilience (FR) metric to select reliable samples that possess globally consistent features. Building upon this, we design a continual pseudo-source memory bank, which maintains the diversity and reliability of the pseudo-source distribution through a progressive dynamic sample update strategy. Furthermore, to compensate for the limitations of linear alignment in correlation alignment, we incorporate imagelevel visual domain prompting techniques to capture complex nonlinear relationships among features. Moreover, we introduce masked consistency alignment, where each target sample is first transformed by a domain-specific visual prompt and then subjected to different masking operations. Based on these prompted and masked views, entropy consistency and entropy ranking losses enforce prediction consistency, stabilizing the learning of target domain-specific knowledge while preserving discriminative power under partial feature occlusion. Extensive experiments on multiple standard benchmarks demonstrate that D-Align consistently achieves stateof-the-art performance under continuously evolving target distributions. In summary, our contributions can be summarized as:

• We make the first attempt to introduce correlation alignment into CTTA and propose CorA, which achieves feature alignment by applying a linear transformation to the output features without updating model parameters, thereby significantly reducing computational overhead.
• Subsequently, we propose a D-Align method that synergistically optimizes correlation and masked consistency alignment. By integrating a continual pseudo-source memory bank for feature rectification and masked views for structural learning, our approach ensures stable adaptation across continually shifting distributions.
• Extensive experiments on four benchmarks demonstrate that D-Align outperforms state-of-the-art methods, achieving error rates of 21.8% on CIFAR100-C and 37.9% on ImageNet-C.

2 RelatedWork

Continual Test-Time Adaptation refers to scenarios where the target domain is non-static, highlighting the need to adapt to continuous domain shifts and thereby extending the practicality of TTA. This paradigm promotes a rethinking of conventional protocols by emphasizing a stronger focus on stability. CoTTA was among the first to formalize the more challenging Continual Test-Time Adaptation setting and employs a mean teacher model to generate more stable pseudo-labels and introduces a stochastic weight restoration technique to mitigate catastrophic forgetting during long-term adaptation [Wang et al., 2022]. NOTE proposes Instance-Aware Batch Normalization (IABN) to mitigate distribution shifts caused by temporally correlated and non-i.i.d. data streams [Gong et al., 2022]. RoTTA further advances the paradigm by introducing Practical TTA (PTTA), simulating real-world dynamics and employing Robust Batch Normalization (RBN) with a memory bank [Yuan et al., 2023]. In a different approach. ViDA designs a dualbranch adapter that explicitly separates domain-invariant and domain-specific features, and dynamically weights domain knowledge based on uncertainty, thereby effectively balancing the learning of new knowledge with the preservation of source domain knowledge during continual adaptation [Liu et al., 2024b]. Continual-MAE enhances domain-invariant representations using a distribution-aware masked autoencoder and leverages Monte Carlo dropout to filter valid target regions based on uncertainty [Liu et al., 2024a]. REM proposes Ranked Entropy Minimization to mitigate model collapse, stabilizing adaptation by explicitly enforcing the rank order of prediction entropy across progressively masked samples [Han et al., 2025]. Recently, PAID employs Householder reflections to update model weights while strictly preserving pairwise angular structures to maintain semantic consistency [Wang et al., 2025].

3 Method

3.1 ProblemFormulation and Overview

Problem Formulation. Let Ds = {(xi, yi)}ni=1 be the source domain consisting of n labeled samples, where each pair (xi, yi) corresponds to an input sample and its groundtruth label. A model f(·; θs) is first trained on Ds, with θs denoting the source model parameters. During deployment, the model encounters an unlabeled target data stream organized as a sequence of target domains {Dτ t }mτ=1 , where Dτ t = {xτt,i }nτ i=1 contains nτ samples drawn from the target distribution P τ t , and m is the total number of domains (or time steps). The goal of continuous test-time adaptation is to update the source-trained model f(·; θs) to handle the continuously evolving target data {Dτ t }mτ=1 in an online manner, without accessing the original source data Ds.

Overall Framework. We propose a dual-alignment method (D-Align) for continuous test-time adaptation, integrating correlation alignment with masked consistency alignment to achieve stable online adaptation. The correlation alignment branch aligns target features with a reliable pseudo-source formed from resilient samples in a class-wise memory bank, using prompt-driven nonlinear alignment to capture complex dependencies. Masked consistency alignment enforces prediction agreement between original and masked inputs, updating only normalization layers and prompts for efficient target adaptation. The overall framework of D-Align is illustrated in Fig. 1.

3.2 Correlation Alignment

The correlation alignment branch selects resilient pseudosource samples, maintains them in a continual memory bank, and applies image-level prompt-driven nonlinear alignment to capture complex feature relationships.

Feature Resilience Criterion for Robust Pseudo-Source Sample Selection. In CTTA, selecting highly reliable samples from the data stream is essential for constructing a stable pseudo-source distribution. Here, we propose a new metric, Feature Resilience (FR). The key idea of FR is that a truly reliable sample should not only yield confident predictions but also maintain robustness under partial feature occlusion. By quantifying the stability of model predictions when salient features are masked, FR provides a more comprehensive measure of sample reliability, enabling more effective selection of highly dependable samples. Specifically, let the original image be divided into patches x = (p1, p2, . . . , pNp ), where Np is the total number of patches, and the most salient patches are subsequently masked according to the selfattention scores of a ViT [Dosovitskiy et al., 2021]:

where A ∈ RNp contains the score for each of the image patches, H is the number of heads in the multi-head attention, Qh,cls and Kh,img are query for class token and key for image token of the h-th attention head, respectively, d denotes the dimension of each attention head. This step can identify the image regions that the model focuses on most when making predictions. Subsequently, we implement a progressive feature masking strategy guided by the score vector A. The mask index set Im is dynamically constructed as follows:

where rm denotes the masking ratio, π denotes a permutation of the image patch index set {1, . . . , Np}, which arranges the elements in descending order according to the values of A, and ⌊·⌋ denotes the floor operation. This design ensures the prioritized masking of the image patches that the model attends to most, thereby systematically evaluating the model's robustness when critical discriminative features are absent. Next, we construct a binary masking vector M ∈ {0, 1}Np , where the i-th element Mi is defined as:

where 0 indicates that the image patch is removed and 1 indicates it is retained. The masked image xm constructed using the mask M is given by:

we employed a multi-ratio masking strategy to obtain a more comprehensive stability assessment. A set of masked images {xm1 , xm2 , . . . , xmN } is obtained using different masking ratios. In the implementation, we construct masked images {xm1 , xm2 } using only two masking ratios, m1 = 0.1

and m2 = 0.2. Based on the constructed masked images {xm1 , xm2 }, FR is calculated as follows:

where ximj is the i-th sample of xmj . K is the number of masking ratios. Ent(x) = - ∑Cc=1 ˆ pc(x) log ˆ pc(x) denotes the entropy of the model's predictive distribution ˆ p(x) over C classes. ¯Ei denotes the average entropy of the i-th sample under different masking ratios. A lower FR indicates that the model's uncertainty remains relatively stable even when certain key features are occluded, suggesting that the model has captured global and holistic representations rather than relying on localized features.

Continual Pseudo-Source Memory Bank. In the CTTA task, data arrives sequentially in an online manner as a stream of batches, requiring the model to generate predictions immediately for each batch without the opportunity to revisit previous samples. This inherent constraint precludes global access to the complete dataset for constructing a stable pseudosource distribution, a fundamental prerequisite for the effective feature alignment demonstrated in TCA [You et al., 2025]. While it is possible to construct local pseudo-source correlations within individual batches, such an approach is prone to unstable feature alignment and potential performance degradation, primarily due to statistical biases and limited sample diversity inherent in small batch sizes. To address this issue, we propose a Continually Pseudo-Source Memory Bank SC, which accumulates and retrieves reliable pseudosource correlations across batches through a dynamic update mechanism, thereby facilitating robust feature alignment on streaming data.

where the output of the model ft(xit ) corresponds to class c. SC = {S0, S1, ..., Sc, ..., SC-1} denotes the partition of SC into C mutually exclusive subsets. Each subset Sc independently stores and updates the pseudo-source samples of class c, maintaining class balance and diversity within the memory bank, thereby mitigating potential class bias during feature correlation alignment. Given the i-th sample, we first use the current model ft(·) to obtain its pseudo label. If the predicted label corresponds to class c, the embedding of this sample is denoted as zc i = ψ(xit ), and the associated prediction score is ϕci . (zcworst , ϕcworst ) denotes the sample with the highest score (i.e., lowest reliability) in Sc. V denotes the capacity of Sc. It is worth noting that the number of samples in SC is kept very small. For instance, in the CIFAR-10-C task, with V = 10 per class, the memory bank holds only 100 samples in total, accounting for merely 1% of the dataset, which ensures low computational and storage overhead during CTTA.

At the initial stage, when a subset Sc ⊂ SC is not yet full, incoming samples are directly admitted until the capacity is reached, and correlation alignment is not applied. Once Sc becomes full, the update strategy switches to a replacement mechanism: each new sample (zc i , ϕci ) is compared with the stored sample (zcworst , scworst ) that has the highest score, and replacement occurs only if the new sample yields a lower score. By iteratively replacing suboptimal samples with higher-quality ones, the memory bank Sc progressively concentrates on reliable target-domain representations, allowing its feature distribution to converge toward the highconfidence region of class c. Correlation alignment is then performed once SC satisfies the following condition:

where I denotes the indicator function, which equals 1 if the condition is true and 0 otherwise, and τ represents the threshold. Once SC is ready, its covariance matrix ˆΣs is computed as the pseudo-source correlation. Correlation alignment is then achieved through a linear transformation matrix W, with the optimization objective defined as follows:

where Σt denotes the covariance matrix of target-domain samples in the current batch, and ∥ · ∥F represents the Frobenius norm.

Unfortunately, linear transformations exhibit inherent limitations in capturing the complex nonlinear relationships underlying real-world distribution shifts [You et al., 2025]. To alleviate this, we propose a novel approach that achieves nonlinear alignment through image-level visual domain prompting, as detailed below.

Prompt-driven Nonlinear Alignment. The key insight is to learn differentiable perturbations directly in the input space, exploiting the powerful intrinsic nonlinear mapping capacity of the pre-trained backbone network to convert simple input-level signals into complex and expressive nonlinear transformations within the feature space. This enables effective nonlinear alignment without additional heavy training overhead. The parameters of these perturbations are optimized according to Eq. (9), as illustrated below:

where AC ∈ Rc×L×r , BC ∈ Rc×r×L are two low-rank matrices following the LoR-VP design proposed in [Jin et al., 2025]. We adopt this formulation to further reduce the number of parameters in visual domain prompting and lower computational overhead. Here, c denotes the number of image channels, L represents the image size, and r is a hyperparameter satisfying r << L. Our visual domain prompting method learns perturbations in the input space and leverages the nonlinear mapping capability of the pre-trained backbone network to transform simple input perturbations into complex and highly nonlinear feature-space transformations. This effectively compensates for the limited representational capacity of linear transformations when handling complex nonlinear relationships, while introducing a small amount of additional computational overhead.

Building on the prompt-driven nonlinear alignment, we can learn a transformation W that effectively performs nonlinear alignment in the feature space. However, directly using the transformed features as the final representations may lead to the loss of target domain-specific information. To strike an appropriate balance between source-domain knowledge and target-domain adaptiveness, we introduce a homeostatic feature fusion strategy, which enhances adaptation robustness by integrating the transformed features with their original counterparts, as described below:

where Z ′ t denotes the final set of output feature vectors, while ˆ Zs = {zi|zi ∈ SC} and Zt = {zi|zi ∈ B} represent the sets of feature vectors from SC and from a batch B, respectively. µt and ˆµs denote the mean embeddings of Zt and ˆ Zs, respectively. α is a hyperparameter. Finally, the prediction results after performing correlation alignment through the Continual Pseudo-Source Memory Bank are given by P ′ t = g(Z ′ t ), where g(·) denotes the classification head.

3.3 Masked Consistency Alignment

While correlation alignment aligns second-order statistics to correct distribution shifts, it does not explicitly encode target domain-specific information. We therefore incorporate domain prompt learning, in which a learnable visual prompt is applied to each target image xt prior to alignment. By parameterizing the prompt in a lightweight low-rank form, the model captures structured domain variations through conditional feature modulation, enabling more effective targetdomain adaptation. Formally, the prompt is computed as:

where AM ∈ Rc×L×r and BM ∈ Rc×r×L are low-rank matrices defined analogously to those in Eq. (10). This learnable prompt functions as a compact memory module that captures and injects target-domain patterns into the model.

Building on the prompted image xtMConA, we introduce a masked consistency alignment mechanism to further enhance model adaptability. While the correlation alignment branch efficiently corrects feature statistics using a pseudosource anchor, it lacks the ability to incorporate targetspecific knowledge. To address this limitation, we adapt the normalization layers to the target-domain distribution through entropy minimization. Inspired by [Han et al., 2025], prediction consistency is enforced between the prompted image and its masked variants. Given xtMConA and its corresponding masked samples {xm1 , xm2 }, the alignment mechanism is built upon two complementary components: the entropy consistency loss (ECL) and the entropy ranking loss (ERL), defined as follows:

where H(p, q) denotes the cross-entropy between two probability distributions p and q, f(x) represents the output of the model, sg(·) is the stop-gradient operation, and MN = {0,m1,m2} denotes the set of mask ratios. ECL aligns the predictions of masked and original images, thereby implicitly reducing prediction entropy. ERL enforces that predictions on images with higher masking ratios must exhibit greater entropy than those with lower masking ratios. Together, ECL and ERL mitigate abrupt entropy collapse and alleviate overconfident predictions caused by unconstrained entropy minimization, while jointly optimizing the visual prompt to encode reliable target-domain knowledge. Through optimization via consistency alignment, D-Align effectively distills and injects target-domain knowledge into the model, thereby enabling adaptive performance under continuously evolving target distributions.

3.4 Optimization Objective

In D-Align, correlation alignment and masked consistency alignment are integrated in a sequential manner to form a unified adaptation framework. This phased design is motivated by the complementary properties of the two mechanisms: masked consistency alignment enables rapid adaptation to the target distribution by updating normalization parameters, while correlation alignment preserves model stability through feature distribution matching. Specifically, the optimization proceeds in two stages. In the first stage, the normalization parameters are updated via masked consistency alignment, with the objective

which enforces prediction consistency between masked and original samples. In the second stage, correlation alignment is applied to the output embeddings by minimizing

Together, this two-stage strategy achieves efficient online adaptation by balancing rapid target-domain adjustment with source-domain knowledge preservation, thereby improving robustness under continual distribution shifts.

4 Experiments

4.1 Experimental Setups

Datasets and Implementations. We evaluate our method on three standard classification CTTA benchmarks, including CIFAR10-to-CIFAR10C, CIFAR100-to-CIFAR100C, and ImageNet-to-ImageNet-C. We further test CLIP on VisDAC, PACS, VLCS, and OfficeHome. Implementation details are in the supplementary material.

ComparisonMethods. For the CTTA setting, we compare our proposed D-Align against ten competitive methods: Tent [Wang et al., 2021], CoTTA [Wang et al., 2022], VDP [Gan et al., 2023], SAR [Niu et al., 2023], PETAL [Brahma and Rai, 2023], ViDA [Liu et al., 2024b], Continual-MAE [Liu et al., 2024a], DPCore [Zhang et al., 2025], REM [Han et al., 2025], and PAID [Wang et al., 2025]. For the TTA setting using CLIP, we evaluate D-Align against four representative methods: Tent [Wang et al., 2021], TPT [Shu et al., 2022], CLIPArTT [Hakim et al., 2025], and WATT (including both WATT-P and WATT-S variants) [Osowiechi et al., 2024].

4.2 Performance of CTTA under Standard Corruption

ImageNet-to-ImageNet-C. Using a source model pretrained on ImageNet, we conduct CTTA by sequentially evaluating it on the 15 corruption types in ImageNet-C. As reported in Tab. 1, D-Align achieves state-of-the-art performance, obtaining an average error rate of 37.9%. This corresponds to a 17.9% improvement over the source model and a 1.3% gain over the previous best method, REM. To further assess the upper bound of adaptation capability, we compare against an ideal supervised oracle trained with target labels. The performance gap is only 2.2%, highlighting that our unsupervised framework is highly effective at leveraging unlabeled target data and achieves performance close to fully supervised adaptation. Furthermore, to evaluate our method on a benchmark that better reflects real-world physical properties, we conducted experiments on ImageNet-3DCC. The detailed results are provided in the supplementary material.

CIFAR10-to-CIFAR10C & CIFAR100-to-CIFAR100C. We further evaluate our method on two classic CIFAR10to-CIFAR10C and CIFAR100-to-CIFAR100C scenarios. As shown in Tab. 2, D-Align achieves an average error rate of 8.6%, corresponding to a 19.6% improvement over the source model and a 0.8% gain over the previous best method, REM. Detailed fine-grained results are provided in the supplementary material. On the more challenging CIFAR100to-CIFAR100-C benchmark, D-Align achieves a mean error of 21.8%. This represents a 13.6% reduction compared to the source model's 35.4% error. Moreover, our method achieves the best performance when compared against strong CTTA baselines, including Continual-MAE, PAID, and REM. Detailed results are provided in the supplementary material.

4.3 Performance of CLIP on Cross-Domain Benchmarks

Natural Domain Shift. As shown in Tab. 3, D-Align improves the average accuracy on CIFAR-10, CIFAR-10.1, and CIFAR-100 by 4.76% over the original model. Compared with the prior state-of-the-art method WATT, our approach achieves competitive results, demonstrating that D-Align is effective not only on ViT-Base architectures but also on vision-language models such as CLIP, thereby broadening its applicability.

Simulated and Video Shifts. To assess performance under large domain shifts from simulated environments to realworld video streams, we evaluate on the VisDA-C benchmark. As shown in Tab. 3, D-Align achieves notable gains. On the highly divergent synthetic 3D domain, it attains an accuracy of 87.19%, surpassing WATT-S (85.36%) by 1.83%. On the realistic YT (YouTube) video domain, D-Align maintains superior performance with a top accuracy of 84.87%. Overall, it achieves an average accuracy of 86.03%, outperforming WATT-S by 1.0%, highlighting robustness under complex synthetic-to-real domain shifts.

Texture and Style Shifts. We further evaluate adaptability under texture, artistic-style, and photographic shifts on Office-Home, PACS, and VLCS. On Office-Home, DAlign achieves 79.71% average accuracy, surpassing WATTP (78.88%) with a notable gain on Art (78.90%). On PACS, it obtains 94.76%, comparable to WATT-P (94.81%). On VLCS, D-Align reaches 81.47%, improving over WATTS (81.14%) by 0.33%. These results show that D-Align achieves leading or comparable performance across domain shift benchmarks, validating its generalization and stable CLIP adaptation.

4.4 Analysis Experiment

Efficiency Analysis. To evaluate the computational efficiency of our method, we compare it with Tent, CoTTA, and recent advanced approaches such as ViDA and ContinualMAE, as shown in Tab. 4. First, regarding the CorAmodule, it completes the full CTTA pipeline in 32m27s, significantly outperforming CoTTA (241m58s) while maintaining comparable accuracy. In terms of memory usage, the additional overhead introduced by CorA is almost negligible (approximately 270 MB). This efficiency comes from our lightweight Continual Pseudo-Source Memory Bank, which stores only a very small subset of samples, and from the fact that feature alignment is performed solely through a linear transformation, without requiring costly backpropagation. These results demonstrate that CorA offers high computational efficiency. Secondly, D-Align offers clear advantages in terms of training time, the number of trainable parameters, and the amount of model state that must be stored. Unlike conventional approaches that rely on costly multiple forward passes for uncertainty estimation, D-Align strikes a better balance between speed and accuracy. It outperforms Continual-MAE by a margin of 31.75%, while requiring only 30% of the inference time and 0.7% of the trainable parameters. Overall, D-Align effectively avoids the burdens of full-model adaptation, offering a practical solution that balances high accuracy with minimal computational cost.

Ablation Study. To validate the effectiveness of our proposed correlation alignment, we conduct ablation experiments by freezing all model parameters and performing evaluations under the online CTTA setting without backpropagation on CIFAR100-C. As reported in Tab. 5, utilizing CorA alone yields a 1.3% performance improvement over the source model on CIFAR100-C. Notably, it surpasses CoTTA [Wang et al., 2022] by 0.7%, thereby validating the efficacy of our proposed CorA module. Additionally, we provide results on CIFAR10-C, ImageNet-C, and ImageNet3DCC in the supplementary material. Subsequently, we conducted a more detailed ablation study on the ImageNet-C dataset under the CTTA setting, as shown in Table 6. Specifically, we remove PCorA (Ex1) to evaluate the contribution of the Prompt-driven Nonlinear Alignment strategy within CorA. Compared with D-Align (Ex0), Ex1 exhibits a 0.3% performance drop, indicating that PCorA effectively compensates for the limitations of purely linear alignment. Besides, we examine the effectiveness of PMConA in Ex2. The results show that removing PMConA incurs a 0.4% performance drop, verifying its contribution. Subsequently, in Ex3, we ablate the Feature Resilience (FR) criterion. This leads to a 0.7% decrease in accuracy, demonstrating that FR enhances the reliability of the pseudo-source memory and thereby facilitates better correlation alignment. Finally, disabling CorA in Ex4 results in a substantial 1.2% performance degradation, further confirming the effectiveness of CorA.

5 Conclusion

In this work, we propose D-Align, a continual test-time adaptation framework that combines correlation alignment and consistency-based entropy minimization for efficient and stable adaptation. D-Align integrates feature resilience, a continual pseudo-source memory bank, and image-level visual prompts to construct robust pseudo-source correlations, and learns target-domain knowledge through stabilized consistency alignment. Extensive experiments on standard benchmarks demonstrate the effectiveness of D-Align under dynamic domain shifts.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grants 62306008, 32501770 and 32472007, the Key Project of Anhui Province's Science and Technology Innovation Tackle Plan under Grant 202423k09020040, and the Natural Science Youth Project of the Education Department of Anhui Province under Grant 2025AHGXZK40156.