Continual Unsupervised Domain Adaptation for Cardiac Image Segmentation with Style-Adapting Generative Replay and Prototype Consolidation

1 Introduction

Cardiac Magnetic Resonance (CMR) imaging plays a pivotal role in diagnosing myocardial pathology and guiding therapeutic interventions [Quarta et al., 2018; Savarese et al., 2022]. While Deep Learning (DL) has revolutionized automated segmentation [Litjens et al., 2019], its clinical deployment is hindered by domain shift, where models trained on specific centers fail to generalize across heterogeneous domains. In real-world clinical practice, this challenge is compounded by two stringent constraints: (1) clinical data arrives sequentially from different centers, and (2) strict privacy regulations prohibit storing or revisiting raw data [Li et al., 2023]. As illustrated in Figure 1, although existing continual unsupervised domain adaptation (UDA) methods have shown promise [Li and Fan, 2023; Chen et al., 2025], two key challenges remain for CMR segmentation.

∗Corresponding author.

First, catastrophic forgetting is exacerbated by interdomain style variations under privacy constraints. Classical continual learning strategies rely on storing and replaying past samples, which is often infeasible under medical settings [Chen et al., 2023]. Generative replay (GR) circumvents this by synthesizing pseudo-historical data. Early approaches based on GANs suffer from training instability and difficulties in preserving fine structures [Wiatrak et al., 2019; Armanious et al., 2020; Saxena and Cao, 2021]. Recent diffusion-based replay methods improve fidelity [Gao and Liu, 2023], but typically generate images around the average domain, leading to style averaging and insufficient preservation of domain-specific appearance.

Second, feature space drift is driven by noisy pseudolabels. In UDA, supervision relies on pseudo-labels, which are inherently noisy at boundaries. Naive self-training leads to error accumulation, where the model reinforces its own mistakes, causing the feature space to drift away from valid semantics [Ye et al., 2024]. While contrastive learning approaches [Huang et al., 2021; Gu et al., 2024] attempt to structure the feature space, they often rely on centroids calculated solely from the current noisy predictions. Consequently, when the model drifts, the anchors shift accordingly and fail to serve as stable references for regularization.

In this study, we propose a novel continual UDA framework for multi-domain CMR segmentation via dual-level alignment. As shown in Figure 2, our key insight is that robust adaptation requires consistency in both the input space and the feature space. First, to mitigate catastrophic forgetting, we propose the Domain-Style Adapting Generative Replay (DSA-GR) module. Unlike most existing methods, DSA-GR serves as a buffer to synthesize high-fidelity and style-aligned historical images without storing raw data. It leverages a Denoising Diffusion Implicit Model (DDIM) with trajectory-consistent distillation to capture the precise structural distribution of past domains. Crucially, we employ a semantic-preserving style adaptation mechanism, ensuring that generated samples align with the target domain's feature while preserving historical anatomical integrity. Second, to combat drift at the feature level, we introduce the Cross-domain Prototype-guided Feature Consolidation (CPFC) mechanism. CPFC constructs a stable semantic manifold by maintaining a prototype memory bank. Innovatively, these prototypes are evolved via a cross-domain update strategy, aggregating features from the current, replay, and reference streams. These robust prototypes serve as anchors for prototype-guided contrastive learning, which rectifies the feature space by pulling embeddings into compact clusters regardless of domain variations. Therefore, this allows the model to achieve a balance between stability and plasticity during continual UDA. Our main contributions are summarized as follows:

• We propose a novel continual UDA framework that balances stability and plasticity via dual-level alignment, including input-level style-aligned replay and feature-level prototype-anchored consolidation, to alleviate catastrophic forgetting and model drift.
• We develop DSA-GR, which combines trajectoryconsistent diffusion distillation with style adaptation to generate high-fidelity replay images rendered in the current domain style, enabling privacy-preserving replay and mitigating forgetting.
• We introduce CPFC mechanism, which maintains a cross-domain prototype memory bank built from current, replay, and reference streams, and uses prototypeguided feature consolidation to rectify the feature space and reduce pseudo-label drift.

2 RelatedWork

2.1 Unsupervised Domain Adaptation

UDA aims to adapt a model trained on a labeled source domain to an unlabeled target domain. Existing methods are broadly divided into pixel-level and feature-level adaptation.

Pixel-level approaches translate source images to match the target style while preserving anatomy. Early GAN-based methods, such as SIFA [Chen et al., 2019], combine imagelevel adaptation with feature constraints to maintain structural consistency, and CySGAN [Lauenburg et al., 2023] enforces structural fidelity via segmentation losses on translated images. Recently, diffusion models have been exploited for style transfer, e.g., Diffuse-UDA [Gong et al., 2024] and SSA-UDA [Ji and Chung, 2024], which use diffusion processes to align target appearance with source labels. Featurelevel UDA methods instead learn domain-invariant representations, often through pseudo-labeling on the target domain. Many works focus on improving pseudo-label reliability: UAST [Cao et al., 2021] uses uncertainty to select confident pseudo-labels, CoUDA [Zhang et al., 2020] performs transferability-aware adaptation to mitigate label noise, and DLaST [Xie et al., 2024] disentangles anatomy and modality factors and enforces shape constraints. In the source-free setting, GAD-PS [Du et al., 2024] augments target samples to construct a pseudo-source domain for alignment.

Despite their effectiveness, these methods are typically designed for one-shot adaptation between a single source and target domain, and do not address continual domain shifts.

2.2 Continual Unsupervised Domain Adaptation

Continual UDA targets the more realistic scenario where a model should adapt to a stream of unlabeled target domains while retaining knowledge from previous domains. Existing approaches can be roughly grouped into regularization-based strategies and generative replay.

Regularization-based methods mainly operate at the feature level. MuHDi [Saporta et al., 2022] combines multi-head knowledge distillation with domain-adversarial learning to alleviate catastrophic forgetting. CoSDA [Feng et al., 2023] adopts a dual-speed teacher-student framework with consistency regularization for source-free adaptation, and online distillation approaches further couple fast-updating students with slow-moving teachers to handle cyclic shifts [Houyon et al., 2023]. More recently, continual adaptation has been reformulated as a likelihood-regularized optimization problem, providing a theoretical view of stability-plasticity tradeoffs [Truong et al., 2024]. Generative replay complements these methods by synthesizing pseudo-historical data. GarDA [Chen et al., 2023] uses a deterministic U-Net to translate source images into the style of the current target domain, while diffusion-based replay has been explored to improve stability and fidelity for continual segmentation [Li et al., 2024; Xu et al., 2025].

Nevertheless, existing continual UDA methods are still limited in handling the complex multi-domain and multi-style variations characteristic of cardiac imaging, often leading to substantial performance degradation in real-world scenarios.

3 Methodology

3.1 ProblemFormulation

Let S = {D1,D2, . . . ,DT} be a sequence of distinct CMR domains. At time step t = 1, the model is trained on a labeled source domain D1 = {(x1, y1)}. For subsequent steps t > 1, the model adapts to an unlabeled target domain Dt = {xt}. Crucially, due to stringent privacy regulations and storage constraints, access to raw data from previous domains D1:t-1 is strictly prohibited. Our objective is to optimize the model θt to effectively adapt to the current domain Dt (Plasticity) while maintaining performance on the historical domains D1:t-1 (Stability). To achieve this balance without accessing previous domains, we propose a novel framework that comprises two synergistic components:

domain-style adapting generative replay (DSA-GR) for input-

level alignment, and cross-domain prototype-guided feature

consolidation (CPFC) for feature-level rectification.

3.2 Domain-Style Adapting Generative Replay

As shown in Figure 3, traditional GR performs poorly in continual UDA because it neither preserves fine-grained details from previous domains nor accounts for style differences with the current domain. To address this issue, as illustrated in Figure 2(a), we propose DSA-GR, which generates structurally consistent replay images rendered in the style of the current domain without accessing original historical data.

Trajectory-Consistent Generative Distillation. We employ a DDIM as the replay generator. Let ϵθ(xk, k) denote the noise prediction network conditioned on diffusion step k. To preserve fine-grained anatomical details from previous domains, we propose trajectory-consistent generative distillation. At step t, we freeze the previous generator ϵθt-1 . We first generate a raw historical sample xr using ϵθt-1 . This sample is then diffused forward to step k to obtain a noisy latent xrk . Instead of merely matching the final output, we train the current generator ϵθt to mimic the denoising trajectory of the teacher model:

This trajectory matching encourages the generative model to retain the precise structural distribution of the anatomical structures across continual adaptation steps.

Semantically Consistent Style Adaptation. While distillation preserves the anatomical distribution of previous domains, the replay samples should also be compatible with the appearance statistics of the current domain Dt. Instead of performing explicit pixel-level style transfer, we regularize replay generation in the feature space by matching style statistics while preserving the feature layout. Specifically, we employ Estyle as a fixed encoder and extract feature maps from the generated replay image (fr) and a real current-domain image (fc). We then construct a style-aligned feature target fg by matching the channel-wise mean µ(·) and standard deviation σ(·):

To avoid excessive deviation from the original replay representation and preserve anatomical structure, we introduce a conservative content regularization term:

The overall generator loss combines the standard diffusion objective, trajectory-consistent distillation, and the above content-preserving style regularization:

By training the generator, we obtain replay images that faithfully preserve the anatomical structures from previous domains while matching the style of the current domain.

3.3 Cross-domain Prototype-guided Feature Consolidation for UDA Segmentation

Although DSA-GR provides style-aligned past domain data, the segmentation model is still supervised by noisy pseudolabels from the target domain, which can lead to model drift in continual UDA. To explicitly counteract pseudo-label drift, as illustrated in Figure 2(b) and Figure 4, we propose the CPFC mechanism. The key idea is to construct a dual-anchored feature space using a robust cross-domain prototype memory bank and prototype-guided feature consolidation.

Cross-domain prototype memory. Let fθt denote the student segmentation network at step t, and fθt-1 the frozen teacher network obtained from step t-1. For the image x, the teacher generates pixel-wise pseudo-labels ˆ y and confidence scores q:

Where zt-1 are the teacher's logits. Simultaneously, to build a comprehensive feature representation, we extract three distinct feature maps:

• Reference Features (F ref ): Pθt-1 (Eθt-1 (xt)). This represents the teacher's view of the current domain.
• Current Student Features (F c ): Pθt (Eθt (xt)), driving

plasticity.

· Replay Student Features (F r ): Pθt (Eθt (xreplay)), en-

suring stability.

We maintain a class prototype memory bank M = {µk ∈ Rd }Kk=1 . Unlike standard methods that update prototypes solely from target data, we propose a cross-domain update strategy to ensure robustness. The prototypes are evolved via Exponential Moving Average (EMA) by aggregating highconfidence pixels from all three streams (F c , F r , F ref ). For a class k, let Sk = {i | ˆ y(i) = k, q(i) > τc} be the set of valid pixels. The update rule is:

Therefore, F ref is integrated to anchor the prototypes to the teacher's feature space to prevent drift, while F r ensures the clusters remain compatible with historical distributions.

Prototype-guided feature consolidation. Given the prototype memory Mt, we design a prototype-based contrastive loss to consolidate the feature space. For each valid pixel feature z with pseudo-label k, we encourage it to be close to its class prototype µk and far from other class prototypes:

Where τ is a temperature parameter. We apply this loss to both current-domain and replay features, using confidence thresholds to downweight uncertain pseudo-labels. This prototype-guided contrastive learning pulls features towards stable cross-domain prototypes, effectively suppressing pseudo-label drift.

To further distill semantic knowledge from the teacher while adapting to Dt, we combine LPCL with a standard selftraining loss LST on pseudo-labels and a Kullback-Leibler divergence loss LKD between teacher and student logits:

In this way, CPFC enforces consistency at both the pixel level (via self-training and knowledge distillation) and the prototype level (via contrastive learning), stabilizing class semantics across domains and mitigating catastrophic forgetting when combined with DSA-GR.

4 Experimental Setup

4.1 Datasets and Implementation Details

The MYOSAIQ 2023 Dataset. The dataset contains 358 LGE-MRI scans from three domains D1-D3 [Qayyum et al., 2024]. D1 contains 98 scans, D2 155 scans, and D3 105 scans. The labels include left ventricle (LV), healthy myocardium (MYO), myocardial infarction (MI) and microvascular obstruction (MVO). To obtain consistent annotation across domains, we merge MI and MVO in D1 and segment four classes: background, LV, healthy MYO and infarct/scar.

The M&MS Dataset. The multi-centre M&MS dataset [Campello et al., 2021] contains 150 short-axis scans with LV, MYO and right ventricle (RV) annotations. We define three domains by scanner vendor: Philips (D1, source), Siemens (D2) and Canon (D3). For both datasets, we utilize the official training and testing splits to ensure fair comparison with existing benchmarks.

Implementation Details. We adopt a 2D slice-based segmentation paradigm and resize all slices to 128 × 128. DSAGR is implemented as a 2D U-Net-based diffusion generator, augmented with self-attention at intermediate resolutions. The generator is trained for 200 epochs with Adam (learning rate 1 × 10-4 , batch size 16). For feature extraction, we employ a VGG-19 network pre-trained on ImageNet. The segmentation network uses a 2D Swin-ViT encoder and a CNN decoder. On the source domain D1, we train for 400 epochs with Adam (initial learning rate 1×10-4 , polynomial decay, batch size 24). For each subsequent domain t > 1, we initialize the student generator and segmentor from step t-1 and jointly optimize them for 200 epochs. Additionally, we set the loss weights to λtraj = 0.5, λcon = 1.0, λPCL = 0.1, and λKD = 0.5. The prototype EMA momentum is set to m = 0.99, the pseudo label confidence threshold to τc = 0.5,

the contrastive temperature to τ = 0.1, and the distillation temperature to T = 10. Random flips and rotations are applied for data augmentation. All models are implemented in PyTorch and trained on two NVIDIA RTX 3090 GPUs. Additionally, DSA-GR introduces cost only during continual adaptation and does not increase inference cost, since final prediction uses only the segmentation network.

Evaluation Protocol and Metrics. We evaluate segmentation performance with Dice and Hausdorff distance (HD), and generation quality with Fr´echet Inception Distance (FID) and Learned Perceptual Image Patch Similarity (LPIPS). To characterise continual domain adaptation over domains, we follow standard continual learning protocols and report average performance (AVG) and backward transfer (BWT).

5 Experiments and Results

5.1 Generative Methods Comparison

We quantitatively compare different generative methods using FID and LPIPS, as reported in Table 1. All models generate 1000 images with a 20-step DDIM sampler. Compared with Vanilla GR, our DSA-GR substantially reduces FID from 68.63 to 26.62 on MYOSAIQ dataset and from 59.47 to 27.19 on M&MS dataset, while also achieving lower LPIPS scores. These results indicate that DSA-GR improves both cross-domain image fidelity and diversity.

Datasets Vanilla GR DSA-GR (Ours) Joint Training

FID (↓) LPIPS (↓) FID (↓) LPIPS (↓) FID (↓) LPIPS (↓)

MYOSAIQ 68.63 0.42 26.62 0.12 21.94 0.11

59.47 0.40 27.19 0.16 20.80 0.14

Qualitative results in Figure 5 further highlight the strengths of our DSA-GR in content preservation and style adaptation. On MYOSAIQ dataset, DSA-GR captures domain-specific features, such as brightness shifts between D1 and D2, which are critical for segmentation generalization. In contrast, while Vanilla GR exhibits structural blurring and texture loss, DSA-GR retains sharper edges and fewer artifacts, approaching the visual quality of the joint-training upper bound.

5.2 SegmentationMethods Comparison

To evaluate our method, we compare it with several state-ofthe-art (SOTA) UDA and continual UDA approaches: standard UDA methods AdaptSegNet [Tsai et al., 2018] and AdvEnt [Vu et al., 2019], source-free UDA methods UGTST [Luo et al., 2024] and SLCL [Gu et al., 2024], and continual UDA methods MuHDi [Saporta et al., 2022], GarDA [Chen et al., 2023], and CLMS [Li et al., 2025].

For a fair comparison, all methods are adapted to the privacy-constrained continual UDA setting, where real source-domain images are unavailable during target adaptation. Following GarDA [Chen et al., 2023], for sourcedependent UDA baselines such as AdaptSegNet and AdvEnt, we use synthetic source images generated by the sourcetrained DSA-GR generator, with pseudo-labels produced by a pre-trained source model. This ensures that no real historical images are revisited while all methods are evaluated under the same continual adaptation protocol.

Performance on the MYOSAIQ Dataset. As shown in

Table 2, the source-only baseline achieves an average Dice

of 71.78% and HD of 9.34, with clearly negative BWT of

-8.67%, indicating severe domain drift and catastrophic for-

getting. In contrast, our method attains an average Dice of

79.16% and HD of 7.45. Compared with continual UDA

baselines, our framework further improves average Dice by

1.81% over MuHDi and 1.57% over GarDA.

Notably, our method achieves the best performance on

the final domain D3, indicating stronger plasticity under se-

quential domain shifts. At the same time, our approach

yields more favourable BWT values than competing meth-

ods, highlighting its ability to suppress catastrophic forget-

ting while adapting to new domains. Qualitative examples in D1 → D2 → D3

Single-Step UDA

AdaptSeg [Tsai et al., 2018] 89.42 81.06 76.46 82.31 87.43 68.09 48.46 67.99 89.48 66.60 49.35 67.48 72.59 -8.17

AdvEnt [Vu et al., 2019]

89.85 81.94 77.35 83.04 87.78 68.73 47.51 68.01 89.77 68.85 51.93 70.18 73.74 -8.14

UGTST [Luo et al., 2024] 90.61 81.37 78.42 83.47 87.48 70.35 52.66 70.16 89.23 72.15 52.05 71.14 74.92 -7.63

SLCL [Gu et al., 2024]

90.59 81.40 78.29 83.43 88.55 72.20 53.87 71.54 89.07 74.13 53.00 72.07 75.68 -7.30

Continual UDA

MuHDi [Saporta et al., 2022] 90.98 81.38 78.00 83.45 91.07 74.22 56.50 73.93 90.74 75.45 57.88 74.69 77.35 -7.97

GarDA [Chen et al., 2023] 90.26 82.09 78.36 83.57 90.89 76.24 57.11 74.75 90.86 76.40 56.07 74.44 77.59 -7.11

CLMS [Li et al., 2025]

91.02 82.11 78.04 83.72 90.40 75.89 57.72 74.67 90.91 76.87 57.12 74.96 77.78 -7.36

Our

91.23 82.54 78.43 84.07 91.19 77.13 60.62 76.31 92.04 78.26 60.97 77.09 79.16 -6.43

D1

Source-Only

3.96 11.98 7.32 7.75 5.94 11.42 14.37 10.57 5.63 9.65 13.79 9.69 9.34 2.92

D1 → D2 → D3

Single-Step UDA

AdaptSeg [Tsai et al., 2018] 4.61 12.72 7.65 8.33 5.36 10.38 12.55 9.43 4.82 8.24 13.25 8.77 8.84 2.81

AdvEnt [Vu et al., 2019]

4.23 13.81 7.87 8.64 5.07 9.95 12.88 9.30 5.07 8.66 12.74 8.82 8.92 2.29

UGTST [Luo et al., 2024]

4.01 13.01 7.53 8.18 5.02 9.67 12.46 9.05 4.65 8.42 12.66 8.55 8.59 2.20

SLCL [Gu et al., 2024]

4.09 12.19 7.79 8.02 4.89 8.98 11.41 8.43 4.86 8.06 12.04 8.32 8.26 2.42

Continual UDA

MuHDi [Saporta et al., 2022] 4.01 12.23 7.34 7.86 4.12 8.69 10.84 7.88 4.37 7.68 11.52 7.86 7.87 2.73

GarDA [Chen et al., 2023] 4.12 11.86 7.56 7.84 4.33 8.21 10.53 7.69 4.42 7.55 11.65 7.87 7.80 2.24

CLMS [Li et al., 2025]

4.06 11.97 7.89 7.97 4.26 8.34 11.07 7.89 4.40 7.46 11.57 7.81 7.89 2.45

Our

3.97 11.77 7.11 7.62 4.05 7.99 9.94 7.33 3.90 7.22 11.08 7.40 7.45 1.87

Table 2: Comparison with several SOTA methods on the MYOSAIQ dataset in terms of Dice (top) and HD (bottom).

Figure 6 also show that our segmentations better delineate lesion boundaries and distinguish MVO from infarct/scar, with fewer false positives and false negatives across both D2 and D3 domains.

Performance on the M&MS Dataset. We further assess generalization and robustness on the M&MS dataset under the same continual setting. Results in Table 3 and Figure 7 confirm similar trends. The source-only baseline suffers from substantial cross-vendor drift, with an average Dice of 74.21%. Our method achieves 81.39% average Dice, outperforming MuHDi (78.67%) and GarDA (79.46%).

Although the inter-domain variation on M&MS is generally stronger than on MYOSAIQ dataset, our approach maintains competitive performance even on the final domain D3, obtaining Dice and HD of 76.71% and 5.11 on key structures (LV, RV, MYO). Qualitative results in Figure 7 further illustrate that our method produces clearer and more anatomically consistent boundaries, especially for the RV, compared with other methods. These results collectively show that our framework can handle continual domain drift, achieving a favourable balance between stability and plasticity.

5.3 Ablation Experiments

Ablation Analysis of Key Components. We conduct ablation studies on the MYOSAIQ dataset and report results on target domains D2 and D3.

As shown in Table 4, the baseline UDA approach suffers from the clear domain shift. Adding DSA-GR to the baseline yields substantial gains, improving Dice BWT from -8.13% to -6.87%, which confirms that high-fidelity, styleadaptive replay effectively mitigates forgetting. Incorporating CPFC alone achieves a similar boost, with an average Dice of 76.92% and HD reduced to 7.63, demonstrating its ability to stabilize adaptation via prototype-guided feature consolidation. Our full model, combining DSA-GR and CPFC, achieves the best performance with an average Dice of 79.16%, HD of 7.45. These results highlight the complementary roles of DSA-GR and CPFC: replay preserves historical knowledge, while prototype-guided regularization improves stability under noisy supervision.

Effectiveness of Components in DSA-GR. We further analyze key components of DSA-GR on the M&MS dataset. Since generative replay is only used on target domains, we report results on target domains D2 and D3.

As summarized in Table 5 and Figure 8, removing either step-wise distillation or style adaptation leads to noticeable performance drops across domains, especially on the final domain at step 3. The full DSA-GR configuration consistently achieves the highest Dice of 81.39%, indicating that trajectory-level distillation and style adaptation are both indispensable and work synergistically to produce high-quality, domain-aligned replay for continual UDA.

Effectiveness of the CPFC Mechanism. We evaluate CPFC on MYOSAIQ dataset via stepwise ablation, starting from a baseline and treating the previous model as a fixed teacher. We then incrementally add Temporal KD and Prototype-guided Contrastive Learning (PCL).

As shown in Table 6, the baseline exhibits poor generaliza-

86.0

81.8

77.6

73.4

69.2

65.0

Table 6: Ablation study of CPFC component on MYOSAIQ dataset.

tion and catastrophic forgetting. Adding KD improves average Dice by 2.49%, confirming the benefit of distilling soft targets from the teacher. PCL further improves performance, leading to higher average Dice and increased stability across domains. The full CPFC configuration achieves a Dice of 79.16% and an HD of 7.45. The t-SNE visualization in Fig-

ure 9 further confirms this observation. Overall, CPFC ef-

fectively curbs feature-space drift, preserves prior knowledge

and facilitates adaptation to new domains.

6 Conclusion

In this study, we proposed a novel continual UDA framework for multi-domain cardiac image segmentation, aiming to enable continuous adaptation to sequentially arriving unlabeled domains without accessing historical data. Experiments on two public multi-domain cardiac MRI datasets demonstrate that our method consistently outperforms SOTA approaches and achieves more robust continual adaptation. Furthermore, the proposed style-adaptive replay and prototype consolidation mechanisms are generally applicable in principle and can potentially be extended to other continual adaptation scenarios where historical data are unavailable.

Acknowledgments

This work was supported in part by the National Science Foundation of China (NSFC) under Grant 62472142, in part by the Beijing Natural Science Foundation under Grant Z210013, and in part by the Jing-Jin-Ji Incorporation Project of the Natural Science Foundation of Hebei Province under Grant H2024202009.