ReplayCAD: Generative Diffusion Replay for Continual Anomaly Detection

1 Introduction

Nowadays, unsupervised Anomaly Detection (AD) is crucial in modern manufacturing [Liu et al., 2024b]. It automatically identifies defects in samples at image and pixel levels, enhancing production efficiency and product quality while reducing reliance on manual inspections. However, in practical industrial production, changes in the generation plan continuously produce new class samples, requiring the model to learn knowledge from new samples while retaining knowledge of previous ones, i.e., Continual Anomaly Detection (CAD).

∗Corresponding author.

The key challenge of CAD lies in two aspects: First, catastrophic forgetting, where models tend to catastrophically forget historical class knowledge when learning new class data. Second, CAD requires not only the identification of imagelevel anomalous samples but also the segmentation of pixellevel anomalous regions. Since anomaly regions are usually small (As illustrated in Figure 3), this requires the model to remember more detailed pixel-level features from historical samples to ensure accurate anomaly segmentation.

A straightforward approach to addressing these two challenges is to apply existing Continual Learning (CL) methods (e.g. EWC [Kirkpatrick et al., 2017], MAS [Aljundi et al., 2018]) to anomaly detection models. However, existing CL methods do not focus on preserving the detailed features of historical samples, making them ineffective for CAD [Tang et al., 2025]. Advanced CAD methods [Li et al., 2022; Liu et al., 2024a] aim to mitigate catastrophic forgetting by compressing and storing features of historical samples. Specifically, DNE [Li et al., 2022] stores the mean and variance of image features. However, DNE remembers only the overall features of the distribution while ignoring the detailed features of individual samples. UCAD [Liu et al., 2024a] divides the image into multiple patches and encodes them into features for storage, resulting in the loss of pixel-level details from historical samples. In summary, these methods fail to preserve the pixel-level detailed features of samples, which negatively impacts segmentation performance.

To address this, we propose ReplayCAD, a novel CAD framework that can replay real and diverse historical data, thus preserving pixel-level detailed features. ReplayCAD consists of two stages: (1) compressing historical data of each class into conditional features; and (2) utilizing these features to replay historical data, and using the replayed data together with new class data for model training. Our key contribution lies in enhancing first-stage data compression to generate more diverse and authentic historical data. Specifically, in the first stage, we employ reverse engineering to learn a class semantic embedding in the conditional space of a pre-trained diffusion model for each class's historical data. This class semantic embedding guides the diffusion model to generate historical data with pixel-level detailed features, significantly enhancing the segmentation performance of anomalous regions. However, as illustrated in Figure 2, samples contain not only semantic features but also rich spatial features. Compressing data using semantic features alone leads to replayed data lacking spatial diversity. To address this issue, we utilize the sample's mask as its spatial features to guide the data compression process. This approach enables controlled generation of spatial positions, resulting in the replay of more diverse historical data. Our method offers several advantages: Firstly, compared to current CAD methods based on feature replay [Li et al., 2022; Liu et al., 2024a] and regularization [Tang et al., 2025; Li et al., 2025], the replay-based approach is more effective in preserving the detailed features of samples, which leads to better performance in anomaly segmentation. Secondly, by considering both semantic and spatial features, we generate high-quality and diverse samples, which significantly mitigates catastrophic forgetting. Finally, by guiding the pretrained diffusion model to perform reverse engineering, we significantly reduce training time and cost compared to existing replay methods [Shin et al., 2017; Wang et al., 2021; Gao and Liu, 2023; Li et al., 2024] that require training the entire generator.

To sum up, our contributions are as follows:

• We propose a novel CAD framework ReplayCAD, which effectively preserves fine-grained pixel details through high-quality data replay, significantly enhancing anomaly segmentation performance. To our best knowledge, this is the first attempt to apply diffusion-driven generative replay approach in CAD.
• To generate more diverse historical data, we employ spatial features of samples to guide data compression, thereby achieving spatially controlled data replay.
• Extensive experiments demonstrate that our method significantly outperforms SOTA approaches, particularly in segmentation performance, achieving improvements of 11.5% and 8.1% on VisA and MVTec, respectively.

2 RelatedWork

2.1 Unsupervised Anomaly Detection

Currently, unsupervised anomaly detection methods can be classified into the following three types: Embedding-based methods [Roth et al., 2022; Deng and Li, 2022; Liu et al., 2023] model the feature distribution of normal samples and detect anomalies by identifying deviations of anomalous samples from this distribution. Reconstruction-based methods [You et al., 2022; He et al., 2024; Zhang et al., 2024a; Wu et al., 2024] train a model to reconstruct normal images and detect anomalies by comparing the differences between the input image and the reconstructed image. Synthesizingbased methods [Hu et al., 2024; Li et al., 2021; Zavrtanik et al., 2021a; Zuo et al., 2024] train a model by adding artificially synthesized anomalies to normal images, thereby improving anomaly detection capability. However, in realworld industrial scenarios where new classes of data frequently emerge, and existing AD methods lack the capability for continual learning, our goal is to equip existing anomaly detection methods with continual learning capability.

2.2 Continual Learning

The current methods for continual learning can be broadly categorized into the following three types: Regularizationbased methods [Kirkpatrick et al., 2017; Pan et al., 2020; Aljundi et al., 2018] prevent forgetting of previous tasks by imposing constraints on model parameters. Replay-based methods [Rebuffi et al., 2017; Liu et al., 2020; Riemer et al., 2019] consolidate the retention of prior knowledge by replaying data or features from past tasks. Architecture-based methods [Mallya and Lazebnik, 2018; Mallya et al., 2018; Yoon et al., 2017] assign specific parameters to each task, isolating them to minimize interference. However, existing CL methods are not specifically designed to preserve detailed features of historical samples in CAD, making them unsuitable for direct application to AD models.

2.3 Continual Anomaly Detection

Current CAD research can be broadly categorized into two approaches: Feature replay-based [Li et al., 2022; Liu et al., 2024a] and Regularization-based [Tang et al., 2025; Li et al., 2025]. DNE [Li et al., 2022] and UCAD [Liu et al., 2024a] store the feature embeddings of samples from previous classes and locate anomalies by comparing the test sample features with the stored embeddings. However, these methods cannot save the pixel-level detailed features, resulting in poor performance of anomalous region segmentation. IUF [Tang et al., 2025] and CDAD [Li et al., 2025]prevent knowledge forgetting by restricting parameter updates for previous classes, but its performance deteriorates in long class sequences. To address these issues, we focus on the data itself to remember detailed features, we propose a novel CAD framework that can replay high-quality historical data.

3 ProblemDefinition

In this section, we provide a detailed introduction to the specific setting of the CAD. Unlike traditional AD task where all classes are known during training, CAD requires the model to continuously learn new classes, essentially falling under the category of class-incremental learning. Specifically, for an anomaly detection dataset with N classes, its training and testing sets are represented as {X1,X2, · · · ,XN} and {Y1, Y2, · · · , YN}, respectively. When learning the i-th class, we can only access the training data Xi and cannot access data from previous classes {X1,X2, · · · ,Xi-1}, during testing, we need to perform testing on {X1, X2, · · · , Xi}. Our goal is to sequentially train on the training set of each class and ultimately evaluate on the testing set {Y1, Y2, · · · , YN}.

4 Methods

4.1 Overview

ReplayCAD consists of two stages: feature-guided data compression stage and replay-enhanced anomaly detection stage. In the first stage, we compress the data of historical classes from both semantic and spatial dimensions, thereby extracting corresponding conditional features, as shown in Figure 4. Specifically, for semantic features, we search for a class semantic embedding within the conditional space of the pretrained LDM to guide the model in generating samples for the corresponding class. For spatial features, we learn the mapping relationship between the mask and the spatial features, where the mask is obtained through SAM [Kirillov et al., 2023]. In the second stage, we use the conditional features learned in the first stage to replay the historical class data, and train the anomaly detection model together with the new class data. The main contribution of this paper is to enhance the data compression of the first stage from both semantic and spatial dimensions, so as to generate more diversified and authentic historical data. Therefore, in the following, we will focus on how to use semantic and spatial features to guide LDM to compress samples in the first stage.

4.2 Semantic-Aware Generative Replay

In this section, we will investigate how to accomplish the two stages of generative replay using only semantic features.

Latent Diffusion Model (LDM) [Rombach et al., 2022] generates data by denoising in a low-dimensional latent space. Compared to directly working in high-dimensional pixel space, this approach significantly improves computational efficiency while preserving essential semantic information, ensuring high-quality outputs. To further enhance controllability, LDM incorporates a conditional guidance mechanism. This allows conditions such as text descriptions, image masks, or class labels to guide the generation process, ensuring that the results closely align with the given conditions while maintaining diversity and flexibility. Specifically, for the input image x , the LDM first encodes it using a latent encoder E and then adds random noise ϵ to generate the noisy image zt for denoising. Subsequently, the noisy image zt and conditions c are fed into the denoising U-Net, where the noise ϵθ is predicted using a cross-attention mechanism [Dai et al., 2023; Dai et al., 2024; Peng et al., 2024; Hu and Huang, 2025]. The optimization function of LDM is as follows:

Instead of directly training the LDM, we learn the conditional feature to guide it in generating samples for the corresponding class. Specifically, to represent semantic features, we first randomly initialize a learnable embedding v ∈ RK×C , which serves as the foundational representation. In addition, we take a text prompt p (e.g., "a photo of a"), encode it using a frozen text encoder T , and then concatenate it with the semantic embedding v to form the final semantic features esemantic, guiding the LDM's denoising process. The optimization objective is as follows:

We store the learned semantic embedding v for each class. When new class data appears, we use these semantic embeddings to guide LDM to replay the historical class data. After replaying data from historical classes, we integrate this data with new class data to train the AD model, effectively mitigating the catastrophic forgetting of the historical classes. Our approach imposes no restrictions on anomaly detection models, in this study, we select the multi-class anomaly detection model InvAD [Zhang et al., 2024b] as the baseline. The multi-class AD model combines samples from all classes during training, eliminating the need to know sample classes beforehand during inference, thus addressing class uncertainty in CAD. InvAD detects multiple types of anomalies by introducing a feature inversion mechanism and leveraging highquality feature reconstruction capabilities. Specifically, InvAD consists of an encoder E and a generator G. For a given normal input sample x, its original feature F is first extracted using a frozen encoder, and then its reconstructed feature F∗ are generated by the generator:

Then, the abnormal region is obtained by calculating the difference between the original feature F and the generated feature F∗ . The loss function is defined as follows:

Enhanced Masks

Figure 5: Overview of Data Replay. We use stored semantic embeddings v, the mlp layer, and mask images to guide the LDM in replaying data from historical classes. To enhance the diversity of replay samples, we apply random augmentations to the masks.

4.3 Spatially-Enhanced Diverse Generation

However, the samples not only present semantic features, but also various spatial features, as shown in Figure 2, focusing solely on learning semantic features may cause the generated samples to overfit certain specific examples [Yang et al., 2025; Hu et al., 2024; Liang et al., 2024], resulting in a lack of spatial diversity. Therefore, we introduce spatial features to guide the data compression process.

The spatial features of a sample include its coordinates and angle, which are difficult to represent directly. To address this, we use a mask image to denote the sample's spatial features. However, due to the lack of mask labels in the training set, we utilize SAM's [Kirillov et al., 2023] powerful zero-shot segmentation capability to generate the mask for each sample. As the diffusion model cannot directly interpret mask information, we optimize a mlp layer to learn the mapping from the mask to spatial features, thus guiding the diffusion model in denoising. Specifically, for an input image x ∈ RH×W×3 , we first generate its corresponding object mask image by SAM. The mask image is then encoded by a latent encoder E to produce feature representations xmask ∈ R H8 × W8 ×4 , which are then reshaped to match the input size required by the mlp layer, resulting in x′mask ∈ R HW 16n ×n . Subsequently, a mlp layer W with a size of (n,m) is applied to compute the spatial feature xspatial ∈ R HW 16n ×m. Finally, xspatial is reshaped again to align with the size of the semantic features, yielding the final spatial features espatial ∈ RM×C .

We concatenate the extracted semantic features esemantic and spatial features espatial to obtain the condition ec, which is used to guide the denoising process of the LDM:

Our goal is to find the best semantic embedding v as well as the mlp layer W for each class, the optimization objectives are outlined as follows:

After training is completed for each class, we save the semantic embedding v and the mlp layer W for subsequent data replay. In addition, M(M ∈ [1, 10]) randomly selected mask images are stored to control the spatial positioning of the generated images. These stored components constitute the conditional features for each class, requiring minimal storage space while enabling precise control over both semantic and spatial aspects during data replay.

In the second stage, to achieve real and diverse data replay, we use stored conditional features to achieve controllable generation in both semantic and spatial aspects, as illustrated in Figure 5. Semantically, we use semantic embedding v to control the class of generated samples. Spatially, we control the spatial position of samples through image masks and the mlp layer W in the spatial encoder. In addition, we randomly rotate and shift the stored masks to enhance their spatial diversity. By achieving semantic and spatial controllable generation, we can generate real and diverse historical data, thus preserving pixel-level details more efficiently.

5 Experiments

5.1 Experiments Setup

Datasets. We conducted experiments on two commonly used industrial anomaly detection datasets: VisA [Zou et al., 2022] and MVTec [Bergmann et al., 2019]. VisA comprises 12 sample classes, with 8659 training samples (all normal) and 2162 test samples (962 normal, 1200 anomalous). MVTec includes 15 sample classes, with 3629 training samples (all normal) and 1725 test samples (467 normal, 1258 anomalous).

Compared Methods. We compare our approach with three methods: (1) CAD Methods. We compare our method with SOTA CAD methods, including CDAD [Li et al., 2025], IUF [Tang et al., 2025], UCAD [Liu et al., 2024a], and DNE [Li et al., 2022]. Additionally, we migrate the framework of DNE to PANDA [Reiss et al., 2021] and CutPaste [Li et al., 2021]. (2) ADMethods. We apply current SOTA AD methods directly to CAD, including PatchCore [Roth et al., 2022], SimpleNet [Liu et al., 2023], MambaAD [He et al., 2024] and InvAD [Zhang et al., 2024b], we fine-tune the old model as new class data comes in. (3) AD+CLMethods. We adapt CL methods, such as EWC [Kirkpatrick et al., 2017] and MAS [Aljundi et al., 2018], to SimpleNet and MambaAD to enhance their adaptability in CAD.

Metrics. Similar to previous methods, we assess the performance of anomaly detection from both classification and segmentation perspectives. For classification performance, we use Area Under the Receiver Operating Characteristics (AUROC) [Zhou and Paffenroth, 2017], while for segmentation performance, we use Area Under Precision-Recall (AP) [Zavrtanik et al., 2021b]. Moreover, we use Forgetting Measure (FM) [Chaudhry et al., 2018] to evaluate the extent to which the model forgets historical classes:

Here, N represents the total number of classes, i denotes the class number, and a indicates the classification or segmentation score for that class.

Implementation Details. We set the number of tokens K for the semantic embedding v to 20. For ViSA, we use pre-trained Stable Diffusion 1.5 weights with a resolution of

(512, 512), where the dimension C of the semantic and spatial features is 768, and the MLP layer size is (128, 196). For MVTec, we use pre-trained LDM weights with a resolution of (256, 256), where the dimension C of the semantic and spatial features is 1280, and the MLP layer size is (128, 200). We generate 800 samples for each class for training the anomaly detection model. We use Invad [Zhang et al., 2024b] as the baseline model for anomaly detection, keeping all settings consistent with the original paper.

5.2 Experiment Performance

Quantitative Evaluation. As shown in Table 1, our method achieves state-of-the-art performance on both image-level and pixel-level metrics across two datasets. Specifically, we improved pixel-level metrics by 11.5% on VisA and 8.1% on MVTec compared to the previous state-of-the-art method. Existing anomaly detection methods often suffer from catastrophic forgetting in continual learning scenarios. Although some continual learning methods can mitigate this issue, there remains a significant gap compared to our approach. We observe that although IUF performs poorly in terms of pixellevel and image-level metrics, it demonstrates relatively low FM scores. This is because IUF experiences severe feature conflicts between new and historical classes when learning data from new classes. This conflict causes the historical best performance aγ,i (Eq. 8) of new classes to fall below their actual potential, thus resulting in lower FM scores. Additionally, although UCAD reported the FM metric in the original paper, its historical best score aγ,i was achieved by storing a larger number of patch features, which fails to reflect the true FMmetric. Therefore, we marked it as "-".

Additionally, we compare our method's storage space with the SOTA methods. As shown in Table 2, different from UCAD storage patch-level features and CDAD storage gradient projection, our approach significantly reduces storage space by storing semantic and spatial features of samples,

• (a) Replayed Data

(b) Real Data

Figure 7: The visual comparison of replayed data and real data on the MVTec and VisA datasets.

while also achieving superior experimental performance.

Qualitative Evaluation. Figure 6 shows the heatmaps of anomaly regions predicted by our method and other SOTA methods. We found that the AD methods, AD + CL methods, and IUF often result in a large area of error prediction. This occurs because, when learning new classes, these methods conflict with the features of historical classes, leading to catastrophic forgetting of previously acquired knowledge. Although UCAD shows significant improvement compared to previous methods, it still suffers frommissed detections in anomalous regions. Visualization results demonstrate that our method achieves higher accuracy in anomaly segmentation.

Visualization of Replayed Data. As shown in Figure 7, we present a comparison between the replayed data and the real data. Since AD is conducted in an unsupervised setting, the replayed data should closely resemble the real normal data. The results indicate that the replayed data exhibit a high degree of similarity to the real data, validating the effectiveness of our framework based on generative replay.

5.3 Analysis

In this section, we analyze the impact of the two proposed features and different AD models on MVTec.

Effect of semantic and spatial features. As shown in Table 3, we analyze the effectiveness of the proposed semantic features esemantic and spatial features espatial. T1 indicates that we do not retain any features of historical samples but instead directly train the AD model sequentially on samples from each class. In this case, the model suffers from catastrophic forgetting, resulting in a sharp decline in performance. T2 and T3 represent the processes of learning and storing spatial features and semantic features of the samples, respectively, followed by replaying historical data using the learned features. The results indicate that both types of features can significantly alleviate catastrophic forgetting. T4 simultaneously learns both spatial and semantic features. Compared to learning them individually, this approach significantly improves classification and segmentation performance, further demonstrating the critical role of these two types of features in replaying historical data.

Transferring to different AD models. To validate the generalization capability of the replayed data, we transfer it to various anomaly detection models for training, including Patchcore [Roth et al., 2022], RLR [He et al., 2025], MambaAD [He et al., 2024], and InvAD [Zhang et al., 2024b], as shown in Table 4. Our replayed data demonstrated strong performance across four methods, confirming the superior generalization ability of the proposed method. Moreover, this indicates that our method can be readily adapted to SOTA AD methods at any time. Compared to other CAD methods that can only adapt to a single AD method, our approach demonstrates greater adaptability and sustainability.

6 Conclusion

In this paper, we introduce ReplayCAD, marking the first use of generative replay based on a pre-trained diffusion model in CAD. Compared to current state-of-the-art methods, our approach more effectively preserves the detailed features of historical samples, leading to significant improvements in segmenting anomalous regions. Additionally, by incorporating semantic and spatial feature guidance into data compression, we achieve high-quality data generation. Experimental results demonstrate that our method achieves SOTA performance on both image-level and pixel-level metrics.

Acknowledgments

The research is partially supported by National Natural Science Foundation of China (No.62176093, 61673182), National Key Research and Development Program of China (No.2023YFC3502900), Key Realm Research and Development Program of Guangzhou (No.202206030001), Guangdong-Hong Kong-Macao Joint Innovation Project (No.2023A0505030016), Guangdong Emergency Management Science and Technology Program (No.2025YJKY001).