DriftRemover: Hybrid Energy Optimizations for Anomaly Images Synthesis and Segmentation

DriftRemover: Hybrid Energy Optimizations for Anomaly Images

Synthesis and Segmentation

Siyue Yao1,2 , Haotian Xu3 , Mingjie Sun4∗ , Siyue Yu1 , Jimin Xiao1 , Eng Gee Lim1

1Xi'an Jiaotong-Liverpool University

2University of Liverpool

3RippleInfo

4Soochow University

Abstract

This paper tackles the challenge of anomaly im-

age synthesis and segmentation to generate var-

ious anomaly images and their segmentation la-

bels to mitigate the issue of data scarcity. Exist-

ing approaches employ the precise mask to guide

the generation, relying on additional mask gen-

erators, leading to increased computational costs

and limited anomaly diversity. Although a few

works use coarse masks as the guidance to expand

diversity, they lack effective generation of labels

for synthetic images, thereby reducing their prac-

ticality. Therefore, our proposed method simulta-

neously generates anomaly images and their cor-

responding masks by utilizing coarse masks and

anomaly categories. The framework utilizes at-

tention maps from synthesis process as mask la-

bels and employs two optimization modules to

tackle drift challenges, which are mismatches be-

tween synthetic results and real situations. Our

evaluation demonstrates that our method improves

pixel-level AP by 1.3% and F1-MAX by 1.8% in

anomaly detection tasks on the MVTec dataset.

Additionally, its successful application in practi-

cal scenarios highlights its effectiveness, improv-

ing IoU by 37.2% and F-measure by 25.1% with

the Floor Dirt dataset. The code is available at

https://github.com/JJessicaYao/DriftRemover.

1 Introduction

Anomaly Images Synthesis and Segmentation (AISS) task

aims to generate various types of anomaly images and their

segmentation labels for downstream tasks [Li et al., 2021;

Lin et al., 2021; Duan et al., 2023; Hu et al., 2023]. The

emergence of AISS task is due to the scarcity of real-world

anomaly samples, such as industrial inspection and home

cleaning scenarios, making it challenging for downstream

methods to locate anomalous regions that deviate from the

normal areas in images. In this way, AISS can significantly

increase the quantity and diversity of anomaly datasets for

downstream anomaly localization and segmentation methods.

∗Corresponding author (mjsun@suda.edu.cn).

Figure 1: Framework comparison between the baseline method and ours. (a) The baseline method directly uses attention maps to create mask labels of synthetic anomalies but suffers from artificial appearance and mismatches. (b) Two modules are proposed in our method to alleviate these issues. The Attention Activation Refinement module (depicted in purple) compels the attention map to converge with the refined mask, improving consistency between the anomaly region and its label. The Anomaly Pattern Optimization module (depicted in pink) ensures that generated anomaly distribution closely aligns with the real dataset, enhancing realism of generated images.

Previous AISS methods, such as AnomalyDiffusion [Hu et al., 2023], heavily rely on the guidance mask, created by additional models, to generate the image with the anomalous region located within this guidance mask, which increases the computational costs and limits synthetic anomaly diversity. To overcome this limitation, AnoGen [Guan Gui, 2024] employs the coarse mask (e.g., bounding box), rather than the precise mask, to guide generation process, enhancing synthetic diversity while reducing computational demands. However, this method cannot obtain precise mask labels for synthetic anomalous images, limiting the applicability of synthetic samples in downstream tasks. To solve this issue, we propose a new pipeline to extract cross-attention maps during anomalies generation, used as masks after binarization, as shown in Figure 1(a). These paired samples serve as images and labels for training in downstream tasks to enhance performance. Nevertheless, simply employing this naive pipeline as a synthetic baseline poses significant drift challenges.

The first drift challenge is the misalignment between generated mask label and synthetic anomaly region. As illustrated in Figure 1(a), the black-and-white image in the lower left corner clearly shows a discrepancy between the mask derived from the attention layer and the generated image in the lower right corner. The reason for the discrepancy is that when generating anomalies with keywords like "crack", the attention map activates broad regions and is accompanied by vague boundaries, leading to inaccurately binarized masks.

Another drift challenge is the distribution discrepancies between generated images and real-world scene images. This issue arises from the diffusion model's tendency to misunderstand ambiguous terms, leading to significant pattern and style inconsistency between synthetic and real anomalies [Kim et al., 2024]. For instance, in Figure 1(a), the synthetic anomalies are represented as a cluster of small white circles, however, the intended anomaly should depict a broken hazelnut, which fails to accurately represent the nature of real anomalies. Consequently, this drift limits the utility of synthetic data for training purposes in downstream tasks.

To address the aforementioned challenges, we undertake additional optimization of the proposed baseline by introducing two novel modules in the synthesis process. These modules specifically solve drift challenges of label mismatches and pattern discrepancies, which are critical for enhancing the authenticity of anomalies and the accuracy of masks.

The first module, Attention Activation Refinement, is meticulously designed to guarantee the congruence between the synthesised anomaly regions and their corresponding segmentation masks, as shown in the pink part of Figure 1(b). It initiates with a preliminary coarse mask, which is then subjected to an advanced refining process to generate a refined mask. By increasing the attention map's activation values within the refined mask region, the attention map achieves more distinct edges, facilitating the production of binarized masks that are more precisely aligned with the anomaly region. This refinement leads to a significant improvement in the accuracy of the mask's shape and position, surpassing the initial coarse mask's accuracy. The accurate mask synthesised by our method in Figure 1(b) corroborates this result.

The second module, Anomaly Pattern Optimization, aligns the category distribution of generated images with anomaly regions in real-world scenarios. Specifically, the method gathers the distribution of anomaly regions for various categories in the existing dataset. It minimizes the difference between the real and generated anomaly patterns to improve the realism of the synthesised anomaly images by making synthetic anomaly region distributions match the real corresponding category distributions. For instance, in the purple part of Figure 1(b), the image synthesised by our method can better reproduce the internal appearance of hazelnuts.

By harmoniously integrating these two optimization modules with the previously proposed baseline, our method, DriftRemover, effectively addresses AISS task and provides high-quality training data, which is crucial for subsequent anomaly localization and segmentation tasks. Our contributions can be encapsulated within the following three aspects:

• Our DriftRemover synthesises anomaly images conditioned on coarse masks and anomaly categories, while generates precise labels utilizing an attention mechanism. This effectively synthesises diverse anomaly images while maintaining high accuracy in labelling, thereby enhancing the downstream task performance.
• Two optimization modules is introduced to resolve drift issues in synthesis process: Attention Activation Refinement (AAR) and Anomaly Pattern Optimization (APO). AAR dynamically refines attention maps for precise segmentation masks, while APO enhances realism of synthesised anomalies by aligning their style with real ones.
• Extensive experiments demonstrate that our generated data effectively improves the performance of downstream tasks. On the MVTec dataset, AP and F1-MAX increase by 1.3% and 1.8%. On the Floor Dirt dataset, IoU and F-measure increase 37.2% and 25.1%.

2 RelatedWork

2.1 Anomaly Detection

Anomaly detection aims to identify samples that significantly deviate from the normal distribution, indicating anomalies. Previous methods fall into three categories: Reconstructionbased methods [Gong et al., 2019; Park et al., 2020; Ristea et al., 2022; Schl¨uter et al., 2022] analyse the residuals between input and reconstructed output, relying on the model's ability to reconstruct normal regions while struggling with abnormal ones. Embedding-based methods [Roth et al., 2022; Yao et al., 2023; Zhang et al., 2023c; Liu et al., 2023], including memory banks, distribution map, teacher-student and one-class classification, utilize pre-trained networks to extract features and compress normal features to separate anomalies, but they lack direct learning of anomaly features. Augmentation-based methods use synthetic anomaly images for training. Image-level approaches [Lin et al., 2021; Zavrtanik et al., 2021] provide detailed anomaly textures but lack diversity, while feature-level methods [Yan et al., 2021; You et al., 2022] are efficient but face challenges in control.

2.2 Anomaly Synthesis

The limited availability of anomaly data sparks significant interest in anomaly generation. Previous methods [Zavrtanik et al., 2021; Zhang et al., 2023a] use Perlin noise and cutpaste techniques but lack realism and diversity. Inspired by GANs [Goodfellow et al., 2014], researchers explore methods [Niu et al., 2020; Zhang et al., 2021] that require defectfree samples as input. DFMGAN [Duan et al., 2023] generates anomaly images and masks but struggles with complex objects. While diffusion models are widely used for image generation [Ho et al., 2020; Song and Ermon, 2019; Zhang et al., 2023b], recent approaches [Hu et al., 2023; Qiu et al., 2025] require separate models for precise masks, resulting in high computational costs. Moreover, AnoGen

[Guan Gui, 2024] uses bounding boxes to eliminate the need for a mask generator but fails to synthesize precise masks.

3 Method

The proposed DriftRemover pipeline, as depicted in Figure 2, effectively synthesises anomaly images while generating mask labels. During training, it employs learnable embeddings to extract features from various anomalies, aiding in both image synthesis and mask extraction. In inference, the Attention Activation Refinement module adjusts attention values to better align with anomaly regions, while the Anomaly Pattern Optimization module enhances the realism of the synthetic anomalies. This approach produces highquality synthetic data and accurately matched mask labels.

3.1 Preliminary

The AISS task recently adopts the state-of-the-art (SOTA) generative model Stable Diffusion (SD) [Rombach et al., 2022]. During the training stage, SD first encodes the real anomaly image xr into the latent space via an image encoder ε(·), resulting in a latent vector zr 0 = ε(xr ) with the shape of W ×H ×C, where W and H represent the size, and C is the number of channels. Afterward, Gaussian noise is gradually added to zr 0 in the forward process, as follows:

where zr t is the noisy latent vector, and αt is the standard deviation of the noise at timestep t. ¯αt represents the cumulative value from α0 to αt. I denotes the identity matrix. Finally, a U-Net model is used to estimate the noise increment at any step, and the objective is to minimize the difference between the output of the U-Net model and the added noise.

During inference, SD converts a normal image xn into a latent vector zn 0 , and then performs a T step forward diffusion process akin to Equation 1 to obtain zn T . In reverse, the welltrained U-Net model estimates the noise increment at each step, progressively inferring the latent vector with anomaly features zs from the noise zn T . Simultaneously, a text encoder converts input prompts into embeddings p ∈ Rl×q with l tokens, which serve as conditions to guide U-Net, as follows:

where q is the dimension of text feature; ϵθ(zs t , p, t) is the predicted noise at timestep t; βt is defined as 1 -αt.

The cross-attention layer in SD takes three inputs: a query vector Q, a key vector K, and a value vector V. K and V are computed from linear projections of the textual embeddings p, and Q is sourced from a linear projection of each convolution block output. By combining the vectors Q and K in different layers, the cross-attention map A is as follows:

where √d is a scaling factor, usually set to 1. For each layer, the map shape is Wk × Hk ×h×l, where k is the reduction factor associated with the block's resolution and h is head number. These matrices can be used to weight the image pixels, thereby influencing the final generated anomaly images.

3.2 Mask Labels Synthesis via AttentionModule

Cross-attention map reflects the model's focus on specific areas of an image based on the input text, allowing it to locate and distinguish objects. Typically, object category labels are used as text to activate cross-attention maps and then create masks [Marcos-Manch´on et al., 2024; Nguyen et al., 2024; Zhao et al., 2023]. However, most anomaly categories include multiple adjectives or verbs instead of simple nouns, making it difficult for activation maps to accurately focus on specific regions. To explore these words effects, we reorganize anomaly categories into descriptive statements to generate different activation maps. For example, the anomaly category "broken small" expands to "There is a small damage or break on the bottle". Figure 3 reveals a key issue: no single word effectively delineates the triggering factors of anomalies. Therefore, there is an urgent need to introduce new keywords or embeddings to accurately capture anomaly category features, which is used to activate attention maps.

Therefore, learnable embeddings are used to capture the semantic features of different types for various objects, such as "hazelnut good" and "bottle broken small". These embeddings are continuously updated until the SD model can effectively generate the corresponding distribution of anomalies. Specifically, an embedding p∗ for any anomaly type is established to replace the text conditions of the SD model, which can be optimized similarly to the LDM loss [Rombach et al., 2022]. Besides, to ensure that the learned text embeddings focus exclusively on the region defined by the coarse mask m without influence from external areas, the model input is constructed as the concatenation of the noisy embedding of the real anomaly image zr t , the coarse mask m used to determine the anomaly location, and the embedding of normal region in the anomaly image with coarse mask area removed ε(xr (1-m)). For simple notation, we still mark this input as zr t . Overall, the loss for training embeddings p∗ is as follows:

where p represents the text embedding of the anomaly category noun with the size Rl×q for initializing the learnable p∗ to reduce training duration. ϵ represents the noise added to the image. In this way, the text embeddings p∗ accurately reflect the characteristic of each anomaly category, thus better activating the corresponding visual representation regions.

Ultimately, the learned embeddings of different anomaly types are used to generate attention maps A in the final timestep. These maps are aggregated and binarized into a mask using Otsu's method [Otsu, 1979] to automatically find the optimal threshold. This process converts the attention maps into binary anomaly masks, resulting in mask labels paired with synthesised images for use in downstream tasks.

3.3 Dynamic Inference for Anomaly Synthesis

To craft a superior synthetic dataset, relying on attention maps for masks is insufficient due to drift issues, resulting in unrealism and label inconsistency. To alleviate such issues, we propose two modules based on energy function mechanism [Chen et al., 2024]. Specifically, the gradient of energy function F(c, zs t ) is computed to update latent variable zs t under conditions c, thereby improving generation quality. Therefore, the conditional sampling formula is written as follows:

where σt = (1 - αt)/αt is a scale factor. By updating the latent variable, all generated images are influenced by backward guidance. Additionally, our method strategically updates the noise vectors to synthesise anomalies without altering the regions outside the coarse masks, similar to the approach outlined in [Avrahami et al., 2022], as follows:

zs t-1 ← zs where zs t-1 is the latent vector synthesised by Equation 2. √ ¯αt-1zn 0 + √1 - ¯αt-1I is equivalent to the latent vector zn t , i.e., adding t step noise to the embedding of a normal image.

Attention Activation Refinement Module

The Attention Activation Refinement (AAR) module improves SD's cross-attention maps to pinpoint anomaly regions as shown in the pink area in Figure 2, using new embeddings for "hazelnut-good" and "hazelnut-crack" to activate the entire object region Agt and the rough region of the anomaly Ast . The overlap of these two activation maps roughly estimates the anomaly location. To keep synthesised anomalies within initial coarse mask, a refined mask ˆm is created by excluding out-of-bounds intersections as follows:

where ⊙ represents element-wise multiplication and η denotes a threshold for binarizing the attention maps to masks.

This mask guides the inference process to produce anomalies that closely match the expected areas. An energy function is tailored to maximize activation within the refined mask and minimize its surroundings on the activation map As of the new anomaly embedding p∗ , as follows:

where t illustrates the timestep. N and ˜ N represent the pixel point set with value 0 and value 1 in ˆm, representing the normal and anomaly regions. w and ˜ w can be regarded as weight measures and are calculated as follows:

where ˜ µ and µ represent the average activation values of the anomaly region and the remaining region of the activation map. All activation maps A are normalized in [0, 1].

Algorithm 1 Inference process of proposed DriftRemover

Input: normal image xn , coarse mask m, input anomaly prompt p∗ , total inference timestep T

Parameter: conditional noise predictor ϵθ(·), image encoder ε(·), binary mask threshold η, last timestep γ for adding AAR module, first timestep δ for adding APO module, function repeat times Γ, energy value threshold Θ, pre-defined parameters ¯αT , I, βt and σt

Output: the synthetic anomaly image latent zs 0

1: zn 0 = ε(xn ); zn T ∼N(0, I) 2: zsT = concat(zn T , m, ε(xn (1 - m))) 3: for t = T to 1 do 4: i = 0 5: if t > γ then 6: Obtain the attention map with new normal embedding Agt and new anomaly embedding Ast . 7: ˆm= (Agt > η) ⊙ (Ast > η) ⊙m ▷ Equation 7 8: while i < Γ and FR(Ast , ˆm) < Θ do 9: zs t ← zs t -σt∇zs t FR(Ast , ˆm) ▷ Equation 8 i+ = 1 10: end while 11: else if t < δ then 12: while i < Γ and FO (f(zr ), f(zs t )) < Θ do 13: zs t ← zs t -σt∇zs t FO (f (zr ), f (zs t ))▷ Equation 10 i+ = 1 14: end while 15: end if 16: zs t-1 = 1 √αt (zs t - 1-αt √1-¯αt ϵθ(zs t , p∗ , t)) + √βtI ▷ Equation 2 17: zs t-1 ← zs t-1m + (√ ¯αt-1zn 0 + √1- ¯αt-1I)(1 -m) ▷ Equation 6 18: end for 19: return zs 0

Anomaly Pattern Optimization Module

The Anomaly Pattern Optimization (APO) module aims to enhance the realism of synthetic anomaly images by minimizing the discrepancy between them and actual anomalies. The process begins with encoding real anomaly images xr through the image encoder ϵ(·), yielding features zr . These features are then segmented into v×v patches, and their mean and variance are calculated to establish the probability density function f(zr ) of the real anomaly distribution. For synthetic anomaly images, patches are extracted directly from the predicted noise features zs t , following the same procedure to determine their probability density. The final step involves computing the KL divergence Dkl between the real and synthetic anomaly distributions to measure their similarity:

By employing the KL Divergence, APO bridges the gap between synthetic and real anomaly images, thereby enhancing the realism of the generated anomaly images.

Efficiency Sampling with Sequential Modules

Using two modules simultaneously updating zs t during sampling is inefficient. The generation process is divided into an early "structure generation" phase and a later "detail refinement" phase [Hertz et al., 2022; Mokady et al., 2023]. Thus, we recommend applying the AAR module initially for better anomaly positioning, then switching to the APO module to enhance realism. A repeat strategy is also implemented to strengthen the impact of energy functions. If the energy value dips below a threshold Θ or the iteration cap Γ is hit at timestep t, we cease repetition and finalize zs t . Details of upgraded sampling algorithm are shown in Algorithm 1. Moreover, at the final timestep, we execute mask labels synthesis procedures in Section 3.2, using attention maps of learned embeddings to generate labels for the synthesised image.

4 Experiment

4.1 Dataset andMetrics

We evaluate our DriftRemover on MVTec [Bergmann et al., 2019] and Floor Dirt dataset. MVTec's original training set consists of 3,629 normal images without any anomaly, while its original test set contains 467 normal images and 1,258 anomaly images along with their corresponding mask labels for the anomaly areas. Subsequently, followed by [Hu et al., 2023], we randomly select 1/3 of the abnormal images for training DriftRemover and the remaining images are used to test the results of the downstream tasks. The Floor Dirt dataset is collected from robotic vacuum cleaners, containing two types of anomalies: stains on the floor (500 images) and pet faeces on the floor (458 images). In our experiments, 3/5 of anomalous images are randomly selected for training our DriftRemover, and 2/5 are used for testing downstream tasks.

The evaluation metrics are divided into two main categories. First, we assess the diversity and quality of the synthesised anomaly images using IC-LPIPS for diversity and Inception Score (IS), where higher values indicate better results. Kernel Inception Distance (KID) is also used for quality, where lower values indicate better results. Second, we measure the accuracy improvement of downstream methods using metrics like Area Under the Receiver Operating Characteristic Curve (AUROC), Intersection over Union (IoU), Average Precision (AP), and maximum F1 score (F1-MAX), all of which show better performance with higher values.

4.2 Implementation Details

Our pipeline is built on Stable Diffusion V1.5 [Rombach et al., 2022], training it for 2,000 epochs with batch size of 4 and image size of 512. The optimizer AdamW utilizes a scaled learning rate initialized to 1e-4. We use 20 steps and a guidance scale of 3.5 for image generation, producing 1,000 images per class for evaluation and training. The threshold Θ and iteration cap Γ are 0.01 and 5. The last timestep γ for adding AAR module is 600, while the first timestep δ for adding APO module is 300. The binary threshold η is 180, patch size v is 3, text dimension q is 768, head number h is 8 and reduction factors k for each layer are 1, 2 and 4.

IS ↑ IC-LPIPS ↑ KID ↓

0.09

0.06

0.07

0.11

0.14

-

0.13

0.28

0.15

0.11

0.20

0.08

0.32

0.13

0.32

0.06

0.22

0.19

0.23

0.10

0.26

0.07

F1-MAX ↑

94.4 / 53.1

94.7 / 62.1

98.7 / 76.3

98.9 / 78.1

4.3 Quality of Generated Anomalies and Labels

Quantitatively compared with the methods in Table 1, our approach obtains the IS of 1.83 on MVTec, exceeding the previous SOTA method AnomalyDiffusion [Hu et al., 2023], whose IS score is 1.80. Additionally, our approach significantly improves the KID value, reducing it to 0.06. Although the IC-LPIPS remains at 0.32, our DriftRemover preserves the diversity of the generated results while achieving a high degree of fidelity consistent with real images, thus bolstering the robustness of the downstream tasks. This consistent performance is further validated on the Floor Dirt dataset, underscoring the method's robustness and effectiveness.

Figure 4 presents a qualitative analysis of our method on MVTec and Floor Dirt datasets, comparing it with DFMGAN and AnomalyDiffusion. Our DriftRemover excels at generating images that closely align with the target anomaly pattern, such as accurately depicting a "leather cut" anomaly as a tear with white debris, unlike the coloured patches from other methods. It effectively addresses drift issues, captures the features of anomalies, and enables the shape and pattern of the anomalous areas to more accurately reflect reality.

Additionally, to validate the accuracy of our synthetic mask labels, we manually annotate the anomalies in the generated images and compare them with our generated labels using IoU metric, as depicted in Figure 5. Our DriftRemover

achieves an average IoU of 63.96% across all categories, significantly outperforming the IoU results of previous methods.

4.4 Effectiveness for DownstreamTask

Anomaly Detection and Localization Task

On the MVTec dataset, Table 2 benchmarks our method against other generation methods to expand training data for leather-cut cable-bent-wire floor-feaces floor-stains Mask labels are placed at the bottom right corner.

downstream detection and localization tasks. We compare with Crop-Paste, DRAEM, PRN, DFMGAN, AnomalyDiffusion methods, and synthesise 1,000 images per class to train a simple U-Net for downstream tasks. Our approach achieves 99.7% AP and 98.9% F1-MAX at the image level, and 82.7% pixel-level AP and 78.1% F1-MAX, surpassing competitors by 1.3% and 1.8%, respectively. This indicates the high quality of our synthetic anomalies and masks, enabling simple UNet to achieve exceptional performance of downstream tasks. Similar AUROC results are observed when compared to both supervised and unsupervised downstream models in Table 3.

Anomaly Segmentation Task

Table 4 evaluates the performance of our DriftRemover against DFMGAN and AnomalyDiffusion on Floor Dirt dataset for anomaly segmentation. We train a SegFormer [Xie et al., 2021] on 1,000 images from each method. Our method achieves the highest IoU of 75.40% and F-measure of 46.37%, surpassing other methods and demonstrating superior real-world capabilities in aiding anomaly segmentation.

4.5 Ablation Study

Effect of Different Components

Our main components: learnable anomaly embedding, Attention Activation Refinement (AAR) and Anomaly Pattern Optimization (APO) are evaluated on MVTec, including five settings in Table 5: 1) with none of these components; 2) only learnable anomaly embedding; 3) learnable anomaly embedding and AAR; 4) learnable anomaly embedding and APO; and 5) the full model (ours). For each setting, 1,000 imagemask pairs are first generated for quality evaluation and then used to train the downstreammodel for the localization evaluation. For setting 1), we directly use the embedding of original keywords (e.g., "crack") to synthesise images and labels while the other settings all use the newly learned embeddings.

The second line of Table 5 shows that the model using the new embeddings can significantly improve the localization task performance. For instance, the pixel-level AP and F1-MAX reach 35.35% and 38.41%, compared to 9.64% and 11.43% for the model using original embedding. This strongly demonstrates the importance and necessity of obtaining embeddings that are tailored to anomaly types, providing a solid data foundation for our future design. Additionally, setting 3) is suitable for the task requiring anomaly localization (83.6% in AP), while setting 4) is suited for applications prioritizing the realism of synthetic images (1.88% in IS). Our model utilizes a balanced setting 5) approach, which achieves a compromise between the generation fidelity (1.83% in IS) and the downstream task performance (82.69% in AP).

Effect of Different Feature Scales

The U-Net model in SD incorporates attention blocks at varying resolutions: 64×64, 32×32, and 16×16, corresponding to reduction factors k of 1, 2 and 4. In our DriftRemover, the 32 × 32 and 16 × 16 attention blocks are combined to form the basis of the synthetic mask label. To assess the effect of these attention blocks, we conducted evaluations with various block combinations, as detailed in Table 6. The findings indicate that the 32 × 32 and 16 × 16 blocks offer optimal performance. The 16 × 16 blocks, being in the deeper layers of the U-Net, are adept at processing semantic details, capturing finer-edge information. Conversely, the 32 × 32 blocks, which operate at a more structural level, provide precise location data of the anomaly regions. The synergistic use of these two resolutions yields an accurate segmentation mask.

5 Conclusion

We present DriftRemover, leveraging coarse masks to synthesise anomaly images and corresponding masks. It utilizes precise anomaly embeddings for attention-guided mask creation and incorporates two optimization modules: Attention Activation Refinement for accurate labels and Anomaly Pattern Optimization for realism. Extensive experiments validate our model's superiority over state-of-the-art methods in realism and its enhancement of downstream task performance. In future, we aim to explore co-optimization of anomaly detection algorithms with synthetic data generation to enhance the feedback loop for creating more effective training samples.

Acknowledgements

This work was supported by Young Scientists Fund of the National Natural Science Foundation of China (Grant No. 62302328), Jiangsu Province Foundation for Young Scientists (Grant No. BK20230482), Suzhou Key Laboratory Open Project (Grant No. 25SZZD07) and Jiangsu Manufacturing Strong Province Construction Special Fund Project (Grant Name: Research and Development and Industrialization of Intelligent Service Robots Integrating Large Model and Multimodal Technology).