CLIP-FSAC: Boosting CLIP for Few-Shot Anomaly Classification with Synthetic Anomalies

1 Introduction

Image anomaly detection (IAD) aims to detect anomalies in given images including classification (AC) and segmentation (AS) [Wu et al., 2023]. As technology develops by leaps and bounds, anomaly detection based on deep learning is widely used in many fields such as intrusion detection, fraud detection and industrial inspection. Especially in industrial inspection, anomaly detection plays an important role which determines the quality of products. However, anomaly detection is a non-trivial task as abnormal images are scarce and anomalies are agnostic and diverse. Meanwhile, annotating anomaly regions is costly and labor-exhausting. To address these issues, more and more anomaly detection methods find anomalies in an unsupervised manner. In this manner, only anomaly-free samples are needed and there are no extra annotations. Training on massive normal samples, these methods can achieve an excellent performance.

∗Co-Correpsonding Authors.

Nevertheless, data collecting in industrial scenarios is also a tricky challenge. With less training samples, the performance of these unsupervised methods will decrease substantially. Recently, some few-shot anomaly detection (FSAD) methods have been proposed. One prevailing paradigm is to solve FSAD by employing meta-learning [Huang et al., 2022] (See Figure 1(a)). Since the difficulty in collecting massive similar normal images, Graphcore [Xie et al., 2023] considers constructing a memory bank, which only needs a few normal images of target category (See Figure 1(b)). However, it neglects the burden brought by the additional inference time.

Pre-trained vision-language models (VLMs) [Alayrac et al., 2022; Radford et al., 2021] have received ever-higher attention recently. In FSAC, WinCLIP [Jeong et al., 2023] benefits from semantic connection and generalization ability of CLIP. It proposes a compositional prompt ensemble (CPE) on state words and prompt templates and aggregation of window/patch/image-level features aligned with texts. Inspired by WinCLIP, utilizing weight-pretrained visual encoder of CLIP as our training paradigm not only enables us to capture broader concepts with open vocabulary but also avoids introducing additional computational overheads brought by modules. However, directly matching representation from the original CLIP is sub-optimal in FSAD.

Text as domain-invariant prompt allows us to learn disentangled domain and category representation. It is crucial to design an optimal text prompt to capture visual-language relationship [Zhou et al., 2022; Yao et al., 2022]. In our earlier exploration, we change different combination of texts in zeroshot anomaly classification with CLIP and find that results vary widely. It demonstrates that it is difficult to find accurate prompts in anomaly classification. Meanwhile, there exists a large distribution discrepancy between industrial and natural images. Manual prompts cannot precisely disentangle the representations of domain and category. To this end, we propose a novel few-normal-shot anomaly classification method, dubbed CLIP-FSAC, focusing on leveraging the complementary advantage of cross-modality fusion to solve FSAC task.

Specifically, we adapt CLIP to identify anomalies via contrastive representation learning [Chen et al., 2020]. Because of fewer normal samples, it is difficult to fine-tune image and text encoders of CLIP. Unlike learnable prompt in CoOp [Zhou et al., 2022], we use two adapters to produce the target-oriented representations from image and text encoders of CLIP. To better transfer CLIP for FSAC, we propose a two-stage training strategy to train image and text adapters with normal and synthetic abnormal samples. Firstly we only optimize image adapter by our proposed fusion-text contrastive loss between adapted language-compatible visual features and text features from text encoder. After training image adapter, we add text adapter following text encoder and optimize it by similar contrastive restraint with frozen image adapter. Image-to-text cross-attention module between visual and text features is introduced to obtain visual-driven text features. Combining visual-driven text features with visual features can facilitate a better language-compatible visual representation. Then we exploit interaction between fused features and text features instead of using pure image features. Our main contributions are as follows:

• We empirically analyze that the designing of text prompts is important in zero-shot anomaly classification and propose CLIP-FSAC, which fully explores the potential of CLIP in few-shot anomaly classification.
• We propose a two-stage training strategy to optimize image adapter and text adapter respectively, which better discovers image-text matching capability of CLIP in few-shot anomaly classification.
• We propose to utilize visual-driven text features to enhance visual representations. Meanwhile, we use fusion-
• text matching task to replace image-text matching in original CLIP.
• Experimental results show superior performance of CLIP-FSAC in few-shot anomaly classification. CLIPFSAC outperforms previous few-shot methods on VisA and MVTEC-AD for 1-shot, 2-shot and 4-shot, even surpassing many full-shot anomaly detection methods.

2 RelatedWork

2.1 Anomaly Detection

Anomaly detection[Akcay et al., 2018] methods focus on unsupervised learning only using normal samples, which can be divided into three categories: reconstruction-based method, embedding-based method, and synthesizing-based method. Reconstruction-based methods [He et al., 2023; Ristea et al., 2022; Guo et al., 2023] employ generative adversarial network (GAN) or diffusion models to identify anomalies through pixel-level comparisons between input images and their reconstructions. Embedding-based methods [Yao et al., 2023; Tien et al., 2023; Bae et al., 2023] find anomalies by embedding features of normal samples into latent space and measuring feature-level deviations. While synthesizingbased methods [Zavrtanik et al., 2021; Liu et al., 2023b; Zhang et al., 2023b] synthesize anomalies on normal images and train network with normal and abnormal samples. To address the scarcity of training samples, several few-shot anomaly detection methods have been proposed recently [Fang et al., 2023]. RegAD uses meta-learning which is a new paradigm to train a category-agnostic anomaly detection model with mounts of samples of multiple other categories. GraphCore uses GNN to extract anomaly-free visual isometric invariant features to construct a memory bank with a few normal images. While we only need few normal samples of target category without memory bank.

2.2 Vision-Language Models

Vision-Language Models (VLMs) [Zhu et al., 2023; Jia et al., 2021] which explore the interaction between textual and visual representation witness significant advancements. VLMs have been successfully transferred in many downstream tasks [Wang et al., 2023; Li et al., 2023], especially CLIP [Radford et al., 2021] trained on a dataset consisted of 400 million image-text pairs which is commonly used pre-trained vision-language models. TCM [Yu et al., 2023] turns CLIP directly for text detection without pre-training. CLIP-LIT [Liang et al., 2023] explores the potential of CLIP for pixel-level image enhancement. DenseCLIP [Rao et al., 2022] applies CLIP in dense prediction task. Recently, some works propose to use CLIP for anomaly detection. WinCLIP [Jeong et al., 2023] uses CLIP for zero-/few-Shot anomaly classification and segmentation without fine-tuning, which designs a set of prompts and proposes Window-based CLIP which divides image into patches to extract dense visual features. Then it detects anomalies via vision-language alignment. But WinCLIP is too dependent on original representation of CLIP, as its architecture is restricted to CLIP. Therefore we aim to exploit the semantic relationships between task-specific images and its associated texts.

3 Method

3.1 Preliminaries: Zero-Shot Anomaly Classification with CLIP

Contrastive language image pre-training (CLIP) can effectively learn joint image-language representations which consists of two encoders, an image encoder f(·) and a text encoder g(·). Given an image, CLIP can find its corresponding language representation. This ability of image-text matching can be directly used in anomaly classification. WinCLIP proposes a zero-shot anomaly classification framework named CLIP-AC illustrated in Figure 2(a). In CLIP-AC, two class prompts, s+ ="normal [o]" and s- ="anomalous [o]" are used to identify anomaly images. [o] is a simple object-level label. For a test image, its anomaly score is defined as consine similarity of image and abnormal text embeddings. To further improve performance, WinCLIP also proposes compositional prompt ensemble (CPE) including combinations of pre-defined lists of state words per label and text templates.

3.2 Overview of CLIP-FSAC

We propose CLIP-FSAC, which further pushes the upper limits of CLIP in anomaly classification task. The overall pipeline is shown in Figure 2(b). Our scheme is built upon CLIP and compositional prompt ensemble with two adapters for distributing representations from original CLIP via twostage training strategy. Before fine-tuning, we generate synthetic anomaly samples based on given normal images. Then we feed positive and negative samples into frozen image and text encoders to obtain visual and text features. In the first training stage, we only use image adapter to transfer visual embeddings and use designed image-to-text cross-attention module to obtain visual-driven text features which are added to transferred visual features in the following. Image adapter and cross-attention module are optimized by fusion-text contrastive loss between cross-modality fused features and text features. In the second training stage, we freeze image adapter and cross-attention module and optimize text adapter using the same loss as in the first stage. Significantly, we only need a few normal images less than 4 for fine-tuning.

3.3 Anomalies Generation

To optimize image and text adapters via contrastive learning, only a few anomaly-free images are not enough as lack of negative samples. Therefore, given k-shot normal samples, we generate synthetic anomaly images as negative samples like data-augmentation in few-shot learning. Due to various distributions of anomalies, we mainly use two methods for generating anomalies to simulate natural sub-image irregularities in two different industrial datasets. The first way to synthesize anomalies is via random perturbation [Cao et al., 2023]. Specifically, we randomly choose some square regions in normal images and replace these regions with random values sampled from a Gaussian normal distribution as shown in Figure 3(a). The second approach proposed by NSA [Schl¨uter et al., 2022] is more complicated than the first method. NSA integrates Poisson image editing and Gammadistribution-based patch shape sampling strategy in anomalies generation process to generate more natural and diverse anomalies as illustrated in Figure 3(b) which are necessary for better performance in testing phase. We denote normal image as x+i and synthetic anomaly image as x-i,j , where (i, j) represents the j anomaly image based on x+i . For each normal image, we will generate more than one anomaly images

3.4 CLIP-FSAC

Adapters

Given a normal and abnormal image pair x+i and x-i,j , the [CLS] token embeddings of image features are denoted as

With normal and abnormal compositional prompts, we compute the average of [EOS] tokens encoded by text encoder g(·) which are treated as text representations to represent the normal and anomalous samples, denoted as T + and T -, T +, T - ∈ R1×C . C is tokens dimension. To better exploit the image-text matching ability of CLIP in anomaly classification, we use two adapters denoted as Af (·) and Ag(·) for image and text adaptation respectively. To avoid loss of prior information of CLIP, we blend original knowledge and adaptive knowledge via residual connection with residual ratio α and β controlling the proportion of the original and adapted representations [He et al., 2016; Gao et al., 2021], AF and AT denote image and text adaptive features which can be written as:

To enhance the visual representations and mine most correlated visual cues, we introduce image-to-text cross-attention module in image adapter to obtain visual-driven text features IT + i . We concatenate AT + and AT - as compositional text features denoted as ψ ∈ R2×C . Then we calculate cross attention between compositional text feature and positive visual embedding as

where Linear(·) is a fully-connected layer. Atten+ ∈ R1×C is positive visual-driven text features. Atten- ∈ R1×C is negative one obtained by the same way of Atten+.

The structure of cross-attention module is illustrated in Figure 4. We add these two visual-driven text features to adapted visual embeddings AF + i andAF - i,j to generate crossmodality fused features instead of pure visual features for normal and abnormal classes, denoted as CF + i and CF - i,j :

We optimize adapters using contrastive representation learning between cross-modality fused features and text features.

Training

CLIP-FSAC contains two-stage of training to optimize image and text adapters respectively. We denote the training set and test set as χtrain including less than 4 normal samples and χtest. In the first training stage, we only optimize image adapter Af(·). Given a normal image x+i ∈ χtrain, we first generate an anomalous image x-i,j as its counterpart sample. With designed text prompts and a pair of positive and negative samples, we train image adapter Af (·) using fusion-text contrastive loss. After training image adapter, we only optimize parameters in text adapter Ag(·) using the same loss as in the first stage. Our proposed fusion-text contrastive loss consists of two parts. The first part is fusion-to-text contrastive loss Lf2t which is calculated as:

s(·, ·) is consine similarity function. The other part is text-tofusion contrastive loss Lt2f . It is calculated as:

Fusion-text contrastive loss Lcon used in our method is a combination of fusion-to-text and text-to-fusion contrastive loss

3.5 Anomaly Classification with CLIP-FSAC

Anomalies Generator is no longer needed during testing and CLIP-FSAC is an end-to-end anomaly classification method. For a test image xtest ∈ χtest, we feed it into CLIP image encoder and adapter to obtain cross-modality fused features CFtest. With specific normal and abnormal text representations AT + and AT -, we can calculate positive score S+ and negative score S- for given testing image xtest as follows:

where τ is a temperature hyper-parameter. Finally, we calculate the final anomaly score as

Notably, we calculate the absolute value of gradients grad = ∣∣∣ ∂AS(xtest) ∂xtest ∣∣∣ ∈ RH×W×3 via back propagation. Then we calculate the mean value of grad along channel dimension as grad map. After visualizing grad map as illustrated in Figure 5, we find that anomalies regions in image xtest contribute more to final anomaly score AS(xtest) than normal regions. This demonstrates cross-modal description ability of CLIP and explains why our method based on CLIP can give a good performance in anomaly classification.

4 Experiments

4.1 Experiment Setup

Dataset

Our experiments are conducted on both MVTEC-AD [Bergmann et al., 2019] and VisA [Zou et al., 2022] datasets. MVTEC-AD includes 15 classes including 5 different texture categories and 10 different object categories with totally 5354 high-resolution images. VisA covers 12 objects such as in 3 domain consisting of 10,821 high-resolution color images.

Evaluation Metrics

Our few-shot anomaly classification performance is evaluated based on three metrics following previous anomaly detection methods: (1) Area Under the Receiver Operating Characteristic (I-AUROC) which is the most commonly used metric for anomaly detection tasks, (2) Area Under the Precision-Recall curve (I-AUPR) which is proposed to address imbalance issues [Zou et al., 2022] and (3) F1-Max which measures F1score for anomaly classification at optimal threshold.

Implementation Details

We utilize OpenCLIP and load their pre-trained checkpoints called LAION-400M [Schuhmann et al., 2021] into image and text encoders like WinCLIP. Meanwhile, we directly use CPE proposed in WinCLIP. In our method, we use random perturbation to generate anomalies on Mvtec AD and NSA on VisA. Our image adapter consists of 2-layer multi-layer perceptron (MLP) and text adapter is composed of one MLP.

We use Adam optimizer with learning rate of 0.0005 for image adapter and 0.0001 for text adapter. Training epoch is set to 100 for all datasets. Our few-shot anomaly classification settings are set to 1-shot, 2-shot and 4-shot. So batch size is set to the number of training samples. α1 and α2 in Equation 3 are both set to 1 for MVTEC-AD and VisA. β1 and β2 in Equation 4 are set 0.1 and 0.9 respectively for MVTEC-AD dataset but both are set 1 for VisA.

4.2 Comparison to the State-of-the-Arts

The results of CLIP-FSAC on MVTEC-AD and VisA datasets are shown in Table 1. We compare our anomaly classification performance with previous state-of-the-art anomaly detection methods. In the few-normal-shot setup, our CLIPFSAC achieves new SOTA of 96.0%, 95.5% and 97.7% of I-AUROC on VisA for 1-shot, 2-shot and 4-shot respectively, improving upon the state-of-the-art WinCLIP by 12.2%, 10.9% and 10.4% on 1-shot, 2-shot and 4-shot considerably. For MVTEC AD dataset, CLIP-FSAC reaches new SOTA of 95.5%, 95.8% of I-AUROC on MVTEC AD for 1shot and 4-shot, surpassing WinCLIP by 2.4% and 0.6%. We note that CLIP-FSAC outperforms almost previous state-ofthe-art anomaly detection methods. To measure the performance of our method comprehensively, we also use two other metrics I-AUPR and F1-Max to demonstrate the classification results except I-AUROC. As shown in Table 1, I-AUPR and F1-Max of CLIP-FSAC also surpass many previous SOTA methods on MVTEC AD and VisA datasets.

4.3 Comparison with Full-Shot Methods

As illustrated in Table 2, we compare our results in few-shot setup with previous full-shot methods on VisA and MVTEC

AD datasets. Though with less normal samples, our classification results are competitive with these full-shot methods. The state-of-the-art method on VisA is DiffusionAD [Zhang et al., 2023a] which can reach 98.8% I-AUROC of using fullshot samples. But I-AUROC of CLIP-FSAC achieves 97.7% which is very close to 98.8% I-AUROC of DiffusionAD. It outperforms most full-shot methods only using four normal training samples on VisA. On MVTEC-AD dataset, we can reach I-AUROC score of 95.8% with 4-shot surpassing many full-shot and few-shot anomaly detection methods.

4.4 Ablation Studies

Effect of two-stage training strategy. Better aligning text and image embeddings is vital for FSAC. There are two approaches to train image and text adapters. The first way is to train image and text adapters simultaneously called joint training strategy. The other way we proposed is to train them separately. After training image adapter, its parameters are frozen and text adapter is trained. We compare the performance of these two training strategies. As illustrated in Table 3, the two-stage training is more effective, improve more than 1% I-AUROC in 1-shot and 4-shot setup on VisA.

Effect of cross attention module. Compare with imagetext matching, we use image-to-text cross-attention module to generate visual-driven text features which are used to enhance image embeddings. Then we optimize adapters via aligning cross-modality and image embeddings. In Table 4, we evaluate the effectiveness of cross-attention module on MVTECAD and VisA. We find that the performance with cross attention module is better on both datasets in any setup. Notably, it improves almost 10% I-AUROC on VisA in 1-shot. This observation suggests that there is a better alignment between cross-modality fused features and text features.

Effect of synthetic anomalies. In ablation studies, we investigate the influence of methods in synthesizing anomalies. Because synthetic anomalies are used for training, we hope that they can simulate realistic anomalies as far as possible. We mainly use random perturbation and NSA to generate anomaly samples. So we use these two approaches for both MVTEC-AD and VisA. The experimental results are shown in Table 5, which indicate that the upper limit of random perturbation is higher than NSA, which surpasses 98% I-AUROC on VisA. But the performance of NSA is better than random perturbation on MVTEC-AD in 2-shot.

Performance comparison of different text prompts. We try to change text prompts in zero-shot anomaly classification based on CLIP. Results vary a lot in different text prompts as shown in Figure 6. CPE means compositional prompt ensemble proposed in WinCLIP. We find that the performance of the sixth group of prompts even surpasses CPE. This demonstrates that it is difficult to design an optimal text prompts.

Visualization of anomaly score distribution. Finally, we visualize the anomaly score distribution of testing samples depicted by histograms. As can be seen in Figure 7, discrepancies between anomaly scores of normal samples and anomaly samples become more and more obvious as training epoch increases. It demonstrates that our CLIP-FSAC is effective and adapts CLIP for better performance in few-shot anomaly classification task.

5 Conclusion

In this paper, we propose a novel fine-tuning framework named CLIP-FSAC to adapt CLIP for few-shot anomaly classification. We use two adapters to embed image and text features into a space for better alignment. Meanwhile, we propose a image-to-text cross-attention module to enhance visual embeddings with visual-driven text features. The adapters and cross-attention module are optimized via two-stage training strategy. Our method achieves state-of-the-art results on VisA and MVTEC-AD datasets, even outperforming many full-shot methods. We believe that cross-modal description ability of the vision-language pre-training model still have a lot of room for improvement in zero-shot anomaly classification. In the future, we will further explore vision-language pre-training model and enhance its alignment capability between texts and images for anomaly classification tasks.

Acknowledgments

This work was supported in part by Guangdong Major Project of Basic and Applied Basic Research under Grant 2023B0303000009 and National Major Scientific Instruments and Equipments Development Project of National Natural Science Foundation of China under Grant 62327808.