Towards VLM-based Hybrid Explainable Prompt Enhancement for Zero-Shot Industrial Anomaly Detection

1 Introduction

Industrial Anomaly Detection (IAD) aims to accurately identify and locate industrial product defects by advanced computer vision models and algorithms, which improves production efficiency and guarantees product quality. This is significant in promoting industrial automation to a higher level of development.

∗Corresponding authors.

Given the scarcity of industrial anomaly samples in the real world, recent works have focused on how to achieve industrial anomaly detection by unsupervised and zero-shot methods. Unsupervised Industrial Anomaly Detection (UIAD) methods utilize normal samples to learn the distributional features of normal industrial products, which in turn identifies the differences between the input image and the normal image. In this way, the detection and localization of industrial anomalies are mainly achieved through reconstruction [Huang et al., 2021b; Huang et al., 2022a] and feature embedding [Huang et al., 2021a; Huang et al., 2022b]. However, these methods usually require numerous normal samples for training to adequately learn the distributional features of normal data, which may be difficult to satisfy in real industrial scenarios. Moreover, the UIAD methods are limited in their effectiveness in detecting unseen anomalies, making it difficult to accurately identify novel anomalies with small differences from seen normal samples [Wang et al., 2025b; Wang et al., 2025a]. The zero-shot industrial anomaly detection methods are dedicated to achieve anomaly detection without training data on objects of the target category.

Inspired by the strong generalization ability of VLM on various downstream tasks [Sun et al., 2024], VisionLanguage Models (VLMs) have achieved growing interest in ZSIAD. WinCLIP [Jeong et al., 2023] proposed the first VLM-based method for ZSIAD, combining state words and text prompt templates to achieve zero-shot anomaly classification and segmentation. It extracted and aggregated multiscale spatial features aligned with language for efficient feature alignment [Huang et al., 2024; Huang et al., 2025].

Although these VLM-based ZSIAD methods show great promise, it is difficult to further improve the performance due to the lack of target-specific prompts for unseen anomalies. Specifically, we consider the following two perspectives as critical limitations to the accurate classification and localization of unseen anomalies by recent VLM-based ZSIAD methods: 1) Unpredictability and Complexity of Unseen Anomalies. Industrial anomalies in the real world tend to be highly uncertain and diverse, which makes the model rely on additional prompts for the accurate identification and localization of unseen anomalies, as shown in Figure 1 (a). AdaCLIP [Cao et al., 2025] introduced learnable hybrid prompts that allow the model to better recognize unseen anomalies, which consist of text prompt templates and learnable dynamic prompts. This dynamic prompt is usually generated in real time by the image encoder and linear projection layer for each test image to enhance the model's adaptability to different kinds of anomalies. However, the learnable prompts are difficult to capture the details of unseen anomalies, leading to models that may not be able to accurately identify and localize unseen anomalies. It naturally brings the second issue: 2) High annotation costs for detailed anomaly prompts. Whether it is during training or inference, integrating detailed prompts can effectively improve the model's ability to identify and localize unseen anomalies. In practice, obtaining high-quality anomaly descriptions often faces numerous challenges, such as the scarcity of anomalous samples, the imbalance of categories, and the complexity of the annotation process. These factors lead to a significant increase in the cost of labeling anomaly data. AnomalyGPT [Gu et al., 2024b] provides the model with more adaptability and accuracy in industrial anomaly detection tasks by generating the simulated anomalous regions and anomaly descriptions, as shown in Figure 1 (b). Specifically, it generates simulated anomalous regions on normal samples by image editing and generates detailed text descriptions for them as prompts, which alleviates the problem of high annotation cost to some extent. However, these methods rely on simulated anomalies, which may lead to poor generalization ability to industrial anomalies in real applications. Therefore, how to obtain reliable and detailed prompts in real-world datasets at low cost has become an urgent problem. In addition, how to fully utilize these prompts for industrial anomaly detection is also a worthwhile and challenging problem.

Facing the above issues, we attempt to address them from the following two perspectives, as shown in Figure 1 (c): (1) proposing a multi-stage prompt generation agent based on MLLM; (2) proposing an explainable prompt enhancement framework that can fully utilize the generated prompts for facilitating the understanding and detection of unseen industrial anomalies in VLM-based methods. Specifically, we first propose a multi-stage prompt generation agent for ZSIAD which can obtain semantic-rich detailed text prompts and attention prompts from unseen industrial samples. On the one hand, we compare the test samples with normal samples to obtain preliminary detailed anomaly information. To address the intrinsic illusion problem of MLLM, we obtain accurate and semantic-rich anomaly descriptions through further refinement and anti-false alarm constraints. On the other hand, we achieve anomaly-related attentional embedding through the Vision Foundation Model (VFM)-based Anomaly Feature Selection (AFS) module, which can facilitate the model to focus on potential anomalous regions in the foreground. Building on this foundation, we propose the Hybrid Explainable Prompts Enhancement (HEPE) framework, which jointly utilizes generated detailed text and attention prompts for further enhancing the performance of VLM-based methods in ZSIAD. The synergistic effect of hybrid explainable prompts allows maximizing the utilization of these prompts.

Therefore, our contribution can be summarized as follows.

1. We propose a multi-stage prompt generation agent. It can leverage MLLM and VFM to generate semantic-rich text prompts and attention prompts for unseen anomalies, thus providing specific anomaly prompts for VLMs at low cost.
2. We propose a hybrid explainable prompt enhancement framework to further improve the performance by jointly utilizing explainable generated text prompts and attention prompts in ZSIAD, which can enhance the model's understanding and detection of unseen anomalies.
3. We have conducted extensive experiments on seven industrial anomaly detection datasets. The experimental results show that our method has significant advantages over the recent state-of-the-art ZSIADmethod.

2 RelativeWork

2.1 Industrial Anomaly DetectionMethods

The goal of industrial anomaly detection is to accurately identify and localize abnormal patterns or defects that deviate from normal operating conditions during the production process, thereby improving production efficiency, reducing losses, and ensuring system stability. According to the type of supervision, it can be divided into three categories: semisupervised methods, unsupervised methods and zero-sample

(a) Multi-stage guided anomaly description pipeline for ZSIAD

(b) Attention agent-based Vision

Figure 2: The proposed Multi-stage guided anomaly description pipeline and Attention agent-based VFM.

methods. Unsupervised IAD methods [Carrat`u et al., 2023; Wang et al., 2024a] detect industrial anomalies that deviate from the normal pattern mainly by learning the feature distribution of normal samples. The advantage of unsupervised methods is that they do not require anomalous samples for training and are suitable for industrial scenarios with scarce anomalous data. Semi-supervised Industrial Anomaly Detection (IAD) methods [Dong et al., 2024] leverage a small number of labeled abnormal samples and a large number of unlabeled normal samples for training, thereby enhancing the model's ability to identify anomalies while reducing the dependence on large-scale labeled data.

Although these methods have achieved a promising performance, they often lack generalization ability when detecting unseen industrial anomalies. Therefore, an increasing number of studies [Zhu et al., 2024; Gu et al., 2024a] are focusing on zero-shot IAD methods. Since the pre-trained vision-language models (VLMs) naturally have strong generalization ability to various types of targets in the real world, they show significant advantages in ZSIAD. By leveraging vision-language alignment and transferring knowledge from pre-trained models, VLM-based ZSIAD methods [Chen et al., 2024] can better understand and identify unseen industrial anomalies in real-world scenarios. Existing VLM-based methods typically guide anomaly detection through learnable prompts or manually designed text prompts. For instance, WinCLIP [Jeong et al., 2023] guides anomaly detection by constructing text descriptions specifically tailored for the "normal" and "abnormal" categories. However, learnable prompts have limitations in capturing the fine-grained details of unseen anomalies. These prompts usually rely on limited labeled data and predefined text templates, making it difficult to accommodate the diversity and complexity of the real-world anomalies. Therefore, how to further improve the adaptability and detection accuracy of VLM-based models for unseen anomalies remains a challenge to be solved.

In this paper, we provide an innovative solution for ZSIAD through the proposed multi-stage prompt generation agent. The agent combines the powerful generalization capabilities of the pre-trained MLLM with the multi-stage prompt strategy to generate semantic-rich text prompts and attention prompts from unseen industrial anomalies.

2.2 Prompt Learning

Prompt learning is a technique for augmenting model inputs with task-specific prompts, thereby adapting pre-trained large models to downstream tasks. Specifically, model inputs can be augmented by well-designed prompts, which mainly include soft and hard prompts. Soft prompts [Jia et al., 2022; Ren et al., 2023] are learnable continuous vectors that are usually optimized for a specific dataset by gradient optimization methods. However, soft prompts have the disadvantage of lacking explainability and tend to perform worse than highquality hard prompts [Wang et al., 2024c]. Hard prompts are natural language instructions [Dong et al., 2022], which have the advantage of being flexible and explainable but designing high-quality hard prompts requires a lot of manual effort.

Recently, several studies [Deng et al., 2024; Qu et al., 2024] have explored the application of prompts to VLMbased IAD methods. WinCLIP [Jeong et al., 2023] have achieved promising performance using learnable soft prompts in IAD. In addition, some studies [Cao et al., 2025; Zhou et al., 2024] have jointly used both soft and hard prompts, improving the performance of ZSIAD by combining the advantages of both. However, further performance improvement is limited by the lack of high-quality hard prompts. In this paper, we generate high-quality text prompts using the multistage prompt generation agent. This agent allows the model to better capture anomalous features while maintaining the model's ability to generalize to unseen anomalies.

3 Method

In this section, we will introduce the proposed hybrid explainable prompt enhancement method and its important components in detail, which mainly include (1) Overview; (2) Multi-stage Prompt Generation Agent; (3) Hybrid Explainable Prompt Enhancement Framework.

3.1 Overview

As shown in Figure 2, we first use the multi-stage prompt generation agent to generate semantic-rich text and attention prompts for unseen anomalies, thereby providing specific anomaly prompts for the model at low cost. For the text prompts, we input normal and test samples into the elaborated MLLM in the first step so that it describes in detail the differences between the test samples and the normal samples. To further obtain anomaly-related information as well as to prevent potential false alarm possibility, we refine the generated detailed anomaly descriptions under the anti-false alarm constraint in the second step to obtain accurate and anomalyrelated text descriptions. In addition, to make the model pay more attention to the anomaly information in the foreground, we obtain anomaly-related attention prompts through the VFM-based anomaly feature selection module.

As shown in Figure 3, the proposed hybrid explainable prompts enhancement framework mainly consists of the visual branch, textual branch, and attention prompts branch. The proposed CLIP-based framework primarily explores the utilization of text and attention prompts for enhancing the model's ability to detect unseen anomalies. In the visual branch, we input the test industrial images into the pre-trained CLIP [Radford et al., 2021] image encoder to extract patchlevel visual embeddings. In the textual branch, we input the generated text prompts representing normal and abnormal respectively into the pre-trained CLIP text encoder to obtain text embeddings. Combining the dynamic and static prompt generators from previous work [Cao et al., 2025], we obtain anomaly mapping by calculating the similarity between visual embeddings and textual embeddings. In addition, we combine the hybrid semantic fusion module (HSF) and generated attentional embeddings for facilitating the model to focus on anomalous regions in the foreground in order to enhance the accuracy and generalization capability of imagelevel anomaly detection.

3.2 Multi-Stage Prompt Generation Agent

For VLM-based ZSIAD methods, high-quality text descriptions of anomalies are essential to prompt the model for understanding and detection of unseen anomalies. However, existing methods either rely on predefined text templates or are annotation-costly. In addition, attention prompts can enhance the model's ability to focus on anomalous regions. Therefore, we propose the multi-stage prompt generation agent for generating semantic-rich text and attention prompts.

Text Prompt. To fully leverage the potential of MLLM in describing anomalies, we first input normal and test sample to MLLM in the first step, and the differences between the two samples are described in detail through MLLM in order to mine potential anomaly details. Subsequently, we bootstrap the process of refining anomaly details by introducing category labels in the second step, which aims at generating anomaly descriptions that are as relevant as possible to specific industrial products. It is noteworthy that LLMs are prone to misidentifying normal differences as abnormal regions due to the potential problems of illusion and misinterpretation. For instance, LLMs may misclassify nuts that are oriented differently from the normal samples as anomalies. Therefore, we further set the anti-false alarm constraint, which is mainly used to prevent LLMs from describing normal differences as abnormal.

Attention Prompt. To enhance the model's ability to focus on anomalous regions, we utilize the VFM and anomalous feature selection module to generate attentional prompts. Specifically, we first input the test industrial images into the VFM image encoder for attentional feature extraction. Then, the anomaly feature selection module can select the most useful regions for anomaly detection as attention prompts from the attentional features. which mainly contains Omnidimensional Dynamic Convolution (OD Conv) [Li et al., 2022] and Depthwise Separable Convolutions (DS Conv) [Chollet, 2017]. OD Conv significantly improves the dynamics of convolutional operations and feature extraction through the multidimensional attention mechanism that can dynamically enhance attention to the anomalous regions. DS Conv can achieve higher performance with a lower parameters through the property of deeply separable convolution.

3.3 Hybrid Explainable Prompt Enhancement

Due to its outstanding zero-shot generalization capability, the vision-language classic framework CLIP has achieved tremendous success across various domains of computer vision. Building upon prior CLIP-based ZSIAD methods [Cao et al., 2025], we propose a Hybrid Explainable Prompts Enhancement that combines generated text descriptions and attention prompts for industrial anomaly detection. The following are each component of the proposed framework.

Static andDynamic Prompts. To improve the ZSIAD performance effectively using the proposed hybrid explainable prompts, we introduce static and dynamic prompts. The static prompt PS serves as a foundational learning cues shared across all images. Furthermore, we introduce dynamic prompts PD, allowing the semantic information contained in the hybrid prompts to be effectively integrated into the CLIP semantic space. PD differs from PS in that they are generated by the prompt generator G for each test image individually. In the proposed method, the composition of G includes class tokens from the CLIP image encoder and a learnable projection layer. This projection layer is used to map class tokens into dynamic prompts PD.

Hybrid Semantic Fusion Module (HSF). Previous anomaly detection methods [Chen et al., 2023] select the maximum value of anomalies as the anomaly score, which is sensitive to noise predictions. To enhance the robustness of the model and the reliability of the detection results, we introduce the HSF module [Cao et al., 2025]. It is adopted to aggregate patch embeddings that are more likely to represent anomalies, which effectively consolidate regionlevel information for robust image-level anomaly detection. The HSF output integrates the semantic information most relevant to anomalies. Combined with the proposed hybrid explainable prompts, it can incorporate even more anomalyrelated semantic information. Compared to the previous max-value-based anomaly detection methods, the resulting semantically enriched vision patch embeddings effectively improve image-level anomaly detection performance.

4 Loss Function

The pixel-level anomaly map in the proposed method is derived by calculating the cosine similarity between patch embeddings QE with integrated attention prompts and text embeddings TA and TN with integrated text prompts. The anomaly map is defined as follows:

where ψ is an interpolate function. The pixel-level anomaly map Mi in i-th block is resized to match the dimensions of the test image. Anomaly maps are extracted from multiple blocks [24] and aggregated to produce the final anomaly map M. In the training process, dice loss LD and focal loss LD is jointly applied as the object function, which is defined as follows:

where α is the balancing factor, γ is the hyper-parameter that adjusts the weight of easily classified samples, pi is the anomaly scores predicted by the model, and gi is the ground truth label. The final object function L is defined as follows:

5 Experiments

5.1 Experiment Setup

Datasets. We conduct experiments using the seven industrial anomaly detection datasets for all experiments: MVTec AD [Bergmann et al., 2021], VisA [Zou et al., 2022], MPDD [Jezek et al., 2021], BTAD [Mishra et al., 2021], KSDD [Tabernik et al., 2020], DAGM [Wieler and Hahn, 2007], and DTD-Synthetic [Aota et al., 2023].

EvaluationMetrics. Consistent with prior ZSIADmethods [Cao et al., 2025], we adopted Area Under the Receiver Operating Characteristic Curve (AUROC) and maximum F1 score (max-F1) to evaluate the anomaly detection performance of both image-level and pixel-level. Beyond the dataset-specific results, we further reported the mean performance across all datasets. This result is quantified using AUROC and max-F1.

Implementation Details. We adopted QWen2-VL-72B [Wang et al., 2024b] to generate detailed descriptions of the anomalies. Furthermore, QWen2.5-7B [Yang et al., 2024] is utilized to extract anomalous information and judge the presence of anomalies. The pre-trained CLIP (ViT-L/14@336px) [Radford et al., 2021] is employed as the backbone for subsequent ZSIAD models, extracting patch embeddings from the 6th, 12th, 18th, and 24th ViT blocks. DINOv2 (ViT-S) [Oquab et al., 2024] is adopted as the VFM. We trained the proposed method for 5 epochs with a learning rate of 0.01. All experiments were performed with a single NVIDIA A100 GPU (80GB).

Comparison Methods. We compare the proposed method with two baselines: with and without training. Specifically, SAA [Cao et al., 2023] and WinCLIP [Jeong et al., 2023] are adopted as our non-trained baselines. For trained baselines, we selected APRIL-GAN [Chen et al., 2023] and AdaCLIP [Cao et al., 2025]. Following the previous work, we trained DINOv2 [Oquab et al., 2024] and SAM [Kirillov et al., 2023], which appends additional linear layers to multiple Transformer layers as a task head for anomaly detection.

5.2 Comparison with State-of-the-Art Methods

Table 1 presents the results of our comparative experiments, demonstrating the superior performance of the trained methods over the untrained ZSIAD baselines. WinCLIP and SAA with the manual text prompts exhibit inferior performance. It is noteworthy that fine-tuning the visual foundation models (DINOv2 and SAM) can achieve satisfactory pixel-level ZSIAD. The impressive performance of VFM-based ZSIAD methods fine-tuned on domain-specific data underscores the fact that pre-trained VFMs have essential knowledge for the anomaly. With minor adaptations, this foundational knowledge can be effectively leveraged for ZSIAD tasks.

Compared to other ZSIAD methods, the proposed method exhibits notable performance enhancements. For example, compared to the sub-optimal method AdaCLIP, the proposed method achieves 2.7% improvement in image-level AUROC and 0.9% improvement in max-F1. Moreover, the pixellevel AUROC reached 95.1% and the max-F1 score attained 51.0%. These results underscore the superiority of the proposed method and validate the efficacy of incorporating textenhanced and attention-based prompts. Furthermore, we visualize the anomaly detection results across different datasets in Figure 4. The proposed method shows notably more precise pixel-level anomaly detection for unseen classes than other methods. This best pixel-level performance can be attributed to the detailed anomaly information provided by textenhanced prompts and the more discriminative visual features captured by attention prompts.

text prompts. In other words, the joint application of the pro-

posed strategies mitigates the noise in generated text prompts.

On the one hand, LLM can correct descriptions that mistak-

enly judge differences in location as errors by applying the

anti-false positive strategy. It allows subsequent ZSIADmod-

els to obtain accurate and semantically rich descriptions of

anomalies. On the other hand, MLLM has a clear reference to

exclude misjudgments caused by the specific attributes of the

objects themselves (such as describing sesame-like spots on

sugar as anomalies), thereby outputting accurate anomaly in-

formation. Consequently, the combined application of these

strategies minimized noise in text prompt generation, result-

ing in optimal model performance.

SAA WinCLIP DINOV2 APRIL-GAN AdaCLIP

Figure 5: Ablation study of different VFM for attention prompts generation. 'I' and 'P' means the Image-level and Pixel-level, respectively.

5.3 Ablation Study

In this subsection, we conduct ablation experiments on the all datasets to validate the effectiveness of different strategies. We quantitatively evaluated the effectiveness of the proposed text and attention prompts by different combinations. The results of the ablation study are shown in Table 2. Compared to the baseline, each combination achieves a better performance. Therefore, we can conclude that both strategies contribute to the ZSIAD performance improvement. In addition, the final ablation study results demonstrate that the joint application of the proposed strategies can further result in greater performance gain, proving the effectiveness and rationality of the proposed strategies.

5.4 Qualitative Analysis

Text Prompt Generation Strategy. Table 3 presents the results of ablation experiments on various text prompt generation strategies. The experimental results clearly demonstrate that the model achieves optimal performance by contrasting anti-false alarm with normal samples in generating Our Different VFM for Attention Prompt. We explored the impact of different VFM-generated attention cues on the proposed ZSIAD method, with results reported in Figure 5. The experimental results clearly shows that using DINOv2 (ViTS) to generate attention prompts achieves the optimal ZSIAD performance for the proposed method. The pixel-level performance achieved with SAM and its variants EfficientSAM [Xiong et al., 2024] remains suboptimal. We believe this is due to DINOv2 being a self-supervised VFM, aimed at learning general-purpose visual features. In other words, DINOv2 extracts discriminative and semantic-rich features from extensive image datasets through a self-supervised training method. These feature representations are essential for ZSIAD tasks. SAM and its derivatives are primarily designed for interactive segmentation, demonstrating strong performance guided by user-provided prompts. SAM exhibits a weaker overall semantic understanding of complex scenes in scenarios without clear prompts.

6 Conclusion

In this paper, we present a novel multi-stage prompt generation agent for ZSIAD to generate semantic-rich text prompts and attention prompts for unseen anomalies. Furthermore, we propose the Hybrid Explainable Prompt Enhancement method, which maximizes the potential of these prompts to enhance the model's understanding and detection of unseen anomalies. Extensive experiments on the widely-utilized benchmarks have validated the effectiveness of our method.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 62371404, in part by the National Natural Science Foundation of China under Grant 62301621, in part by Shenzhen Science and Technology Program (No. 20231121172359002), in part by Shenzhen General Research Project (No. JCYJ20241202125904007), and in part by Guangdong Basic and Applied Basic Research Foundation (No. 2025A1515011398).