GRASP: Enhancing Zero-Shot Multi-Modal Anomaly Detection via Geometric Refinement and Semantic Prompting

1 Introduction

Multimodal anomaly detection plays a pivotal role in industrial quality control, leveraging complementary geometric information to identify structural defects invisible in conventional RGB images [Bergmann et al., 2020; Pang et al., 2022; Xie et al., 2023; Gu et al., 2024]. While essential, the application of such systems is often hindered by the difficulty of collecting sufficient normal training samples in dynamic industrial environments. Such scarcity has motivated a shift toward zero-shot anomaly detection (ZSAD) [Ming et al., 2022;

∗Corresponding Author

(b) Preprocessing Speed

77.8

1.4

Rendering FDE

Figure 1: (a) Comparison of inference performance between GRASP and PointAD under unimodal setting and multimodal setting. (b) Comparison of the proposed FDE module with traditional rendering-based preprocessing.

Jeong et al., 2023; Zhou et al., 2024a], empowered by VisionLanguage Models (VLMs) like CLIP [Radford et al., 2021]. By aligning visual concepts with textual descriptions, these models offer strong generalization capabilities across diverse tasks [Zhou et al., 2022; Rao et al., 2022; Khattak et al., 2023; Kirillov et al., 2023].

While the integration of 3D geometry with zero-shot capabilities shows promise, critical impediments regarding efficiency and feature interaction remain. Firstly, processing 3D data is computationally expensive. Projection-based pipelines are typically employed by state-of-the-art methods [Zhou et al., 2024b; Chen et al., 2023; Bergmann and Sattlegger, 2023], rendering 3D point clouds into 2D images. This process incurs prohibitive overhead, creating a bottleneck for real-time deployment. Secondly, a misalignment exists between global semantic prompts and local anomalies. Standard text prompts describe holistic objects, whereas industrial defects are typically fine-grained and localized. Furthermore, most existing methods primarily rely on external semantic matching to identify anomalies, while overlooking the fact that anomalies are defined not only by semantic inconsistency but also by statistical deviations from their local context.

To systematically resolve these challenges, a unified framework, GRASP (Geometric Refinement And Semantic Prompting), is proposed to enhance zero-shot multimodal anomaly detection while reducing computational cost. As demonstrated in Figure 1(a), compared with PointAD, GRASP consistently achieves superior inference speed and significantly reduced computational cost. In GRASP, a Fre-

FPS

75

50

25

0

quency Domain Enhancement (FDE) module is firstly introduced to bypass the rendering bottleneck, utilizing spectral transformations to directly synthesize depth images. As shown in Figure 1(b), compared with the rendering operation, preprocessing efficiency is improved by over 50× with FDE. To bridge the semantic gap, a Prompt-Conditioned Variational (PCV) module is incorporated for feature refinement, alongside a Dual Cross-Injection Alignment (DCIA) module for bidirectional multimodal fusion. Finally, a Pyramid Anomaly Map Recalibration (PAMR) module is utilized to highlight anomalies through multi-scale statistical consistency. Experimental results on MVTec 3D-AD [Bergmann et al., 2019] and Eyecandies [Bonfiglioli et al., 2022] indicate that GRASP consistently surpasses existing state-of-theart methods across all four metrics. Notably, it yields substantial improvements of 3.0 points in image-level AUROC and 2.7 points in pixel-level AUPRO relative to the previous best method MCL-AD [Li et al., 2025]. Furthermore, ablation study is conducted to demonstrate the effectiveness of each proposed module, and we also provide the universality analysis for the proposed FDE module.

The main contributions are summarized as follows:

• A novel ZSAD framework GRASP is proposed to utilize mathematical transformations in the spectral domain to replace costly 3D rendering.
• The PCV and PAMR modules are introduced to bridge the gap between semantics and local defects through semantic alignment and internal statistical coherence.
• The DCIA module is designed for multimodal fusion, which effectively integrates the geometric feature from depth map and the semantic feature from RGB image.
• Extensive experiments demonstrate that GRASP consistently achieves SOTA performance across different metrics in both multimodal and unimodal settings.

2 RelatedWork

2.1 Zero-shot Anomaly Detection

Traditional 2D anomaly detection typically operates in an unsupervised manner on benchmarks like MVTec AD [Bergmann et al., 2019], relying on representationbased paradigms such as teacher-student networks [Deng and Li, 2022], normalizing flows [Rudolph et al., 2021; Gudovskiy et al., 2022], and memory banks [Roth et al., 2022], or reconstruction-based approaches involving autoencoders [Zavrtanik et al., 2021; You et al., 2022] and diffusion models [Zavrtanik et al., 2022]. While some methods introduce synthetic defects for supervised learning [Li et al., 2021; Liu et al., 2023], their dependency on target-domain training data limits flexibility. To address this, Zero-Shot Anomaly Detection (ZSAD) has emerged, leveraging Vision-Language Models like CLIP [Radford et al., 2021] to align visual and textual features without training samples. Recent works have effectively adapted CLIP for this task: WinCLIP [Jeong et al., 2023] utilizes handcrafted prompts with multi-scale aggregation; AnomalyCLIP [Zhou et al., 2024a] introduces learnable prompt tuning; and AdaCLIP [Sun et al., 2023] and PromptAD [Tian et al., 2023] further enhance generalization via category-aware semantics and negative prompting, respectively. However, these methods remain largely confined to the 2D domain, neglecting geometric anomalies.

2.2 Multimodal Fusion

To transcend the limitations of RGB-only detection, multimodal anomaly detection integrates 3D depth or point cloud data [Bergmann et al., 2022; Bonfiglioli et al., 2022] to capture structural defects. Early fusion strategies adapted 2D architectures to depth inputs, such as AST [Rudolph et al., 2022] for background removal and 3D-ST [Bergmann and Sattlegger, 2023] for local geometric description, while subsequent memory-based methods like M3DM [Wang et al., 2023] and reconstruction pipelines like EasyNet [Chen et al., 2023] focused on hybrid feature fusion. More recently, projection-based methods such as CPMF [Cao et al., 2024], PointAD [Zhou et al., 2024b] and MCL-AD [Li et al., 2025], convert 3D data into multi-view images to leverage 2D pre-trained feature extractors. Despite their effectiveness, these approaches often suffer from high computational overhead due to multi-view rendering or rely on extensive targetdomain training. This motivates our proposed lightweight framework, which directly synergizes RGB and depth features for efficient zero-shot detection without costly rendering processes.

3 Method

3.1 Overview

The overall architecture of the proposed method GRASP is illustrated in Figure 2. Specifically, given paired RGB images and raw depth maps as inputs, the depth stream is firstly processed by the Frequency Domain Enhancement (FDE) module, where spectral transformations are applied to reconstruct high-fidelity representations. Subsequently, hierarchical visual features are extracted from both the RGB and enhanced depth streams via a frozen image encoder. In the textual branch, learnable prompts are projected into global textual context representations by a frozen encoder, whereas for capturing diverse semantic nuances, the Prompt-Conditioned Variational (PCV) module is introduced to map learnable prompts into local textual context representations. Finally, the Dual Cross-Injection Alignment (DCIA) module is designed to unify the multimodal information flows, and the final anomaly score is obtained by combining its output with the global textual context representations. Moreover, the fused features are spatially refined by the Pyramid Anomaly Map Recalibration (PAMR) module to generate the precise pixel-level anomaly map.

3.2 Frequency Domain Enhancement Module

As illustrated in Figure 3, the proposed FDE module is designed to convert raw depth inputs into representations with enhanced structural fidelity. Given depth maps x ∈ RH×W , the processing pipeline comprises two sequential stages. The first stage executes structure-aware local enhancement via gradient-based modulation and Local Contrast Normalization (LCN), while the second stage performs frequency domain stripe attenuation. Through this two-stage refinement, the module effectively preserves depth variations corresponding to physically meaningful geometric structures.

Structure-Aware Local Enhancement

Local geometric variations are first characterized by computing the gradient magnitude along the horizontal and vertical directions, derived as:

, where Gradd(x) means computing gradient along the d direction and a small constant ε is introduced to ensure numerical stability. This gradient magnitude serves as a robust indicator of geometric transition strength, effectively capturing high-frequency structural details such as edges and boundaries.

Building upon the derived gradient magnitude, a nonlinear modulation mechanism is formulated to self-adaptively amplify structural discontinuities. Specifically, a modulation weight map xw is computed as:

, where β > 0 controls the enhancement amplitude and γ > 0 regulates sensitivity to gradient variations. Subsequently, the gradient-aware depth representation xgrad is generated by imposing this weight onto the raw input:

, where ⊙ denotes element-wise multiplication and Norm(·) represents per-sample min-max normalization. This operation effectively highlights structural boundaries while constraining the dynamic range for numerical stability.

While the gradient modulation effectively highlights transitions, the resulting representation remains susceptible to global depth bias, particularly where large-scale object geometries overshadow subtle defect signatures. To mitigate this, the Local Contrast Normalization (LCN) mechanism is introduced to enforce statistical consistency within local neighborhoods. Formally, for each coordinate (i, j) in xgrad, we define a k×k sliding window and compute the local statistics efficiently using average pooling operations:

Based on these statistics, the locally normalized depth map xlcn is derived via a nonlinear mapping:

, where α amplifies local contrast. By standardizing intensity distributions within local windows, this operation effectively decouples fine-grained structural details from low-frequency global variations, thereby enhancing the prominence of potential anomalies.

Frequency Domain Stripe Attenuation

Applying local normalization to continuous geometric transitions inevitably induces periodic stripe-like artifacts, which can be misclassified as anomalies. To mitigate the stripe-like artifacts exacerbated by the preceding stage, the Frequency Domain Stripe Attenuation sub-module is introduced, leveraging the Fourier Transform to isolate and suppress the spectral signatures of these periodic disturbances. To be specific, the enhanced depth map is first transformed into the frequency domain:

, where FFT(·) denotes the Fast Fourier Transform and Shift(·) moves the zero-frequency component to the center of the spectrum, with (u, v) denoting the frequency coordinates relative to the spectral center (uc, vc). The stripe patterns are modeled as narrow-band spectral components concentrated along the principle axes. Their suppression is achieved using a separable Gaussian attenuation mask:

, where r controls the effective bandwidth ratio relative to image dimensions (H,W) and η ∈ (0, 1) determines the attenuation strength. Finally, the clean depth map is reconstructed via the inverse transformation:

, where FFT-1 and Shift-1 denote the inverse operations of their respective forward functions. This frequency domain filtering effectively dampens the energy of enhancementinduced stripes while preserving the low- and mid-frequency components that encode genuine geometric structures.

The proposed FDE module resolves the intrinsic conflict between detail enhancement and noise amplification through a cascade of efficient analytical operations. Unlike complex rendering-based pre-processing or heavy reconstruction networks, our approach relies solely on fast matrix and frequency transformations, thereby offering superior computational efficiency. This lightweight design ensures that the resulting multi-channel representation ˜x enriches the feature space with global, local, and denoised geometric cues, while introducing negligible latency overhead compared to conventional pseudo-color rendering pipelines.

3.3 Prompt-Conditioned Variational Module

To effectively align enhanced geometric cues with high-level semantics, the limitations of static textual spaces which often struggle to capture fine-grained industrial structural patterns must be addressed. Therefore, the PCVmodule illustrated in Figure 4 is introduced to generate visually grounded prompt representations. Formally, utilizing a CLIP-based encoder augmented with E-attn [Salehi et al., 2025], spatially dense visual features V ∈ RN×d are extracted. These features serve to condition the base prompt embedding T ∈ Rt×d , where N and t denote the number of visual patches and prompt tokens, respectively.

Latent Space Encoding

To embed visual context into textual representations, a crossmodal attention mechanism is employed where text embeddings T serve as queries and visual features V as keys/values.

This dynamically aligns semantic tokens with visually relevant regions:

The aggregated representation H captures the prompt distribution explicitly supported by visual evidence, accounting for spatial heterogeneity in anomaly localization.

Subsequently, the encoder maps H to a variational posterior [µ, logσ2 ] = Encoder(H). To ensure differentiability, the latent variable Z is sampled via the reparameterization trick:

Crucially, this formulation acts as a controlled information bottleneck. Unlike rigid feature concatenation, the variational projection filters out task-irrelevant noise, ensuring the final representation remains robust while strictly grounded in visual evidence.

Variational Prompt Generation

To mitigate information loss within the bottleneck, the decoder is augmented with a secondary cross-attention mechanism that re-injects visual cues. The latent variable is refined via

, and subsequently converted into the enhanced token-wise prompt representation T ′ = Decoder( ̂ Z). The training objective balances semantic adaptivity with linguistic stability. A reconstruction loss employing a hinge-based formulation is introduced to control the semantic shift. This allows visually induced modifications only within a tolerance margin m:

This mechanism disentangles directional alignment from intensity consistency, preventing uncontrolled semantic drift. Combined with the standard KL divergence regularization, the total objective is defined as:

3.4 Dual Cross-Injection Alignment Module

Existing multimodal anomaly detection methodologies often rely on naive fusion strategies that fail to account for the inherent modal gap, leading to feature redundancy or the suppression of subtle anomalous cues. To address these limitations, the DCIA module is proposed. To be specific, DCIA first employs a semantic-guided aggregation strategy to refine image-level representations. Subsequently, a symmetric cross-injection mechanism is implemented to bridge the inherent modal gap and facilitate bidirectional information flow, ensuring the fused features are both semantically aligned and structurally enriched. The outputs of DCIA are combined with the global textual context representations (shown in Figure 2), and their cosine similarity is computed to obtain the final anomaly score.

Semantic-Guided Feature Aggregation

To extract discriminative image-level representations while preserving fine-grained details, enhanced prompt tokens T ′ is utilized to guide the aggregation of visual features. Specifically, normal and anomaly text anchors tn, ta are derived from T ′n , T ′a by CLIP text encoder. Subsequently, a preliminary semantic anomaly map Msem is computed via cosine similarity:

Rather than employing standard global average pooling, Msem is utilized as a spatial attention mask to re-calibrate the aggregation process. The aggregated features f for each modality are formulated as:

, where v and V denote the cls token and local feature maps, respectively. This ensures that the global representations fr (RGB) and fd (Depth) are specifically enriched with localized evidence.

Shared Latent Projection

With fr and fd obtained, the DCIA module addresses the modality-specific noise by projecting features into a compact latent manifold via linear layer W:

, where ϕ(·) denotes ReLU activation. This compression forces the model to discard high-frequency noise while retaining essential semantic density. To further bridge the modal gap, a shared transformation U maps these latent vectors into a common semantic subspace:

By sharing U, the network is constrained to align semantically corresponding structures from both modalities into similar regions of the feature space, establishing ur and ud as modality-agnostic priors.

Symmetric Cross-Injection

The core innovation of DCIA lies in its complementary reconstruction phase. Rather than independently recovering features, a symmetric cross-injection strategy is implemented where each modality utilizes the semantic prior of another modality to guide its reconstruction. The updates are synthesized as:

Here, ud acts as a geometric guide for the RGB stream, while ur provides texture context for the depth stream. This forces the network to "fill in" the information bottleneck by mining complementary cues from the paired modality. Finally, to preserve the original signal fidelity and prevent catastrophic forgetting, these cross-enhanced updates are integrated via residual connections:

This formulation ensures that multimodal fusion acts as a constructive refinement, yielding features that are both semantically aligned and structurally enriched.

Based on DCIA, the final image-level anomaly score can be calculated as:

, where sm is obtained by calculating the cosine similarity between the fused features ˜ fm of the m modal and the global text. This weighting mechanism adaptively highlights discriminative modalities while suppressing uncertain ones by calculating the squared deviation of si from 0.5, thereby enhancing the robustness of multimodal fusion. To jointly optimize global anomaly scoring and pixel-wise localization, a hybrid objective function is introduced. During training, the Binary Cross-Entropy Loss (Lglobal) is employed as the sample-level objective to optimize the model for accurate anomaly probability estimation in an end-to-end manner. For the segmentation branch, the Focal Loss and Dice Loss (Llocal) is utilized on Msem. The total training objective is formulated as:

3.5 Pyramid AnomalyMap RecalibrationModule

While the semantic pipeline established by the PCV provides a robust vision-language alignment, exclusive reliance on external textual prototypes remains vulnerable to semantic hallucinations. This pathology occurs when complex but normal geometric textures are misidentified as defects due to the inherent ambiguity of zero-shot descriptions. To resolve this, the PAMR module is introduced as a rigorous internal verification layer that validates semantic predictions through structural statistical consistency, as shown in Figure 6.

Recognizing that industrial defects manifest across diverse scales, a pyramid strategy is employed to process images at multiple resolutions, extracting hierarchical deep representations Fs. Unlike the semantic branch which is constrained by linguistic definitions, this process is entirely data-driven. For each patch feature Fp,s at scale s, its "isolation" in the feature space is quantified by computing the average cosine distance to its k-nearest neighbors within the same image:

, where Ωk(Fp,s) denotes the set of k most similar features in the current map. This metric effectively captures local incoherence: true anomalies naturally emerge as statistical outliers relative to the dominant normal patterns, resulting in a scale-invariant recalibration mapM.

The final decision is synthesized by fusing the external semantic prediction Msem with the internal statistical evidence M through an ensemble strategy:

, where λ is a balancing factor. The core innovation of this formulation lies in the geometric term √Msem ⊙M, acting as a strict consistency gate. In contrast to standard arithmetic summation, which allows high-confidence semantic errors to persist, the geometric term necessitates simultaneous high activation from both semantic and structural sources. If a region is semantically ambiguous but statistically coherent with its surroundings, the geometric term forces the final score towards zero. This dual-evidence mechanism ensures that detections are not only semantically grounded but also statistically validated, effectively suppressing false positives and enhancing the reliability of zero-shot inspection.

4 Experiments

Datasets. GRASP is trained on the dataset MVTec 3D-AD and evaluated on the dataset Eyecandies, each containing 10

object categories with corresponding RGB and depth information, to simulate scenarios of unseen categories. MVTec 3D-AD captures real-world industrial anomalies with structured scenes, while Eyecandies features more diverse and colorful object types, providing a challenging testbed for zeroshot generalization.

Metrics. Following the evaluation protocol in PointAD [Zhou et al., 2024b], two sets of standard metrics are employed: I-AUROC and I-AP for global image-level anomaly detection, and P-AUROC and P-AUPRO for local pixel-level anomaly segmentation. Since GRASP utilizes depth maps rather than raw point clouds, all metrics adhere to the RGB-based evaluation protocol to ensure methodological consistency.

Baselines. To comprehensively evaluate the effectiveness of the proposed GRASP, extensive comparisons are conducted against a wide range of state-of-the-art methods. Specifically, for zero-shot evaluation, representative CLIP-based approaches are selected, including PointCLIP [Zhu et al., 2023], AnomalyCLIP [Zhou et al., 2024a], FAPrompt [Zhu et al., 2025], Crane [Salehi et al., 2025], PointAD [Zhou et al., 2024b], and MCL-AD [Li et al., 2025]. Furthermore, to establish rigorous performance boundaries, prominent methods M3DM [Wang et al., 2023], ShapeGuided [Chu et al., 2023], CFM [Costanzino et al., 2024] and CIF [Lin et al., 2026] are included under few-shot setting as references. Notably, these few-shot models are evaluated under a challenging 2-shot setting to serve as a competitive baseline. All comparisons are performed across both multimodal and unimodal benchmarks to verify the versatility of the framework.

Implementation Details. The pre-trained CLIP model from open clip is adopted as both the visual and text encoder, with all encoder weights frozen. Following AnomalyCLIP [Zhou et al., 2024a], the number of learnable tokens in object-agnostic prompt templates is set to w = 12. Both RGB and depth inputs are resized to 336 × 336 before being fed into the visual encoder. During training, DCIA is trained for 20 epochs, while the remaining components are trained for 6 epochs. All experiments are conducted on a single NVIDIA RTX 2080Ti GPU using PyTorch 2.0.0.

4.1 Main Results

As reported in Table 1, GRASP establishes a new state-ofthe-art across all evaluated metrics in the multimodal setting, achieving a peak I-AUROC of 84.1% and a P-AUPRO of 88.1%. These results represent a significant advancement over the previous leading method, MCL-AD, with substantial improvements of 3.0 points in image-level detection and 2.7 points in pixel-level localization. Furthermore, in the unimodal scenario of depth maps, GRASP continues to demonstrate exceptional robustness even when color or texture information is missing. To be specific, GRASP achieves an I-AUROC of 73.0% and a P-AUPRO of 72.5%, widening the performance gap against the strong baseline PointAD by 3.5 and 1.1 points, respectively.

These consistent improvements across diverse settings validate the synergistic effectiveness of the proposed modules. At the foundational level, FDE establishes a high-fidelity geometric baseline by bypassing traditional rendering artifacts, ensuring the visual encoder receives denoised and structurally accurate features. This high-quality input allows the PCV module to effectively address the limitations of static prompts by capturing diverse semantic nuances, which significantly enhances the model's generalization capabilities across unseen industrial categories. To integrate these heterogeneous signals, the DCIA mechanism facilitates robust bidirectional interaction between RGB and geometric streams, effectively resolving semantic misalignment through symmetric crossmodal alignment. Finally, the PAMR module serves as a critical internal verification gate that utilizes multi-scale statistical consistency to suppress semantic hallucinations, ensuring that anomaly predictions are not only semantically grounded but also statistically distinct from the dominant normal patterns of the object.

4.2 Ablation Study

The contribution of each individual component is systematically evaluated in the ablation study, as presented in Table 2. Starting from the baseline, the proposed modules are progressively incorporated, revealing a synergistic evolution in performance. Specifically, with the integration of FDE, a substantial improvement is observed, particularly in the I-AUROC and I-AP metrics, which establishes a robust foundation for feature extraction. Subsequently, the pixellevel localization capability is significantly enhanced by the PCV module, yielding sharp increases in P-AUPRO and PAUROC. This localization precision is then further refined by the addition of the PAMR module. Furthermore, an alternative ablation setting excluding PAMR highlights the independent impact of the DCIA module, which elevates imagelevel anomaly detection metrics by precisely aligning the geometric cues from the depth stream with the semantic prompts from the RGB stream. Ultimately, the integration of all four modules demonstrates that the proposed components are not merely additive but highly complementary, collectively forming a unified system for high-precision anomaly detection.

4.3 FDE Universality Analysis

Given that low-fidelity geometric inputs act as a universal bottleneck in multimodal anomaly detection, the FDE module is utilized to rectify data and improve ZSAD performance. To demonstrate its architecture-agnostic universality, FDE is integrated into the AnomalyCLIP and FAPrompt frameworks as a plug-and-play booster. As shown in Table 3, transformative improvements are achieved across all baselines, such as significantly increasing the unimodal P-AUPRO of AnomalyCLIP from 38.7% to 60.3% and the multimodal I-AUROC of FAPrompt from 63.8% to 69.6%. These results substantiate that the FDE module effectively captures subtle structural features, establishing it as a universal enhancement for zeroshot anomaly detection.

5 Conclusion

In this paper, a unified and robust framework GRASP is proposed to overcome the critical challenges of computational inefficiency and semantic gap in zero-shot multimodal anomaly detection. By transitioning from heavy 3D rendering to efficient spectral transformations via the FDE module, a preprocessing speedup of over 50x is achieved while maintaining superior structural fidelity. To mitigate the semantic gap between textual prompts and local visual details, the PCV module enables visually grounded prompt refinement, thereby enhancing the generalization capability of the framework. Furthermore, the DCIA module strengthens multimodal fusion by dynamically integrating bidirectional crossmodal information flows, ensuring robust and consistent feature alignment across modalities. Moreover, the PAMR module improves anomaly localization reliability by suppressing semantic hallucinations through multi-scale internal statistical consistency validation. Extensive experiments on the MVTec 3D-AD and Eyecandies datasets demonstrate that GRASP establishes a new state-of-the-art, providing a resilient and high-precision solution for real-world industrial inspection tasks.

Acknowledgements

This research is supported by the Fundamental Research Funds for the Central Universities under Grant No. 2021QN1075.