Omni-Dimensional State Space Model-driven SAM for Pixel-level Anomaly Detection

1 Introduction

Pixel-level anomaly detection [Huang et al., 2022d; Huang et al., 2024b] is a technique that precisely identifies anomaly pixels in images, which has emerged as an indispensable tool for industrial defect detection and medical diagnosis. However, its reliance on the expertise and proficiency of individual analysts poses a significant challenge. Consequently, it is necessary that develop an automated method to replace the previously labor-intensive and technically demanding manual detection processes [Huang et al., 2022b].

∗Corresponding author

Deep learning models have emerged as promising tools for pixel-level anomaly detection due to their ability to learn complex image features. However, the widespread prevalence of Pixel-level anomaly detection algorithms tailored to specific imaging modalities or objectives hinders progress. While the specialization of existing methods can achieve high accuracy in controlled settings, it comes at the cost of limited adaptability and generalizability across diverse anomaly scenarios [Mamonov et al., 2014]. Vision Foundation Models (VFMs) have captivated the attention of researchers due to their remarkable generalization capabilities across diverse downstream tasks. Among these, the Segment Anything Model (SAM) [Kirillov et al., 2023] has emerged as a pioneering contribution to image segmentation due to the widespread acclaim for its ability to generate precise object masks in both fully automated and interactive scenarios. However, pixel-level anomaly detection, a critical subfield of image segmentation, poses unique challenges that may impede the performance of SAM in this field. This stems from the inherent disparities between natural images and medical images. Some works [Li et al., 2025; Liu et al., 2024] have demonstrated that the direct application of the SAM to pixel-level anomaly detection tasks results in unsatisfactory performance. Therefore, it's necessary to perform specific improvements and optimizations to SAM so that improve the performance and generalization ability of the model in pixel level anomaly detection tasks.

Moreover, the work of [Cai et al., 2024] has shown the significance of multi-scale features for pixel-level anomaly detection. Nevertheless, we found that the original SAM lacks a multi-scale feature extraction component. This is due to SAM being an interactive segmentation model that only requires segmenting objects in a specified region. However, when dealing with an image containing numerous anomaly targets, performing interactive segmentation for each target individually becomes inefficient. Consequently, some methods [Xie et al., 2024; Shaharabany et al., 2023] have been proposed to automatically generate SAM prompts without manual intervention. Nonetheless, these automated prompting approaches merely utilize partial image features as prompts and fail to incorporate crucial features such as multi-scale image features to generate more suitable prompts. This limitation results in the generated prompts that cannot fully capture anomalous targets in the image, thereby affecting the accuracy and robustness of anomaly detection. Therefore, the challenge faced in this article is how to design a prompt encoder that can effectively integrate multi-scale features as prompts to improve its ability to detect anomalies in complex scenes.

To address this challenge, we propose a novel end-to-end method for pixel-level anomaly detection in industrial and medical images. Specifically, we propose a novel prompt encoder to generate prompts based on the input image fed into SAM. This contrasts with the rudimentary prompts (point, bounding box, and mask) adopted in the vanilla SAM architecture. We propose a State-Space Model-based residual Omni-Dimensional module (SSMODC) for the prompt encoder. This module can effectively leverage multi-scale and global information, facilitating an iterative search for optimal prompts in the prompt space. The identified optimal prompts are then fed into SAM as high-dimensional tensors. Notably, our method does not alter the SAM, which ensures ease of implementation and avoids finding an optimal training schedule for fine-tuning SAM. Extensive experiments demonstrate that with our proposed prompt encoder, the detection performance of the SAM is significantly improved. It surpasses state-of-the-art models.

The contributions of this work can be summarized as:

1. We propose a novel Omni Dimensional State Space Model-driven SAM (ODS-SAM) for pixel-level anomaly detection. Specifically, ODS-SAM utilizes an innovative prompt encoder to adaptively generate prompts based on input images, effectively integrating multi-scale features. This adaptive prompt generation mechanism significantly improves the quality of prompts and the accuracy of detection.
2. We design a State-Space Model-based residual OmniDimensional module (SSMODC), which effectively utilizes multi-scale and global information to iteratively search for the optimal prompt in the prompt space. The introduction of the SSMODC module enables the model to comprehensively understand and respond to the features of the input image, improving the accuracy and robustness of anomaly detection.
3. Experimental results on medical and industrial image datasets demonstrate that our proposed ODS-SAM outperforms several state-of-the-art methods on the task of pixellevel anomaly detection.

2 RelatedWork

2.1 Pixel-level Anomaly Detection

Pixel-level anomaly detection is an anomaly detection[Zhang et al., 2022; Huang et al., 2021a; Huang et al., 2021b; Wang et al., 2025; Huang et al., 2022c; Huang et al., 2022a; Huang et al., 2024c; Huang et al., 2025] task aimed at accurately identifying abnormal regions in an image, which has attracted attention from both academia and industry due to its broad application prospects. Initial efforts [Bae and Yoon, 2015; Reiss et al., 2021] employed elemental features like texture, color, shape, and appearance, but their performance is often hindered by the inherent limitations of these low-level features. The advent of deep learning [Chen et al., 2020] revolutionized the pixel-level anomaly detection task, and deep neural network-based methods achieve remarkable progress.

Notably, encoder-decoder architecture networks, exemplified by U-Net [Ronneberger et al., 2015] and the enhanced version U-Net++ [Zhou et al., 2018], have demonstrated significant performance gains by leveraging multi-level feature integration to generate high-resolution detection results. Additionally, researchers have further enhanced the effectiveness of anomaly detection models by incorporating supplementary boundary information [Fang et al., 2019]. These methods represent the conventional deep learning approaches. In contrast, the proposed method leverages the powerful generalization and spatial awareness capabilities of the visual foundation model SAM, thus achieving accurate pixel-level anomaly localization and detection without the need for additional information.

2.2 Segment AnythingModel

SAM (Segment Anything Model, SAM) [Kirillov et al., 2023] is a groundbreaking prompt-based image segmentation model. It is trained on the extensive SA-1B dataset. This dataset has an astounding tens of millions of imageannotation pairs. It grants the model exceptional zero-shot generalization capabilities. SAM adopts a Transformer-based architecture, demonstrating remarkable effectiveness in natural language processing and image recognition tasks. Specifically, SAM employs a Vision Transformer (ViT)-based image encoder [Dosovitskiy, 2020] to extract image embeddings, a prompt encoder to integrate user interaction through diverse prompt modes, and a lightweight mask decoder to predict segmentation masks by fusing image embeddings and prompt embeddings. However, existing SAM requires relatively high-quality prompts (i.e., points, boxes, and masks) to achieve satisfactory performance in segmentation tasks. In previous methods, the required prompts are generated from ground-truth labels during testing [Wei et al., 2023]. Nonetheless, creating accurate and reliable prompts still necessitates specific domain expertise, which may not be readily accessible. Moreover, low-quality prompts arising from noisy annotations can significantly deteriorate segmentation accuracy. Consequently, exploring automatic prompt mechanisms aims to establish a robust adaptive framework to alleviate the performance of SAM variability and facilitate more reliable and accurate results across various segmentation tasks. This paper proposes a prompt encoder that generates prompts tailored to the given prompt for SAM input images. Specifically, it extracts multi-scale features, crucial for pixel-level anomaly detection tasks, as prompts for SAM. Additionally, the prompt encoder can identify the optimal prompt mode by optimizing parameters during training.

3 Methodxiangology

3.1 Preliminaries: State Space Model

State Space Model (SSM) originates from continuous systems that transform a sequence x(t) → y(t) through a hidden state function h(t) ∈ RN . It can be expressed as the following equations:

where A ∈ RN×N is a state matrix, and B, C ∈ RN are the projection matrix.

S4 and Mamba [Gu and Dao, 2023] are discrete versions of the aforementioned continuous system, which adopt a time scale parameter ∆A and ∆B that converts continuous parameters A, B into discrete parameters A, B. The zero-order hold is adopted as the discretization rule, defined as follows:

where J is a unit matrix.

After discretizing A and B, Eq. 1 can be written as follows:

Furthermore, a global convolution is adopted to compute the output, defined as:

where L denotes the length of the input sequence and K ∈ RL is a structured convolution kernel.

3.2 Overview

The proposed method is shown in Fig 1. Specifically, it consists of three parts, the SAM encoder EncSAM, the proposed prompt encoder Encp, and the SAM decoder DecSAM. EncSAM is adopted to extract visual features. We freeze for both components during the training phase to maintain the powerful feature extraction capabilities of EncSAM and the remarkable spatial reasoning prowess of DecSAM. Subsequently, the proposed State-Space Model-based residual Omni-Dimensional module is adopted to comprehensively explore the global information of anomalous targets from multiple directions. Moreover, it captures the edge details of anomalies through multi-scale representation. Thus, the prompt encoder can deeply understand anomalous features at different scales while iteratively searching for the optimal prompt in the prompt space, achieving more accurate anomaly detection results. Finally, both prompt and visual features are fed into the SAM decoder to generate the final anomaly mask.

3.3 State-Space Model-based residual Omni-Dimensional module

Previous research has demonstrated that leveraging multiscale representations to capture contextual information can effectively improve edge segmentation accuracy. Furthermore, global information offers more holistic scene understanding, which plays a pivotal role in detecting the overall structure of anomalous objects. However, current pixel-level anomaly detection methods fail to integrate global and rich contextual information containing multi-scale information effectively. Thus it is hard to accurately segment anomalous objects in complex scenarios, particularly while both local details and global context demand comprehensive consideration. To address these limitations, we propose the State-Space Model-based residual Omni-Dimensional module, which encodes these two types of information as prompts and feeds them into the SAMmodel for pixel-level anomaly detection.

Specifically, we adopt the state-space model (SSM) to extract global information. SSM can process sequence data with linear complexity, which significantly improves the efficiency of sequence processing. However, SSM is hard to capture global information due to the fixed scanning direction. Furthermore, SSMs exhibit a propensity to forget previous inputs, causing the model to prioritize recent inputs during processing at the expense of early information in the sequence. This tendency impedes the model's ability to adequately focus on the middle region of images in image segmentation tasks, potentially leading to the oversight of anomalous information. To address these limitations, we introduce the Omni-directional State-Space Model (OD-SSM). OD-SSM improves the extraction of global information by scanning the input sequence in multiple directions, which can simultaneously pay attention to the middle region and corners of the image. Consequently, it can effectively avoid the omission of anomalous information. Additionally, we adopted a depthwise separable convolution (DW Conv) layer before ODSSM to retain local details. It can extract near-neighbor features before modeling global information. Depthwise separable convolution can effectively capture salient features in local regions. Notably, this refinement does not impede the proposed method extract global information, which ensures the integration of local details with global information. As shown in Fig 2, we adopt Omni-Dimensional Dynamic Convolution (OD Conv) [Li et al., 2022] with different dilated rates to extract multi-scale information. OD Conv incorporates a multi-dimensional dynamic attention mechanism to optimize convolution kernel shapes and enhance feature sensitivity. This mechanism empowers convolution kernels to adaptively focus on crucial information within input data, which bolsters the representative capacity of the model. Moreover, the model effectively captures multi-scale features by adopting ODConv with varying dilation rates, which can learn rich representations from local to global contexts. This complements the global information extracted by SSM and provides the decoder with more comprehensive prompts.

3.4 Loss Function

We adopt a composite loss function in training. This function combined the Binary Cross-Entropy (BCE) loss and Dice loss in a weighted manner. This combined function can balance the prediction accuracy and boundary alignment, which improves training efficiency and segmentation accuracy for pixel-level classification tasks.

The BCE and Dice function can be defined as follows:

where N is the total number of samples, yi is the ground truth label for the i-th pixel, and ˆ yi is the predicted probability for the i-th pixel. Additionally, ϵ is a small constant added for numerical stability.

The final combined loss function L is defined as follows:

Based on experiments, we set λ1 = 0.5 and λ2 = 0.5.

4 Experiments

4.1 Dataset

MVTec AD [Bergmann et al., 2020] is adopted to evaluate the proposed method for pixel-level anomaly detection in industrial scenes. It is a large-scale dataset for industrial anomaly detection, which contains 1258 abnormal images in 15 categories. The MoNuSeg dataset [Kumar et al., 2017] comprises 30 microscopy images from 7 organs in the training set, with annotations for 21,623 individual nuclei. To align with prior work, we resized the all images to 512 × 512. The Gland segmentation (GlaS) challenge [Sirinukunwattana et al., 2017] encompasses 85 images for training and 80 images for testing. All images in experiments are resized to 224 × 224. Moreover, four Polyp datasets (Kvasir-SEG [Jha et al., 2020], ClinicDB [Bernal et al., 2015], ColonDB [Tajbakhsh et al., 2015], and ETIS [Silva et al., 2014]) are adopted to evaluate the proposed method. The experiment setup is followed [Fan et al., 2020].

4.2 Implementation Details

During the training, we adopted the Adam with an initial learning rate of 0.001 and a weight decay regularization parameter of 1e-5. The batch size is 10 and trained on an NVIDIA A100 GPU. The maximum number of epochs is 200. SAM pre-trained weights adopted in all experiments is based on ViT-H.

4.3 EvaluationMetrics

To align with prior research, we evaluate the performance of all methods on the MVTec dataset using the following two metrics: mean E-measure (mE) [Fan et al., 2021] and mean absolute error (MAE). Additionally, we assess the performance of all methods on the GlaS, MoNusEG, and four polyp datasets using Dice and IoU [Li et al., 2023] metrics.

4.4 ComparisonMethods

On the medical image datasets, comparison methods include U-Net [Ronneberger et al., 2015], U-Net++ [Zhou et al., 2018], SFA [Fang et al., 2019], MSEG [Huang et al., 2021c], DCRNet [Yin et al., 2022], ACSNet [Liu et al., 2021], PraNet [Fan et al., 2020], EU Net [Patel et al., 2021], SANet [Wei et al., 2021], COMMA [Shin et al., 2022], SAM-EG [Trinh et al., 2024], LViT [Li et al., 2023], and methods from literature [Huang et al., 2024a], Axial Attention [Wang et al., 2020], MedT [Valanarasu et al., 2021], UCTransNet [Wang et al., 2022], 3P-SEG [Shaharabany and Wolf, 2022], MedAdaptor-SAM [Wu et al., 2023]. The comparison methods on industrial images include SFA [Fang et al., 2019], ACSNet [Zhang et al., 2020], PraNet [Fan et al., 2020], and methods from literature [Huang et al., 2022b], AutoSAM [Shaharabany et al., 2023], I-MedSAM [Wei et al., 2023].

4.5 Comparison with State-of-the-Art Methods

Result on MVTec Dataset

As shown in Table 1, the proposed method outperforms all existing SOTA methods on the MVTec AD dataset. Specifically, the proposed method achieves the best mean MAE and mean squared error (mE) results. The method achieves an MAE of 0.009, which outperforms the SOTA method [Huang et al., 2022b] by 18.2%. Moreover, our method increases mE by 5.1% compared to the SOTA method [Huang et al., 2022b]. These results demonstrate that the proposed method can accurately detect pixel-level anomalies on industrial datasets.

Result on Medical Image Dataset

Table 2 and Table 3 demonstrate that the proposed method outperforms other methods across all medical image datasets. Specifically, compared with existing SOTA methode SAMEG, our method achieved 0.6% and 1.4% improvement in Dice and IoU scores on the Kvasir33 dataset, respectively.

Surprisingly, our method achieves significant improvement, with Dice and IoU scores increasing by 3.68% and 5.19% on the MoNuSeg dataset, respectively. These results underscore the superiority of our method for pixel-level anomaly detection in medical images.

4.6 Ablation Studies

Effectiveness of Each Component

Table 4 reports the ablation experimental results about the effectiveness of each component. These results indicate that all of these components are effective for the proposed method. Our ablation study aimed to assess the contributions of ODSSM and multi-scale OD Conv to medical image segmentation. When employed individually, OD-SSM led to modest performance gains on MoNuSeg and GlaS, validating its effectiveness. OD-SSM enhances global context extraction by scanning the input sequence multi-directionally. Simultaneously, it ensures comprehensive image coverage, which enhances the perception ability of anomalies.

Furthermore, the exclusive utilization of multi-scale OD Conv results in substantial performance improvements. It indicates that multi-scale representations can improve edge segmentation accuracy due to the multi-scale feature contains richer contextual information.

The combination of OD-SSM and OD Conv yielded the best performance, demonstrating a synergistic effect. Extracting global context can provide a more comprehensive scene understanding, which improves the ability of the model to parse the overall structure of anomalies. Meanwhile, OD Conv effectively integrated global and multi-scale contextual information, enhancing segmentation accuracy in complex scenarios.

Impact of different OD Conv quantities

Table 5 presents an ablation study for varying numbers of OD Conv. Specifically, the model exhibited optimal performance when equipped with two ODDC convolutional layers. This superior performance can be attributed to the following reasons: firstly, the integration of two ODDC layers could provide a balanced capacity for feature extraction. Secondly, the dual-layer configuration may enhance the model to capture multi-scale and multi-dimensional features, which is essential for the nuanced detection of anomalies within the pixel-level domain. Furthermore, the results suggest that the addition of more than two ODDC layers does not significantly improve the performance of the model, which is due to the diminishing returns on the complexity added to the network architecture.

Impact of Different OD-SSM Quantities

In the ablation study, we further explored the impact of the number of Omni-directional State-Space Models (OD-SSMs) on the performance of pixel-level anomaly detection. The results are shown in Table 6. We constructed model variants with 1, 2, and 3 layers of OD-SSMs on the MoNuSeg and GlaS datasets, respectively. It demonstrates that when the model incorporates a single OD-SSM layer, it achieves Dice coefficients of 84.02% on MoNuSeg and 92.74% on GlaS, corresponding to IoU scores of 72.52% and 87.01%, respectively. These results suggest that a solitary OD-SSM layer is sufficient to capture anomalous features within images and produce reasonably accurate anomaly segmentation. Increasing the number of OD-SSM layers to two resulted in a slight decrease in the Dice coefficient on MoNuSeg to 83.21% and a corresponding decrease in IoU to 71.35%. On GlaS, the Dice coefficient was 92.61% and the IoU was 86.77%. These findings suggest that while an additional OD-SSM layer can provide supplementary global information, it offers diminishing returns in terms of performance improvement, and may even introduce a marginal performance degradation.

Time and Space Complexity Analysis

A comparative analysis of time complexity was conducted between the proposed method and two existing automatic prompt SAMs, AutoSAM and I-MedSAM. The results are shown in Table 7. Due to the strategy of freezing SAM image encoder and decoder adopted by each method, we focus on comparing the time complexity of learnable modules. Computational complexity (measured in GFLOPs) was chosen as the metric, as it directly reflects the number of floating-point operations required by the algorithm. In addition, the space complexity of learnable modules is provided for comparison in the form of parameter quantities. It can be seen that the proposed method is significantly lower than AutoSAM in terms of the number of parameters and computational complexity of the learnable module, reducing them by 39.4% and 32.9%. This result indicates that our method can achieve better anomaly detection results at lower complexity.

Visualization Results

Figures 4 and 5 show the visualization results of our method on the MVTec AD dataset and medical image dataset, respectively. The visualization results show that the proposed method can achieve excellent pixel-level anomaly detection performance in diverse and challenging industrial and medical image scenarios. This achievement is attributed to the carefully constructed modules in this work. Specifically, the SSMODC module effectively searches and determines the best prompt for SAM to use. This method maintains the parameters of each component of SAM unchanged during the training process, aiming to preserve SAM's excellent spatial perception ability, thus enabling high-precision anomaly detection on different datasets.

5 Conclusion

This work presents a novel pixel-level anomaly detection approach based on the Segment Anything Model (SAM), enhanced by the integration of the State-Space Model-based residual Omni-Dimensional module (SSMODC). Unlike conventional pixel-level anomaly detection techniques, our approach eliminates the need for manual prompt generation and the fine-tuning of SAM, thereby streamlining the implementation process while maintaining the generalization capabilities of the model. It can accurately capture anomalous regions within images by combining multi-scale features and global information, especially in complex scenarios. Extensive experimental results confirms the superior performance of our method on multiple datasets, particularly on the MVTec AD, MoNuSeg, and GlaS datasets, where it outperforms existing state-of-the-art methods across various metrics, including mean Absolute Error (MAE) and mean E-measure (mE). Ablation studies further substantiate the effectiveness of the proposed SSMODC module and multi-scale Omni-Dimensional Convolution.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant 62301621, in part by Shenzhen Science and Technology Program (No. 20231121172359002), in part by Guangdong Basic and Applied Basic Research Foundation (No. 2025A1515011398), and in part by Science and Technology Development Fund, Macao S.A.R (FDCT) 0028/2023/RIA1.