SAM-GPT: Hilbert Curve Enhanced Mamba for Brain Lesion Segmentation and VLM-based Analysis

1 Introduction

Brain lesions profoundly affect quality of life, with ischemia and glioma being among the most common types [Hu et al., 2020]. In MRI analysis, although the lesions originate from different mechanisms, they share the same imaging characteristics such as small lesion size, sparse distribution, blurred boundaries, and low contrast against surrounding tissues, which significantly increase the difficulty of lesion localization for vision-language models. Radiological diagnosis often relies on correlating clinical information with multi-

and demands extensive expert knowledge. Consequently, radiology diagnosis accuracy is vulnerable to human error and limited physician experience [Awadalla et al., 2023].

Early downstream methods adopted encoder-decoder architectures, using CNNs for visual feature extraction and RNNs or Transformers for text generation [Chen et al., 2020; Liu et al., 2021]. Yet these models were barely trained on medical image-text pairs. Recently, the CLIP model [Radford et al., 2021] has provided a new perspective for classification tasks. With the development of Vision-Language Models (VLMs), new models like Qwen2.5-VL [Bai et al., 2025], GPT-4o [Hurst et al., 2024] have recently emerged. These VLMs are pre-trained on vast amounts of image-text pairs. They then require only a limited number of medical pairs for fine-tuning to adapt to specific domains.

However, in complex medical scenarios, VLMs such as Qwen2.5-VL struggle to accurately localize lesions, often leading to misclassification, as shown in Figure 1. This stems from insufficient spatial grounding. Segmentation models trained on natural images perform poorly on medical lesion localization, while insufficient spatial continuity modeling further amplifies VLM misclassification for small and sparse lesions. Although State Space Models like Mamba efficiently capture long-range dependencies, standard row-byrow flattening separates vertically adjacent pixels by large intervals. This disruption limits the modeling of continuous lesion structures and hinders the capture of complex tumor boundaries [Liu et al., 2024]. This motivates us to introduce a new scanning mechanism that better preserves both local continuity and global spatial relationships in medical images.

To mitigate these limitations, we propose a novel SAMGPT framework, integrating SAM 2 [Ravi et al., 2025] to guide Qwen2.5-VL in lesion localization. In order to leverage information in diagnostic images with varying contrasts (i.e., different modalities) to complement features, we design a novel modality interaction approach utilizing the Hilbert curve [Jagadish, 1997; Wu et al., 2024]. Specifically, we embed a Hilbert Mamba Agent into the SAM 2 encoder for efficient multi-modal feature interaction and fusion. We further redesign the decoder as a 3D progressive upsampling architecture with a Memory Encoder and Memory Bank, replacing self-attention with Hilbert Mamba to better capture fine-grained lesion boundaries. Lastly, we utilized the lesion range, pixel area, and lesion location obtained from the segmentation results of SAM 2 to enhance Qwen2.5-VL's input prompts. Experiments on public brain lesion datasets (glioma, focal cortical dysplasia, and ischemic stroke lesions) show that SAM-GPT achieves an average Dice of 72.80% (up to 81.73%) and over 80% accuracy in lesion classification, establishing a new state-of-the-art.

Our main contributions are summarized as follows: (i) We propose a segmentation-guided VLM framework that converts lesion attributes into textual prompts, enabling more accurate diagnosis; (ii) we design a Hilbert Mamba based segmentation architecture to preserve local continuity and global dependencies; (iii) Extensive experiments on brain lesion datasets demonstrate that our model achieves state-of-the-art performance on multiple benchmarks and different tasks.

2 RelatedWorks

Vision-Language Models in Medicine. Early automated medical diagnosis relied on encoder-decoder architectures integrating CNNs and RNNs [Chen et al., 2020]. Later, instruction-tuned Large Language Models (LLMs) such as Med-PaLM M [Tu et al., 2024] and XrayGPT [Thawakar et al., 2024] have demonstrated strong clinical capabilities. More recently, general vision-language models like Qwen2.5-VL [Bai et al., 2025] have further advanced visual recognition tasks, including medical image diagnosis. Despite their success in medical diagnosis, most operate on coarse and noisy features, which limits their ability to recognize small pathological regions.

Medical SegmentationModels. 3Dmedical segmentation architectures have transitioned from CNNs to Transformers, UNETR [Hatamizadeh et al., 2022b] and SwinUNETR [Hatamizadeh et al., 2022a] set strong baselines. The Segment Anything Model (SAM) [Kirillov et al., 2023] further introduced foundation models to the medical domain, such as MedSAM [Ma et al., 2024]. Most recently, SAM 2 [Ravi et al., 2025] has been adapted for 3D segmentation, improving the segmentation ability. However, when applied directly to small and sparse brain lesions, these models often lead to coarse localization.

Visual State Space Models. State Space Models (SSMs), particularly Mamba [Gu and Dao, 2024], enable efficient long-sequence modeling. In the vision domain, VMamba [Liu et al., 2024] introduced 2D scanning mechanisms, which were extended by SegMamba [Xing et al., 2024] to 3D medical segmentation tasks, demonstrating competitive performance compared with Vision Transformers. However, traditional scan methods disrupt 2D spatial adjacency during sequence flattening, causing models to struggle to capture finegrained features.

To address these issues, our framework captures local spatial dependencies and precise small lesion features to generate a more accurate diagnosis, achieving state-of-the-art performance in both lesion segmentation and diagnostic classification.

3 Methods

Figure 2 illustrates SAM-GPT, a two-stage framework for brain lesion segmentation and VLM-based diagnostic classification. Specifically, in the first stage, HilbertSAM outlines the lesion boundaries from the input image sequence X = {xmi |m = 1, 2, ...,M; i = 1, 2, ...,N}, where M denotes the number of modalities (e.g., T1, T2, FLAIR) and N denotes the flattened spatial tokens of each image slice. The process generates size data and spatial location information for the lesions. Qwen2.5-VL utilizes these spatial priors to enhance the prompt, classifies the lesions, and combines the lesion information from the first stage with the classification information from the second stage to generate the final diagnostic result.

HilbertSAM primarily consists of a Hilbert-Enhanced Encoder and a Hilbert-Enhanced Decoder. Specifically, the Hilbert-Enhanced Encoder differs from the SAM 2 Encoder by supporting both single-modal and multi-modal inputs. To enhance the recognition capability of small lesions, we modified the SAM 2 decoder, the Hilbert-Enhanced Decoder effectively leverages features stored in the Memory Bank while incorporating Hilbert Mamba to scan different windows and perform progressive upsampling, ultimately achieving precise localization of lesion spatial positions.

3.1 Preliminaries

State Space Models. Structured State Space Models (S4) [Gu et al., 2022] and Mamba [Gu and Dao, 2024] are sequence models that utilize a hidden state h(t) ∈ RN×1 to map a one-dimensional input signal x(t) ∈ R to an output y(t) ∈ R, where N indicates the dimensionality of the hidden state. Mathematically, the state space model (SSM) [Chen, 1984] transforms input data through the ordinary differential equations (ODE) h′ (t) = Ah(t) + Bx(t) and y(t) = Ch(t), where A ∈ RN×N , B ∈ RN×1 , and C ∈ R1×N are the system's evolution and projection parameters. For deep learning applications, the continuous system is discretized via Zero-Order Hold (ZOH), resulting in a discrete model, hk = ¯ Ahk-1 + ¯Bxk and yk = Chk, where the parameters are transformed as ¯ A = exp(∆A) and ¯B = (∆A)-1 (exp(∆A) - I) · ∆B. This framework is further leveraged by converting to a convolutional neural network (CNN) form for parallel computation, represented as y = x ⋆ M, with M being the convolution kernel derived from the state space model.

Hilbert Curve. The Hilbert curve is a type of space-filling curve (SFC) [Moon et al., 2001] that can be represented as σ : [0, 1] → [0, 1] × [0, 1], mapping any point in the one-dimensional interval [0, 1] to coordinates in a twodimensional unit square. We denote the order of the Hilbert curve as n, which approximately corresponds to the height and width of each frame in discrete cases. For any two points u and v in the interval [0, 1], their spatial-to-linear ratio (SLR) is defined as:

The expansion factor (DF) of a space-filling curve is defined as the upper bound of the SLR. A lower DF value indicates a tighter mapping in the unit square, aligning with the requirements for preserving locality. Research indicates that the DF of the Hilbert curve is 6, while the DF of the conventional Zigzag curve is given by 4n - 2n+1 + 2, which increases towards infinity as the order n of the curve increases. Therefore, as image resolution improves, the Hilbert curve can more effectively maintain locality when mapping any two points in a one-dimensional sequence to multi-dimensional space compared to the Zigzag curve. To establish global and local correlations, we attempt to incorporate the Hilbert curve into state-space models (SSMs).

3.2 Hilbert Mamba

As shown in Figure 3, we first construct a 3D Hilbert curve that adheres to the principles of Hilbert curves, scanning through every point in the 3D Hilbert space. The 3D Hilbert algorithm recursively subdivides the 3D space into cubic subspaces and aggregates the generated Hilbert segments into a complete traversal curve. Subsequently, we convert each point on the 3D curve into a unique one-dimensional index. This indexing process flattens 3D space following the scanning trajectory of the Hilbert curve, mathematically represented as:

where (ˆx, ˆ y, ˆ z) denotes the coordinate index of each point, and Q represents the width length of the input features. Based on the Hilbert index, the features ˆ F ∈ RC×T×H/4×W/4 (C, H, W respectively denote channel, height, and width dimensions, T denotes the temporal dimension, i.e., slice depth) are flattened into a one-dimensional sequence ˆ V ∈

RC×(H/4×W/4×T) , which enhances local dependencies compared to the global sequence V . After the local feature enhancement via Hilbert scanning, the sequence is input into the Mamba block, which leverages spatiotemporal corrections in a local context. Ultimately, we reshape the output sequence back to its original dimensions, defined as ˆ F ′ ∈ RC×T×H/4×W/4 . Unlike other scanning methods, the Mamba enhanced with Hilbert scanning places greater emphasis on capturing local dependencies across both spatial and temporal dimensions.

3.3 Hilbert Mamba Applications

Hilbert-Enhanced Encoder. To enable multimodal fusion in the SAM 2 encoder and improve sensitivity to small lesions, we designed a novel Hilbert Mamba Cross Attention specifically for the Encoder of SAM 2 as illustrated in Figure 2. The Hilbert Mamba Cross Attention primarily consists of our newly designed Hilbert Mamba Agent. The Hilbert Mamba Agent dynamically aggregates cross-modal features through Hilbert-based localized scanning and QKV interaction provided by different modalities.

After the local features are scanned by the Hilbert Mamba, to facilitate interaction between different modalities, the Hilbert Mamba Agent incorporates a novel approach to Agent queries. Specifically, given two input modalities m1 and m2, the Hilbert Mamba Agent accepts the Query from m1 and the Key and Value from m2. We introduce a new weight B that replaces the Q values of m1. To prevent a significant increase in computational cost due to a large N, the dimension of B differs from that of Q, where B has a dimension of n, and the value of n is much smaller than N. Mathematically, the interaction of the modality can be expressed as:

where C1 ∈ Rn×N , C2 ∈ RN×n , Qm1, Km2, Vm2 ∈ RN×C , and B1 ∈ Rn×C . C1 and C2 are set as learnable parameters for biases. σ(·) denotes the Softmax function. Depthwise convolution (DWC) is introduced to mitigate the issue of insufficient feature diversity in agent attention.

To enhance the deep interaction between different imaging modalities and facilitate continuous low-level feature extraction, after the first Hilbert Mamba, we utilize a Multi-Layer Perceptron (MLP) to extract the first fusion's K and V. We then interact the Query values from m2 with the extracted K and V values. During the Q, K, V interaction, the Hilbert Mamba scans the newly synthesized sequence again. By continuously scanning local regions, we aim to improve the segmentation of small lesions at the pixel level (small lesions may occupy only 1-10 pixels in the overall pixel space).

Hilbert-Enhanced Decoder. Hilbert-Enhanced Decoder is composed of a memory encoder, a memory bank and a Hilbert-Enhanced Scale-Aware decoder. Memory encoder and memory bank create efficient and rich memory representations and store these memory representations. HilbertEnhanced Scale-Aware decoder utilizes these representations to optimize the continuity of the three-dimensional feature output. Specifically, to help the decoder decode subtle features from the feature sequence, we attempt to combine a progressive decoding approach with a Hilbert Mamba enhanced 3D decoder. Specifically, we replace the self-attention mechanism in the original decoder with Hilbert Mamba, enabling the decoder to re-scan output tokens and reconstruct adjacenttoken features instead of relying on global scanning. Structurally, the Hilbert-Enhanced Decoder employs three 3D decoders to fuse the pyramid feature maps of the current frame f t from the mask decoder and the previous frame f t-1 from the memory bank at the same resolution through concatenation and convolution. In addition, transposed-convolution is used in the 3D decoder to perform 2x upsampling on the feature maps, which are then fused with upper feature maps. The process is formulated as follows:

where UP(·) represents the transposed-convolution. The output feature maps of the Hilbert-Enhanced Scale-Aware Decoder are combined with the feature maps of the mask decoder to enhance the perception capacity of small targets.

3.4 Prompt Enhancement

Considering the Qwen2.5-VL model's limitation in capturing critical lesion information, we propose a Prompt Enhancement method. We use the HilbertSAM to accurately delineate lesion regions and generate lesion-aware spatial priors. Specifically, the segmentation results from HilbertSAM are converted into enhancement prompts, including lesion pixel regions, boundaries, and locations, to assist Qwen2.5-VL in accurate image recognition. We further analyze theoretically why enhancement prompts can improve the classification capability of Qwen2.5-VL.

We define the output of the model without enhancement prompts as Rbase = f(I), and the output of the model with enhancement prompts as Renhanced = f(I, P), we introduce a gain function G(P) to measure the knowledge gain of Qwen2.5-VL introduced by enhancement prompts:

where ∆f(I, P ) is the improvement of the model's capability of prediction under the enhancement prompts, defined as:

The term H(I, P) represents the information richness of the image-prompt pairs, α and β are weighting factors.

To evaluate whether the enhanced classification result is close to the ground-truth diagnosis, we define loss of knowledge as:

where Y denotes the ground-truth diagnosis label. The function can be further expressed as:

where ∥Renhanced - Y ∥2 is the Mean Squared Error (MSE) measuring the deviation between the enhanced output of Qwen2.5-VL and the ground-truth knowledge. Additionally, C(Renhanced, Y ) represents the loss of classification task between the enhanced prediction and the true label. γ and δ control the weights of the different loss components.

Consequently, the classification result can also be conceptually expressed as:

where G(P) · P(Y |I, P) represents the classification gain brought by Prompt Enhancement. P(Y |I, P) denotes the prompt conditioned confidence of Qwen2.5-VL for category Y , namely the confidence that the prediction is correct based on I and P . The overall metric measuring loss of knowledge is defined as:

where Jreg is a regularization term intended to prevent overfitting.

Overall, the Prompt Enhancement method is proposed to provide lesion-aware spatial knowledge for Qwen2.5-VL. A lower value of Jsa indicates that the gain effect introduced by Prompt Enhancement is more effective and that the enhanced model output is closer to the ground truth.

4 Experiments

4.1 Analysis of Misclassification

To validate the interference of image backgrounds on Qwen2.5-VL's ability to discern images, we designed the following experiment: First, we selected 200 medical images (including 50 T1 images of brain gliomas, 50 FLAIR images of brain gliomas, 50 T1 images of epilepsy, and 50 FLAIR images of epilepsy) and directly fed them to Qwen2.5-VL for discrimination. Next, we utilized the unfinetuned SAM 2 to segment these images. Finally, we employed the MedSAM 2 [Zhu et al., 2024b], to segment the images and then provided them to Qwen2.5-VL for assessment. The experimental results presented in Table 1 demonstrate that the image background significantly interferes with Qwen2.5-VL's correct identification of lesions and that precise removal of the image background shows a positive correlation with Qwen2.5-VL's accuracy in recognizing lesions.

4.2 Datasets

In the task of precise image background removal, we conducted a comparative evaluation of the Hilbert-enhanced SAM 2 network against state-of-the-art methods on three publicly available MRI brain lesion benchmark datasets (e.g., BraTS2021 [Baid et al., 2021], FCD2023 [Schuch et al., 2023], ISLES2022 [Petzsche et al., 2022]). We utilized the data format (3D and 2D) of each dataset to ensure fair comparison. We employed metrics such as the Dice coefficient, Intersection over Union (IoU), Precision, and Recall to quantitatively compare the performance of different methods.

For the lesion classification task of Qwen2.5-VL in recognizing lesion types, considering that the raw medical image format cannot be recognized by VLMs, we randomly selected 4,800 labeled 2D slices from the constructed brain lesion dataset (the dataset includes 1,600 slices of gliomas, 1,600 slices of epilepsy, and 1,600 slices of cerebral infarction). We conducted a comparative evaluation of our SAMGPT network on the 4,800 slices against the latest methods. We utilized ACC, PREC, RECALL, and F1 scores for the quantitative comparison of different methods.

4.3 Implementation Details

The proposed SAM-GPT was implemented using Python 3.10 and PyTorch 2.1.0 on a Ubuntu 22.04 server equipped with ten V100 GPUs. In our SAM-GPT model, we configured the initial learning rate to 1×10-5 . Throughout training, a cosine annealing scheduler is implemented, which adjusts the learning rate via cosine annealing. A warm-up phase is employed for the first five epochs, during which the learning rate is gradually increased to stabilize the training process.

4.4 Comparisons with State-of-the-art Methods

To evaluate segmentation performance, we compare HilbertSAM with eleven state-of-the-art methods, including five single-modality 3D segmentation networks (e.g., UNETR

[Hatamizadeh et al., 2022b], SwinUNETR [Hatamizadeh et al., 2022a], nnFormer [Zhou et al., 2023], MixUNETR [Shen et al., 2025], STU-NET [Huang et al., 2023]), six singlemodality segmentation networks based on SAM [Kirillov et al., 2023] and SAM 2 (e.g., SAM Med 2D [Cheng et al., 2023], SAM 2 [Ravi et al., 2025], SAM Med 3D [Wang et al., 2025], MedSAM 2 [Zhu et al., 2024b], FSMedSAM 2 [Bai et al., 2024]). Notably, the performance of zeroshot SAM 2 baseline is limited, illustrating the gap between generic segmentation models and medical-adapted segmentation models for small and sparse lesion targets. As shown in Table 2, SAM-GPT achieved the highest Dice coefficient (81.73%), IOU (72.80%), and Recall (79.56%) in the benchmark BraTS2021 dataset tests. It also shows consistent performance on the other two datasets, indicating its superior capability in brain lesion perception.

To validate the effectiveness of the Hilbert Mamba scanning approach, We further compare different Mamba scanning strategies integrated with SAM 2-VMamba [Liu et al.,

2024], EVMamba [Pei et al., 2025], and JamMa [Lu and Du, 2025]-on the BraTS2021 dataset. As shown in Table 3, Hilbert Mamba combined with SAM 2 achieves a Dice coefficient of 81.73% and an IoU of 72.80%, outperforming the other scanning methods and maintaining a good balance between model complexity and computational efficiency.

4.5 Discrimination Capability Comparison

For the task of lesion discrimination accuracy, we compare SAM-GPT with representative VLM-based methods (e.g., R2Gen [Wang et al., 2023], XrayGPT [Thawakar et al., 2024], XrayPULSE [OpenMedLab, 2024], Med-PaLM M [Tu et al., 2024], MiniGPT4 [Zhu et al., 2024a]). As shown in Table 5, SAM-GPT achieved the highest ACC(80.56%), Recall(83.23%), Prec(87.30%), and F1 scores(0.85) in testing 4,800 lesion images. To validate the effectiveness of our proposed solution for misclassification, we re-input the outputs of SAM 2 into these five models, accordingly enhancing their prompts. As shown in Table 5, all five models demonstrated a notable performance increase when augmented with inputs from SAM 2.

4.6 Visual Comparison

Figure 4 visually compares the lesion localization results predicted by our network and state-of-the-art methods on the brain lesion dataset we constructed. Compared to other methods, our network is more effective in localizing lesions.

4.7 Ablation Study

Baseline Design

We perform ablation experiments to verify the effectiveness of three critical components of our network, namely the Hilbert Mamba Cross Attention (HMCA), the Hilbert Mamba in the Decoder of SAM 2 (DHM), and the Prompt Enhancement for Qwen2.5-VL (PE). First, we define a baseline model referred to as "M1", which excludes these three primary components. Next, by incorporating HMCA, we enhance the baseline model "M1", resulting in model "M2". Building upon "M2", we further integrate the Hilbert Mamba from the SAM 2 Decoder to construct "M3". Finally, we introduce the prompt enhancement for Qwen2.5-VL. This series of experiments aims to validate the contribution of each component to the overall performance of the network.

Quantitative Comparison

The ablation study results presented in Table 4 provide quantitative results of different components of the SAM-GPT method on our constructed dataset of 4,800 brain lesion images. Specifically, compared to "M1", "M2" improves performance by introducing HMCA, achieving an 11.39% increase in ACC. The addition of DHM in "M3" leads to further improvements, particularly with Recall rising from 74.34% to 83.47%. Finally, incorporating Prompt Enhancement (PE) into the complete SAM-GPT model results in optimal performance, achieving an ACC of 80.56%, a Prec of 83.23%, a Recall of 87.30%, and an F1 Score of 0.85.

5 Conclusion

In this work, we propose SAM-GPT to address VLM misclassification caused by background interference, effectively mitigating background distractions and enhancing the VLM model's ability to distinguish lesions across different diseases. By integrating the Hilbert curve with Mamba and SAM 2, SAM-GPT captures local feature continuity and leverages multimodal inputs to improve lesion localization between small lesions and normal tissues. Additionally, we introduce a Prompt Enhancement method that leverages lesion size, pixel regions, and lesion coordinates to assist the VLMmodel in accumulating prior knowledge about lesions. Comprehensive experiments demonstrate that our method surpasses the current state-of-the-art in both lesion localization and discrimination capabilities.

Acknowledgments

This research was funded by the Fundamental Research Funds for the Central Universities of China (Grant No. XJ2026000301) and the Central China Normal University Wollongong Joint Institute.

Contribution Statement

Equal contribution (∗): Jinfu Wang and Qiyuan Wang. Corresponding author (†): Jinhua Zhao.