RRG-Mamba: Efficient Radiology Report Generation with State Space Model

1 Introduction

Automatic radiology report generation (RRG) has emerged as an influential research area in medical imaging, driven by the increasing demand for efficient and accurate diagnostic tools [Hou et al., 2023; Bu et al., 2024]. Traditional radiology report generation relies on radiologists' expertise, making it time-consuming and susceptible to errors such as misdiagnoses or missed findings [Yan and Pei, 2022; Wang et al., 2023]. This challenge has led to growing interest in leveraging deep learning technologies to automate extracting critical information from medical images and generate professional reports, which can significantly enhance the efficiency of medical service delivery, optimize diagnostic accuracy, and alleviate the heavy workload of radiologists [Li et al., 2022b; Tanida et al., 2023; Bu et al., 2024].

∗Corresponding author.

Recent advancements in image captioning and encoderdecoder frameworks have driven significant progress in RRG. By leveraging convolutional neural networks (CNNs) for image encoding [Zhang et al., 2020; Huang et al., 2023; Bu et al., 2024] and Transformer models for report decoding [Li et al., 2022a; Li et al., 2023; Wang et al., 2023], these systems [Yan et al., 2021; Wang et al., 2023] aim to provide comprehensive and precise insights for timely and accurate medical diagnosis. This task involves two distinct modalities: visual (image) and textual (report) information. To facilitate cross-modal alignment and semantic integration, several approaches [Chen et al., 2020; Chen et al., 2021; Shen et al., 2024] incorporate a memory matrix mechanism that effectively integrates key features from both modalities, allowing the model to retain critical interactions between the images and the generated text, thereby improving the accuracy and consistency of the reports. Additionally, given the highly technical and medically specialized nature of radiology reports, recent studies [Zhang et al., 2020; Wang et al., 2022a; Yang et al., 2023; Huang et al., 2023] have focused on integrating multi-source medical knowledge, such as posterior-and-prior knowledge [Liu et al., 2021], medical knowledge graphs [Hou et al., 2023], and dynamic knowledge graphs [Li et al., 2023] to better understand medical image content, improving the visual feature representation and enhancing the overall report generation process.

Despite the significant progress in recent RRG methods, they still have several limitations. Traditional methods utilizing pre-trained CNNs [Zhang et al., 2020; Huang et al., 2023; Bu et al., 2024] focus on local feature extraction but struggle to model global context and long-range dependencies due to their limited receptive fields [Zhu et al., 2024; Liu et al., 2024]. Vision Transformers (ViT) address the above limitations by employing self-attention mechanisms to capture global dependencies [Li et al., 2022a; Wang et al., 2022b; Li et al., 2023]. However, this comes at the cost of increased computational resources and storage requirements, particularly on large-scale datasets. The self-attention mechanism requires computing correlations between all positions in the input sequence, leading to a quadratic growth in parameter count with sequence length. Consequently, the trade-off between global context modeling and computational efficiency in the RRG task remains an unresolved challenge.

To fill the gap, we propose RRG-Mamba, a novel framework that leverages Mamba's capability for global information modeling in radiology report generation. Furthermore, RRG-Mamba introduces rotary position encoding (RoPE) to optimize the vanilla Mamba, improving its flexibility and efficiency in capturing relative positional information and longrange dependencies from the visual feature sequences. Building on this, we design a global dependency learning module that dynamically captures relative positional information while effectively modeling complex spatial relationships. Finally, comprehensive experiments have demonstrated the effectiveness of RRG-Mamba. As shown in Figure 1, this approach not only alleviates the model's computational overhead but also preserves its accuracy, achieving a superior trade-off between efficiency and performance.

The main contributions in the paper are as follows:

• We identify a key challenge in radiology report generation: the inherent trade-off between effectively capturing global dependencies and maintaining computational efficiency. To our knowledge, this work marks the pioneering attempt to apply Mamba for addressing this critical issue within the task.
• We propose RRG-Mamba, a novel framework for efficient visual representation. Additionally, RRG-Mamba designs a global dependency learning module that integrates rotary position encoding, enhancing the vanilla Mamba for effectively modeling of long-sequence visual feature dependencies.
• We conduct comprehensive experiments across two public datasets, meticulously evaluating the performance of our proposed RRG-Mamba. The results demonstrate RRG-Mamba's superior efficacy over multiple baselines and establish new performance benchmarks, while also improving computational and memory efficiency.

2 RelatedWork

2.1 Radiology Report Generation

RRG is a critical task in medical artificial intelligence, which aims to automatically generate descriptive and clinically relevant reports from medical images. In recent years, researchers [Huang et al., 2023; Xue et al., 2024; Shen et al., 2024; Hou et al., 2025] have made significant progress in RRG. [Chen et al., 2021] used a shared memory matrix in the encoder to fully explore the association between medical images and texts, thereby promoting the interaction between cross-modal information. [Li et al., 2023] proposed a dynamic knowledge graph-enhanced model for radiology report generation that integrates medical knowledge to improve visual feature representations and optimizes dynamic graph retrieval through contrastive learning, thereby enhancing report accuracy. [Wang et al., 2023] employed a Vision Transformer as an encoder to extract visual features. They introduced multiple learnable "expert" tokens in both the encoder and decoder to interact with visual tokens, thereby enhancing the model's attention to fine-grained lesion areas. Although these methods have made significant progress in RRG, they still face limitations in extracting fine-grained pathological features and enhancing the representation of key lesion areas.

2.2 State Space Models

Recently, state space models (SSMs) [Hui et al., 2019; Gu et al., 2021; Gu et al., 2022] have received widespread attention due to their potential in sequence modeling. In particular, the enhanced Mamba model, leveraging the SSM, significantly accelerates inference speed while effectively modeling long-range dependencies in sequential data through a hardware-aware parallelization strategy [Gu and Dao, 2023]. Inspired by Mamba, multiple models have demonstrated remarkable advantages across diverse applications. For example, [Liu et al., 2024] proposed a state-space model with a global receptive field, incorporating multi-directional scanning and hierarchical networks to comprehensively capture information at every position within the input sequence. [Yue and Li, 2024] utilized grouped convolutions and channel shuffling to achieve efficient and generalized medical image classification while significantly reducing computational overhead. Similarly, [Zhu et al., 2024] employed a bidirectional state-space model to effectively capture global information, complemented by a position embedding module for local semantic feature perception. The successful application of SSM to complex sequence modeling tasks underscores their efficacy and offers valuable insights that pave the way for further advancements in medical image analysis.

3 Preliminary

State Space Models. The SSM-based models, e.g., Mamba, map the one-dimensional input sequence x(t)∈R to the output sequence y(t)∈R, capturing their relationship through a hidden state h(t)∈RN for sequence modeling and prediction. The calculation process is as follows:

where A∈RN×N is evolution parameter, B∈RN×1 and C∈RN×1 refer to the projection parameters.

To adapt SSM for deep learning, it is necessary to convert it from a continuous-time model to a discrete-time model by introducing a time scale parameter ∆ and applying the zeroorder hold (ZOH) discretization rule, as follows:

where ¯A and ¯B denote the discrete parameters.

After discretization, the Eq.(1) is written as follows:

Finally, the SSM model utilizes a global convolution to compute the output:

where ¯K∈RL and ∗ denote a structured convolutional kernel and the convolution operation, respectively. L represents the length of the input sequence x(t).

Radiology Report Generation. RRG involves automatically generating descriptive and accurate reports R={w1,w2, ...,wNR } based on medical imaging I={i1, i2, ..., iNI }, where NR and NI are the number of tokens in reports and the number of images, respectively.

4 Method

This section details RRG-Mamba, comprising the visual extractor, global dependency learning module, and report generator, as illustrated in Figure 2.

4.1 Visual Extractor

For each image i, the corresponding visual features v∈RH×W×C =Vencoder(i) are extracted through the visual encoder, where H and W represent the spatial dimensions of the feature map, and C is the number of channels. Vencoder(·) denotes the visual encoder.

Subsequently, the visual feature v generates two distinct visual representations: the global visual feature representation vG∈RHW×C and the serialized token feature representation vS∈RHW×C . The detailed computation procedure is outlined as follows:

where 'AvgPool', 'LN', 'Proj' and 'Flat' represent global average pooling, layer normalization, projection and flattening operations, respectively.

4.2 Global Dependecy LearningModule

Due to the limitations of CNNs' local receptive field, we design the global dependecy learning module (GDLM), leveraging the SSM backbone to effectively model long-range dependencies within medical image sequences and capture critical lesion features. Additionally, we integrate rotary position encoding (RoPE) [Su et al., 2023] to enhance the vanilla Mamba, improving its efficiency and flexibility in capturing relative positional information from visual feature sequences.

Specifically, inspired by dynamic position bias (DPB) [Chu et al., 2023], we first use DPB to adjust the spatial information of the serialized tokens vS, thereby enhancing RRGMamba's sensitivity to the spatial regions of visual features and capturing spatial structural information. The calculation process is as follows:

where vP∈RHW×C is the visual feature sequence adjusted by DPB.

Then, to enhance the correlation between various positions in medical images, we input the feature vP adjusted by DPB into the convolution operation Conv(·) to extract richer local spatial features and strengthen global dependency modeling:

where vcon is the feature representation obtained after the convolution operation.

To further capture global long-term dependencies and contextual relationships within the visual feature sequence, we enhance Mamba by integrating rotary positional encoding (RoPE), which improves the model's ability to preserve spatial positions and rotational invariance. The specific calculation process is as follows:

where vm con and vs con represent the intermediate features obtained by projecting to the hidden space. vm con,m, vm con,n are the m-th and n-th visual embedding vectors in the input sequence vm con , respectively. RdΘ,m and RdΘ,n are orthogonal matrices. Wq and Wk are weight matrices. vrp is the feature sequence obtained by the RoPE, and vM represents the output of the improved Mamba. 'SiLU', 'DWconv', 'Linear', 'Norm'and 'SS2D' represent activation function, depthwith convolution, linear layer, normalization layer and 2D selective scan, respectively. The ⊙ is element-wise multiplication.

Finally, we concatenate vG and vM to obtain the output of the GDLM:

where v′G∈RHW×C is the global visual features obtained by GDLM and 'Concat' denotes the concatenate operation.

In Eq.(11), SS2D is a pivotal component of the Mamba framework, with its structure depicted in Figure 2. In this process, visual features are initially partitioned into nonoverlapping patches, then scanned along four distinct paths, generating four independent sequences. Each sequence is subsequently processed by the selective scan space state sequential model (S6) [Gu and Dao, 2023], which extracts spatial information from various directions while effectively preserving critical contextual features. Finally, the four sequences are merged to form the consolidated 2D visual feature representation. This multi-path scanning and selective spatial processing enable RRG-Mamba to effectively capture richer spatial dependencies, which are essential for accurately modeling the complex visual patterns in medical images.

4.3 Report Generator

Cross-modal Adapter. We introduce cross-modal adapter to enhance the interaction and fusion of features across modalities. Specifically, inspired by [Wang et al., 2024], we employ Chexpert [Irvin et al., 2019] to generate pseudo labels y for each visual feature v′G, automatically annotating the presence of 14 prevalent diseases in medical images. This process strengthens the semantic alignment between visual and textual features, improving cross-modal consistency. The label generation is formalized as:

where NC is the number of disease categories and Wc is weight parameter. Φ(·) is the label generation function.

Then, we apply a projection layer to projects the visual feature sequence vlG fused with the pseudo labels and the hidden state H from the decoder layer into a shared feature space, obtaining the cross-modal featureM, as follows:

Encoder-Decoder. We employ a Transformer-based model to generate radiology report. Specifically, at time step T, the encoder maps the visual features v′G into intermediate feature representations hi∈R1×d . Next, we use the embedding layer to obtain the word embedding ji∈R1×d of each word token wi in the report R. d denotes the dimensionality of the hidden states. Then, the decoder generates the output of the current time step. In this cycle, a complete radiology report is gradually generated. The process is as follows:

where wi and pT are the i-th word in the ground truth and the predicted word at time step T, respectively. NS denotes the number of the intermediate features. The 'Embed' is embedding layer.

4.4 Objective Function

The cross-entropy loss function LCE is utilized to measure the divergence between the predicted report {pi}NR i=1 and the corresponding ground truth {wi}NR i=1 , thereby enhancing the model's ability to predict accurately, as follows:

5 Experiments and Analysis

5.1 Experiment Settings

Datasets. We evaluate our model on two publicly available RRG datasets: IU X-Ray [Shin et al., 2016] and MIMICCXR [Johnson et al., 2019]. IU X-Ray is a public radiology report generation dataset released by Indiana University. MIMIC-CXR is a large-scale chest X-ray dataset, which is widely used in tasks such as medical image processing.

Metrics. We use natural language generation (NLG) metrics to evaluate the quality of generated medical reports, including BLEU [Papineni et al., 2002], METEOR [Denkowski and Lavie, 2011], and ROUGE-L [Lin, 2004]. These metrics can measure the fluency and accuracy of the generated report and evaluate its similarity with the reference report, thereby effectively reflecting the generated report's language quality and content consistency. To assess the clinical utility of generated reports, we employ clinical efficacy (CE) metrics, including precision, recall, and F1-score. The evaluation focuses on disease-specific keywords derived from radiology reports, where we convert unstructured radiologist narratives into 14 structured labels.

Implementation Details. Following [Yan and Pei, 2022; Shen et al., 2024], we adopt the pre-trained DenseNet-121 [Huang et al., 2017] as the visual encoder. We provide several other variants of visual encoders, including ResNet-101

[He et al., 2016], ViT (ViT/B-16 and ViT/B-32) [Dosovitskiy, 2020] to further explore the performance of different visual encoders in RRG tasks. We configure the word identification ratio as k=0.5, thereby controlling the proportion of significant words. According to [Gu et al., 2022], we design three versions of GDLM with different structures (tiny, samll and base) to explore the impact of model capacity on RRG-Mamba.

5.2 Main Experiment

Overall Performance. We evaluate the performance of our proposed model by comparing it with state-of-the-art (SOTA) methods on both datasets. These methods are categorized based on their visual encoders: ResNet-101 (R2Gen [Chen et al., 2020], R2GenCMN [Chen et al., 2021], PromptMRG [Jin et al., 2024], R2Gen-Mamba [Sun et al., 2025]), DenseNet-121 (PPKED [Liu et al., 2021], Clinical-BERT [Yan and Pei, 2022], MAN [Shen et al., 2024]), and Vision Transformer (ViT) (BLIP [Li et al., 2022a], DCL [Li et al., 2023], METransformer [Wang et al., 2023]).

Table 1 summarizes the experimental results of our model on both datasets. The findings indicate that our model consistently outperforms most SOTA methods for the radiology report generation task across various evaluation metrics. Specifically, on the IU X-Ray dataset, compared to the MAN model (DenseNet-121), RRG-Mamba achieves notable improvements in BLEU{1-4} scores by 2.7%, 4.0%, 4.1%, and 3.7%, respectively, as well as enhancements in METEOR and ROUGE-L scores by 0.2% and 2.2%, respectively. Similarly, on the MIMIC-CXR dataset, RRG-Mamba (DenseNet-121) demonstrates superior performance, further validating its efficacy. Additionally, R2Gen-Mamba directly combines the vanilla Mamba with ResNet-101 to improve training and inference efficiency. In contrast, our proposed RRG-Mamba extends Mamba further by introducing RoPE to enhance relative position representation and incorporating a DPB to strengthen the model's ability to capture spatial positional information. These technical advancements lead to significant performance gains, with our method outperforming R2Gen-Mamba by 4.6% and 5.4% in BLEU-1 scores on two benchmark datasets. Next, we analyze the superiority of our proposed method from the following three perspectives.

Analysis on Visual Encoders. Table 1 shows the experimental results of our model RRG-Mamba using different visual encoders (ViT-B/32, ViT-B/16, ResNet-101, and DenseNet-121) on both datasets. Table 1 presents RRG- Mamba outperforms the Vision Transformer when using pretrained CNNs, particularly with DenseNet-121. This suggests that RRG-Mamba effectively extracts local pathological features frommedical images through multi-level convolutional operations and captures global context information via global dependency learning module. Consequently, RRG-Mamba acquires rich semantic representations that significantly enhance the discriminative power of medical image features.

Analysis on Clinical Efficacy Metrics. We evaluate the clinical accuracy and effectiveness of the generated medical reports using the CE metrics. Table 2 compares the performance of RRG-Mamba with existing models on the MIMICCXR dataset. The results show that RRG-Mamba achieves notable improvements of 8.7%, 5.5%, and 8.6% in precision, recall, and F1-score, respectively, compared to the MAN. These enhancements may be attributed to RRG-Mamba's ability to effectively capture fine-grained lesion features, resulting in reports with higher clinical accuracy and relevance.

Analysis of Model Complexity. Table 3 presents the model complexity experimental results of RRG-Mamba using different visual encoders on the IU X-Ray. A comparative analysis of the performance across various encoders reveals that the pre-trained CNN-based encoder offers superior reasoning capabilities and training efficiency compared to the ViT-based encoder. Specifically, the CNN-based encoder requires fewer training parameters, considerably reduces training time, and shows faster convergence during training.

Although the ViT excels in global context modeling, the SSM-based design of RRG-Mamba effectively mitigates the computational challenges inherent in ViT. Our approach not only efficiently captures long-range dependencies but also significantly enhances inference efficiency and accelerates training speed. This highlights the advantage of combining global dependency learning with optimized visual extraction, leading to both improved performance and reduced computational overhead.

5.3 Ablation Study

We perform an ablation study to validate the effectiveness of RRG-Mamba's core components on both datasets. Four variants are tested: w/o GDLM, which removes the global dependency learning module (GDLM) and omits long-range visual feature modeling; w/o RoPE, which excludes rotary position encoding within the GDLM, thereby neglecting relative positional information; w/o CMA, which eliminates the crossmodal adapter and adopts a simplistic approach for crossmodal feature fusion; and w/o CMA+GDLM, which ablates both the GDLM and CMA, relying solely on the model's basic structure for report generation.

Table 4 presents the results of the ablation study. The removal of the GDLM leads to a marked performance drop, underscoring its role in capturing long-range dependencies in visual features. Excluding RoPE diminishes performance, highlighting the importance of modeling relative positional information. The ablation of the CMA results in a significant decline, emphasizing the critical role of effective cross-modal interaction in improving overall performance. Finally, the simultaneous removal of both modules yields the lowest performance, further reinforcing the essential contributions of these two core components.

Analysis of Different Versions of Global Dependency Learning Module. Table 5 shows the experimental results of different versions of the GDLM (tiny, small and base) with varying model capacities on the IU X-Ray. Among these, GDLM-B achieves the best performance, with BLEU1, BLEU-4, METEOR, and ROUGE-L scores of 0.528, 0.207, 0.215, and 0.408, respectively.

These results underscore the critical role of model capacity in effective global dependency learning for medical images. GDLM-B, with its more sophisticated network architecture, is better equipped to model long-range dependencies, capturing both complex global and local features within medical images. These findings demonstrate that increasing model capacity can significantly improve the performance of global context modeling, enabling precise and comprehensive analysis of medical images.

5.4 Hyperparameter Study

We analyze the impact of the hyperparameter k (representing the word identification ratio) on RRG-Mamba's performance across both datasets. k quantifies the proportion of key terms identified in the generated text, optimizing the model's ability to effectively capture critical terms and control information redundancy. As depicted in Figure 4, RRG-Mamba achieves peak performance at k=0.5. However, increasing k beyond this value leads to performance degradation, likely due to the inclusion of redundant high-frequency terms (e.g., "the" "there"), which introduce noise and disrupt the crossmodal integration process, thereby impairing the accuracy of the generated reports. Conversely, excessively low k values may omit critical medical terms (e.g., "heart"), undermining the comprehensiveness and clinical relevance of the generated reports. Thus, the selection of k is crucial for optimizing report quality and enhance model performance.

6 Case Study

To explore the efficacy of RRG-Mamba, we randomly select a case from the MIMIC-CXR for detailed analysis. Figure 3 shows the medical reports and image-to-text attention heatmaps generated by the RRG-Mamba model and other comparison models (Transformer and R2GenCMN). We use orange fonts to mark organs or diseases and blue fonts to mark semantic descriptions similar to ground truth. It can be intuitively observed from Figure 3, the radiology reports generated by the RRG-Mamba can accurately identify the key feature areas in medical images and generate descriptions that contain professional terms and are in line with clinical reality.

Specifically, compared with the Transformer model, the RRG-Mamba model can capture subtle lesions (such as "a small consolidation" and "interstitial edema") and generate more accurate semantic descriptions. In contrast, the Transformer model has a weaker recognition ability for these lesions, and the generated reports are more general and lack detailed support. Compared with the RRG-Mamba, the R2GenCMN model lacks detailed descriptions and semantic richness, making it less effective in comprehensively analyzing medical images. Differently, RRG-Mamba model generates more comprehensive results, which not only identifies specific lesions ("a small consolidation") but also captures subtle findings in medical images and exclude other potential pathological features ("the right lung is otherwise clear. No pneumothorax or pleural effusions seen").

7 Conclusion

We propose RRG-Mamba, a novel framework that leverages Mamba's capability for global information modeling in radiology report generation. Additionally, RRG-Mamba designs a global dependency learning module that integrates rotary position encoding, enhancing the vanilla Mamba for effective modeling of long-sequence visual feature dependencies. At last, we conduct extensive experiments on two publicly available datasets, demonstrating RRG-Mamba's superior effectiveness compared to representative baselines and establishing new performance benchmarks. Furthermore, RRGMamba exhibits significant computational and memory efficiency advantages over prevailing neural network architectures, such as CNNs and Transformers.

Acknowledgments

This work is supported by grant from the Natural Science Foundation of China (No.62372077).

Contribution Statement

Xiaodi Hou and Xiaobo Li made equal contribution.