Directing Mamba to Complex Textures: An Efficient Texture-Aware State Space Model for Image Restoration

1 Introduction

Image restoration as a fundamental task in computer vision and image processing, aiming to recover details and improve image contrast from degraded images [Liang et al., 2021a; Tsai and Huang, 1984; Guo et al., 2024a; Zamir et al., 2022; Xiao et al., 2022], which has been widely applied in imaging devices and various vision systems [Zheng et al., 2022;

Figure 1: (a-b) We compute the residual error map between the highresolution and low-resolution images in the DIV2K and Manage109 datasets and find that the degree of degradation varies across different regions of the image. (c) We divide image patches into 10 groups sorted by texture complexity (measured by statistical variance) and calculate the average PSNR for each group in all Manage109 datasets. It can be observed that regions with richer textures suffer more severe degradation, resulting in lower PSNR values.

Li et al., 2022]. With the continuous advancements and widespread deployment of modern smartphones and cameras, the demand for higher image quality has significantly increased from the 1K resolutions (1280 × 720) to 4K (4096 × 2160) [Kong et al., 2021]. Therefore, balancing the growing demand for high-quality restoration quality and model efficiency has become a critical challenge in image restoration [Liang et al., 2021b; Guo et al., 2024a].

With the rapid advancement of deep learning, various image restoration methods have been proposed. Early research in image restoration primarily focused on convolutional neural networks (CNNs) [Zhang et al., 2018; Sun et al., 2023], which offer high computational efficiency. However, limited by the local receptive fields of CNNs, these methods struggle to model long-range dependencies, resulting in suboptimal restoration quality. Therefore, Vision Transformers are introduced to improve image restoration performance due to their ability to model long-range dependencies [Chen et al., 2023a; Zhou et al., 2023; Cai et al., 2023a]. Despite these advancements, the high computational complexity of Transformers remains a major obstacle for real-world deployment, especially for high-resolution images.

To seek efficient frameworks, many previous works have proposed combining the efficiency of CNN with the global modeling capabilities of Transformers, resulting in hybrid CNN-Transformer models that aim to balance performance and complexity [Liang et al., 2021b; Zhou et al., 2023; Fang et al., 2022]. However, these approaches often overlook the spatial characteristics of degradation, where regions with richer textures tend to suffer more severe damage, applying uniform deep representation extraction across the image. This uniform approach makes it challenging to achieve a trade-off between performance and efficiency. Specifically, we argue that regions with richer textures are generally more prone to severe quality degradation. To clarify this observation, we conduct a statistical analysis on image super-resolution (additional analyses of low-light image enhancement are shown in the appendix). First, we divide the low-resolution or lowquality images into patches and measure the texture complexity of each patch using statistical variance. We then evaluate the PSNR between the degraded patches and their ground truth counterparts. The results, as shown in Figure 1, reveal that regions with richer textures tend to suffer more severe degradation, resulting in lower PSNR values. This highlights the importance of paying more attention to regions with complex textures. Although these methods [Kong et al., 2021; Jeong et al., 2025] attempt to address this issue by classifying image patches/pixels and applying different-sized CNNs, they still face challenges in perceiving image textures and modeling contextual information, making it difficult to achieve high-efficiency.

To address these issues, we propose a novel Texture-Aware State Space Model (TA-SSM) to simultaneously achieve efficient contextual modeling and texture awareness, enabling an optimal trade-off between performance and efficiency. Specifically, we modulate the transition matrix of state-space equations to mitigate catastrophic forgetting in regions with richer textures, enhancing texture perception. Furthermore, by focusing more on challenging texture regions, the proposed method improves overall efficiency. Additionally, for the first time, we introduce positional embeddings into SSM, improving their ability to perceive contextual positions. To further reduce computational costs while maintaining multidirectional perception, we design a Multi-Directional Perception Block to enhance efficiency. Extensive experiments on various image restoration benchmarks including image superresolution, image deraining, and low-light image enhancement, demonstrate that the proposed method significantly improves the efficiency of Mamba-based image restoration. This provides a more powerful and efficient framework for future image restoration tasks.

The contributions of this paper can be summarized:

• We propose a novel and efficient Texture-Aware State
• Space Model (TA-SSM) that perceives complex textures by modifying the state-space equations and transition matrices and simultaneously focuses on more challenging texture regions to enhance efficiency.
• An efficient Multi-Directional Perception Block is proposed to expand the receptive field in multiple directions while maintaining low computational costs. Furthermore, position embedding is introduced into SSM to enhance its capability of perceiving contextual positions.
• Based on these, a straightforward yet efficient model TAMambaIR is proposed, showcasing superior performance in both efficiency and effectiveness across various image restoration tasks and benchmarks, offering an efficient backbone for image restoration.

2 RelatedWork

2.1 Image Restoration

Images captured in complex real-world scenarios often suffer from degradations like low resolution, low-light, rain, and haze, resulting in reduced contrast and detail loss [Rim et al., 2020; Wang et al., 2018; Li et al., 2025; Peng et al., 2024a; Peng et al., 2025; Peng et al., 2024c; Peng et al., 2024b]. Image restoration aims to enhance contrast and recover details, improving visual quality. Advances in deep learning have significantly boosted its effectiveness [Liang et al., 2021b]. Dong et al. [Dong et al., 2015] introduced SRCNN in 2015, pioneering deep learning for image super-resolution (SR). Since then, numerous CNN-based methods have emerged to address image restoration tasks [Zhang et al., 2018; Dai et al., 2019], though their lightweight designs often limit receptive fields. To address this issue, Vision Transformers (ViTs) have been introduced to leverage long-range modeling for image restoration. Chen et al. [Chen et al., 2021] demonstrated their effectiveness in image denoising, while the Swin Transformer [Liang et al., 2021b] captured multi-scale features hierarchically. However, increasing image resolutions pose significant computational challenges for Transformerbased methods [Guo et al., 2024a]. While CNN-Transformer hybrids aim to balance performance and efficiency, they often overlook the spatial characteristics of degradation, where regions with richer textures tend to suffer more severe damage. As a result, they apply uniform enhancements and struggle to effectively focus on the damaged textures. Patchor pixel-level classification methods [Kong et al., 2021; Jeong et al., 2025] address this partially but fail to effectively model textures and contextual information. To tackle this, we adopt efficient global modeling with State Space Model (SSM) and introduce texture-aware capabilities to better handle complex textures, achieving a balance between performance and efficiency.

2.2 State Space Model

State Space Model (SSM), originally developed in the 1960s for control systems [Kalman, 1960], has recently been extended to applications in computer vision [Zhu et al., 2024; Xiao et al., 2025]. The introduction of SSM into computer vision was pioneered by Visual Mamba, which designed the VSS module to achieve superior performance compared to Vision Transformers (ViTs) [Dosovitskiy et al., 2020], while maintaining lower model complexity. This breakthrough inspired a wave of research utilizing SSM in various tasks [Guo et al., 2024b; Qiao et al., 2024]. Notably, MambaIR [Guo et al., 2024b] is the first to employ SSM for image restoration, demonstrating improved efficiency and enhanced global perceptual capabilities. However, directly applying SSM to image restoration poses challenges in effectively enhancing texture-rich regions, which limits the full potential of the State Space Model in this domain.

3 ProposedMethod

3.1 Traditional State Space Model

Let's briefly review the traditional State Space Model (SSM), which maps sequence input x(t) to output y(t) through an implicit latent state h(t) ∈ RN [Guo et al., 2024b] and can be represented as a linear ordinary differential equation:

where N is the state size, A ∈ RN×N , B ∈ RN×1 , C ∈ R1×N , and D ∈ R. Discretized using a zero-order hold as:

After the discretization, the discretized version of Eq. (1) with step size ∆ can be rewritten as:

The advent of SSM has garnered significant attention for its ability to perform global modeling with linear complexity in image restoration [Guo et al., 2024a]. However, traditional SSM still faces two critical challenges: (1) It consistently handles flat and texture-rich regions, spending too much computation on easily recoverable flat areas (as shown in Figure 1), resulting in inefficiency. (2) It lacks an effective texture-awareness mechanism, which not only leads to catastrophic forgetting of texture information but also hinders the model's ability to focus on challenging texture-rich regions.

3.2 Texture-Aware State Space Model

To address these limitations, we propose a novel TextureAware State Space Model to fully unlock the potential of SSM in image restoration, as shown in Figure 2. We tackle the aforementioned challenges from two perspectives. First, we propose a novel Texture Area Filtering method, which utilizes variance statistics to identify texture-rich and flat regions and pay less/more computation on flat/texture-rich regions, thereby improving efficiency. Second, we propose a novel Texture-Aware Modulation, which ranks regions based on their texture richness in descending order and enhances SSM's historical retention of texture-rich information by modulating the transition matrix in the state-space equation. Texture Area Filtering. We first patchify the image/feature x ∈ RC×H×W into patches p ∈ RC×h×w to obtain a sequence {Pi}, as follows:

where N represents the number of patches. Then, we sort this patch sequence based on the texture complexity from high to low, which is measured by statistical variance (other measurements of compression are provided in the appendix), resulting in a new ordered patch sequence {P ′ i }, as follows:

Then, we process the top p% patches with the highest texture complexity and skip the easy flat regions (which will be enhanced by the following full-sequence scan and convolution as shown in the (a) of Figure 2 ), directing SSM focus on the more challenging region as follows:

Texture-Aware Modulation In traditional SSM, the transition matrixes B and ∆ are expressed as follows:

This way may lead to two key issues: (a) The transition matrix relies solely on the input itself and lacks the ability to perceive texture information, resulting in catastrophic forgetting of texture-related representations in the state-space history, thereby weakening the model's ability to enhance fine details, as shown in the (d) of Figure 2. (b) Flat regions, which are easier to enhance, dominate the state-space history, leading to an excessive focus on flat areas and hindering the model's ability to address challenging texture-rich regions. To address these issues and enhance texture awareness, we first sort the patches as described in Eq.6, and then propose modulating the transition matrix within the state-space model to improve its capability of preserving historical texture information, as shown in Figure2 (b) and (e).

Through this method, the transition matrices Bvar and ∆car are able to capture the varying texture complexities of different patches. Furthermore, they retain the historical information of regions with richer textures, effectively alleviating catastrophic forgetting and enhancing the ability to represent fine details in image restoration, as illustrated in Figure 2 (e). Finally, the state-space function formulation of TextureAware State Space Model can be expressed as follows:

Position Embedding. It is well known that the traditional SSM adopts a linear sequence scanning method, which inevitably limits the contextual receptive field. Although SSM proposes scanning in four directions to partially improve the contextual receptive field, the perception capabilities of contextual positions remain limited. On the other hand, considering that our TA-SSM employs a texture complexity-based scanning method, this further exacerbates the contextual position perception process. To address this, we propose to introduce position embedding, which is widely used in Transformers, into SSM to enhance its contextual position awareness. This method enables SSM to capture the global positional relationship of the current token during each sequence scan, thereby improving its ability to perform global texture arrangement and contextual modeling. Specifically, we add a learnable position embedding Pos to each patch after Eq. 6:

3.3 Multi-Directional Perception Block

Through Texture-Aware State Space Model can model the global information of different patches based on their texture complexity. However, the contextual perception within a patch remains critical. Existing approaches often adopt multidirectional scanning within a single SSM, which can lead to significant computational overhead. To address this issue, we propose a novel Multi-Directional Perception Block that sequentially connects four different scaning directions (TopLeft Horizontal, Bottom-Right Horizontal, Top-Left Vertical, and Bottom-Right Vertical) to reduce computational cost while enhancing multi-directional perception, as shown in Figure 2 (a) and (c). Additionally, low-texture complexity regions (e.g., flat areas), which are relatively easier to enhance, are also crucial for image restoration. Therefore, we introduce a full-scan SSM and convolution layer to ensure the network can also focus on these regions, facilitating interactions between high-texture complexity patches and flat patches.

3.4 TAMambaIR

Following the previous approach in [Guo et al., 2024a], we adopt a simple yet efficient architecture to build TAMambaIR, as depicted in Figure 2. TAMambaIR comprises a feature extractor, a Texture-Aware State Space Group, and an upsampler. Specifically, convolution layers and pixel shuffle are used in the feature extractor and upsampler. Due to limited space, the implement details of the Texture-Aware State Space Group and Texture-Aware State Space Block are provided in Appendix Section 8.

3.5 Loss Function

Following previous works [Zamir et al., 2022; Cui et al., 2023], we utilize the L1 loss L1 and frequency loss Lfft for training. The total loss is presented as follows:

where O and Ogt denote the model output and the ground truth, respectively. The parameters λ1 and λ2 are balancing factors. We set λ1 and λ2 to 1, 0.05, respectively.

4 Experiences

4.1 Experimental Settings

Training Details. Following prior works [Liang et al., 2021b; Guo et al., 2024a], the training batch sizes for image super-resolution, image deraining, and low-light image enhancement are set to 32, 16, and 16, respectively. During training, the original images are cropped into 64 × 64 patches for image super-resolution, and 256 × 256 patches for image deraining and low-light image enhancement. Top p% epirically is set at 0.5. Following [Liang et al., 2021b; Guo et al., 2024a], we use the UNet architecture for image deraining and low-light image enhancement. The Adam optimizer [Kingma and Ba, 2014] is used to train our method, with β1 = 0.9, β2 = 0.999, and an initial learning rate of 2 × 10-4 . The standard model consists of 7 intermediate blocks with a depth of 7, while the small version consists of 6 intermediate blocks with a depth of 6. All experiments are conducted on 8 NVIDIA V100 GPUs. Additional details of the implementation are provided in Appendix Section 9.

Dataset. To evaluate our method, we select three popular and representative image restoration tasks, including image super-resolution (SR), image draining (Derain), and low-light image enhancement (LLIE). Specifically, for SR, DIV2K [Timofte et al., 2017] and Flickr2K [Lim et al., 2017] are employed to train the network, while Set5 [Bevilacqua et al., 2012], Set14 [Zeyde et al., 2012], B100 [Martin et al., 2001], and Manga109 [Matsui et al., 2017] are used for evaluation. For Derain, we follow [Chen et al., 2023b] and validate our approach on both popular benchmarks: Rain200H and Rain200L [Yang et al., 2017], which contain heavy and light rain conditions, respectively, for training and evaluation. For LLIE, we follow [Cai et al., 2023b] and validate our approach on the synthetic version of the LOL-V2 dataset [Yang et al., 2021]. More details about the datasets used are provided in the Appendix Section 10.

Evaluation Metrics. Following previous work [Chen et al., 2023b; Cai et al., 2023b; Guo et al., 2024a], we use Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) as evaluation metrics. For the image superresolution and deraining tasks, evaluation is performed on the Y channel of the YCbCr color space, while for the low-light enhancement task, evaluation is conducted in the RGB space. Comparisons with State-of-the-art Methods. We compare with many existing state-of-the-art methods. Image super-resolution: we compare our method against thirteen state-of-the-art methods, including: RCAN, SAN, ClassSR, IPT, CSNLN, NLSA, EDT-B, EDSR, RDN, HAN, SwinIR, SRFormer, and MambaIR. Image deraining: we compare our method against fifteen state-of-the-art methods, including: DDN, RESCAN, PReNet, MSPFN, RCDNet, MPRNet, SwinIR, DualGCN, SPDNet, Uformer, Restormer, IDT, DLINet, DRSformer, and MambaIR. Low-light image enhancement: we compare our method against fifteen stateof-the-art methods, including: RetinexNet, KinD, ZeroDCE, 3DLUT, DRBN, RUAS, LLFlow, EnlightenGAN, Restormer, LEDNet, SNR-Aware, LLFormer, RetinexFormer, CIDNet, and MambaIR. Due to the limited space, more detail and references are provided in Appendix Section 11.

4.2 Ablation Study

Table ?? presents the results of our ablation study on TAMambaIR-S at Manga109 datasets, showing the impact of removing critical components from the model. Excluding positional embeddings (w/o PosEmb) reduces the PSNR to 40.11, indicating their importance in enhancing the model's ability to perceive and utilize spatial context, which is especially beneficial for accurately restoring global structures. Removing the Multi-Directional Perception Block (w/o MDPB) and using a single direction to replace it reduces the PSNR to 40.14, demonstrating its role in capturing features from multiple directions efficiently. Replacing TA-SSM with traditional SSM and keeping the similar complexity causes the PSNR to decline to 39.98. This emphasizes the crucial role of TA-SSM in prioritizing complex textures and effectively modeling challenging regions. The complete model achieves the highest PSNR of 40.23, underscoring the necessity and complementary contributions of all components in delivering optimal performance. More ablation studies are provided in Appendix Section 12.

4.3 Quantitative Results

Image Super-Resolution. We compare our method against thirteen state-of-the-art CNN-based, transformer-based, and Mamba-based SR approaches. As shown in Table ??, our method achieves the best performance across all four benchmarks. Specifically, the small version of our method surpasses SRFormer by 0.15 dB and 0.20 dB on the PSNR of Manga109 and Set14 datasets, respectively. Moreover, the standard version of our method outperforms the current stateof-the-art method, MambaIR, with PSNR improvements of 0.07 dB and 0.05 dB on the Manga109 and Set14 datasets, respectively. These results highlight the superior performance and effectiveness of our approach.

Model Complexity Comparison. As shown in Table ??, benefiting from its strong texture-awareness capability, our method not only achieves the best performance but also demonstrates significant advantages in computational efficiency, including FLOPs (with input size 64 × 64) and model parameters. Specifically, the small version of our model TAMambaIR-S achieves state-of-the-art performance with relatively low complexity, surpassing SRFormer in terms of performance while reducing FLOPs by 6.07G. Furthermore, the standard version of our network outperforms MambaIR while achieving reductions of 20.5G (18.5%) in FLOPs and 4.35M (21.3%) in parameters. These results highlight the efficiency of our method. More comparisons of inferencing time are provided in Appendix Section 13.

Image Deraining. We compare our method against fifteen state-of-the-art deraining approaches, including CNN-based, transformer-based, and Mamba-based methods. As shown in Table ??, our method achieves the best performance on both Rain200L and Rain200H datasets in terms of PSNR and SSIM. Specifically, our method achieves a PSNR of 41.25 dB and an SSIM of 0.9896 on Rain200L, surpassing DRSformer, the previous best method, by 0.02 dB and 0.002, respectively. Similarly, on Rain200H, our approach attains a PSNR of 32.19 dB and an SSIM of 0.9345, outperforming MambaIR by 0.01 dB and 0.005, respectively.

Low-Light Image Enhancement. We compare our method with thirteen state-of-the-art approaches, including both CNN-, Transformer- and Mamba-based methods. We follow the Normal and GT Mean test settings in [Feng et al., 2024], and the results are shown in Table ??. Our method achieves the best performance on all settings in terms of PSNR and SSIM. Specifically, under the Normal setting, our method achieves a PSNR of 26.735 dB and an SSIM of 0.951, significantly surpassing RetinexFormer by 1.03 dB in PSNR and 0.021 in SSIM. Under the GT Mean setting, our method achieves a PSNR of 31.358 dB and an SSIM of 0.961, outperforming MambaIR by a notable margin. These results demonstrate that our approach effectively enhances low-light images while preserving details.

4.4 Qualitative Results

We present qualitative comparisons to demonstrate the effectiveness of our method on both image super-resolution (SR) and low-light image enhancement (LLIE), as shown at the top of Figure 3. For SR, our method reconstructs finegrained textures and sharp details that are highly consistent with the ground truth (GT), in regions with complex textures and high-frequency details. This highlights the superiority of our approach in generating realistic and visually pleasing results. For LLIE, our method achieves significant improvements in both visual clarity and natural color restoration, as shown at the bottom of Figure 3. Compared to existing approaches, our results better preserve structural details and generate more accurate brightness adjustments, making them closely aligned with the GT. More visual comparisons are provided in Appendix Section 14. Furthermore, to evaluate the visual quality, a user study is conducted for the image super-resolution. Specifically, we randomly select 15 images from the test datasets. Fifteen participants were asked to rate the visual quality of each processed image on a scale from 0 (poor quality) to 10 (excellent quality). The aggregated results, as shown in Figure 4, reveal that existing methods often fail to fully restore image quality, leading to lower user satisfaction. In contrast, our method achieves the highest average score of 8.4, demonstrating superior visual performance and generalization capabilities.

5 Limitaion and FutureWork

While our TAMambaIR demonstrates strong performance on three representative restoration tasks, its evaluation is currently confined to these specific domains. Future work will aim to extend our method to a broader range of low-level vision tasks, such as image dehazing, debluring, denoising, and inpainting, among others, to further investigate its generalizability. Additionally, the top p% of different images in the current framework is uniform, which may not fully account for the varying texture complexities of different images. Images with richer textures often pose greater challenges than those with fewer textures. Developing adaptive processing strategies, such as texture-aware top p% selection mechanisms, could enhance performance by dynamically allocating computational resources based on image characteristics. These adaptations will be a key focus of our future research.

6 Conclusion

In this paper, we proposed TAMambaIR, a novel framework for image restoration that achieves a balance between high restoration quality and computational efficiency. Leveraging the statical characteristics of image degradation, a novel Texture-Aware State Space Model is introduced, which enhances texture awareness and improves performance by modulating the state-space equation and focusing more attention on texture-rich regions. Additionally, the Multi-Directional Perception Block expands the receptive field in multiple directions while maintaining low computational overhead. Extensive experiments on benchmarks for super-resolution, deraining, and low-light image enhancement demonstrate that TAMambaIR achieves state-of-the-art performance with significant improvements in efficiency, providing a robust and effective backbone for image restoration.

Acknowledgments

This work was supported by the Natural Science Foundation of China under Grants 62225207,62436008 and 62206262

Contribution Statement

Renjing Pei and Yang Wang are the corresponding authors. Long Peng and Xin Di contributed equally.