Unlocking Dark Vision Potential for Medical Image Segmentation

1 Introduction

Accurate segmentation of lesion areas in medical images is a key step in diagnosing, planning treatments, and monitoring various diseases [Aglinskas et al., 2022]. The precise excavation of disease markers directly affects subsequent clinical decisions and the effectiveness of interventions [Isensee et al., 2021]. Due to the principles of noninvasive imaging, achieving high-precision segmentation is significantly challenged by factors such as artifacts, low contrast, and inherent tissue characteristics that result in blurry images. Despite the powerful capabilities of deep learning models based on Convolutional Neural Networks (CNNs) [Yang and Yu, 2021; Wang et al., 2024c], Transformers [Dosovitskiy et al., 2021], or Mamba [Gu and Dao, 2024], they struggle to effectively address interference caused by image blurriness, leading to unsatisfactory segmentation results.

∗Corresponding author tongy@cse.sc.edu, sq.ma@siat.ac.cn, and guofei@csu.edu.cn

Various methods have been attempted to mitigate this issue, including model-based deblurring, image enhancement, and the use of deep learning for feature extraction and image reconstruction [Lian and Wang, 2023]. For instance, some studies utilize CNNs to learn the mapping from blurry images to clear images, while others explore the application of Generative Adversarial Networks (GANs) [Goodfellow et al., 2014] in image deblurring. In addition, efforts have been devoted to improving imaging equipment and acquisition techniques to reduce blurriness.

However, these methods often face common challenges. Firstly, the complexity and diversity of blurry images demand models with high adaptability and generalizability. Current deep learning models often require a large amount of annotated data for training, which is, however, scarce and expensive in the medical field. Secondly, key features may be lost due to blurriness, increasing the difficulty for models to learn these features. Lastly, although deep learning models perform well in object detection and classification, they still have limitations in interpreting the physical causes of blurry images and restoring clear images. How to enhance the model's understanding and processing capabilities of blurry images, especially with limited data, remains an open research question.

Through an in-depth understanding of the principles of photoreception and imaging of the eye, we have discovered that noninvasive medical imaging shares similarities with visual imaging in low light conditions, particularly in terms of contrast between the foreground and background [Hofmann and Lamb, 2023], as illustrated in Fig. 1. In addition, medical image processing and analysis exhibit a high degree of consistency with the contrast enhancement mechanisms of scotopic vision. In low light environments, rod cells are much more sensitive to dim light than cells [McKyton et al., 2024]. This characteristic suggests that an effective medical image analysis method needs to capture subtle signals.

Inspired by the scotopic vision perception mechanism, we proposed a dark vision network (DVNet) with a waveletdriven perception module (WDPM) to better perceive features in blurry image areas, as shown in Fig. 1. The wavelet transform [Tian et al., 2023] can decompose an image into a sum of coarse to fine image features, representing different structures of the image. Using the wavelet transform, the model can more easily extract the structural and detailed information from the original image.

We further proposed a Mamba-Enhanced Fusion Module (MEFM) to simulate the antagonistic receptive fields of horizontal and bipolar cells, which enhance the contrast between the central and peripheral visual fields through increasing the brightness difference and thus achieve a highlight representation of the central area. The Mamba framework [Gu and Dao, 2024] combines state-space models with selective mechanisms, effectively processing long-sequence data, which contributes to obtaining global information and localizing lesions. Our method, under a global view, can enhance the contrast between the lesion area and surrounding healthy tissues, achieving perceptual activation of the lesion area. The attention mechanism endows the model with the ability to interact and associate directly between different positions within the sequence. By adaptively weighting, it enhances the features of key areas in the image and suppresses the visual response of surrounding areas, thereby achieving enhanced contrast. In summary, the key contributions of this paper are as follows.

• By delving into the mechanisms of scotopic vision, we proposed a novel DVNet with a WDPM and a MEFM to enhance the model's ability to perceive blurred areas in non-invasive medical imaging. To the best of our knowledge, DVNet is the first medical image segmentation algorithm inspired by scotopic vision.
• The DVNet is plug-and-play and can be deployed in various architectures based on CNNs, Transformers, or Mamba. Extensive experiments have demonstrated that the DVNet can consistently enhance the performance of its baseline model by enhancing the model's visual acti-

vation of lesion areas through contrast enhancement.

2 RelatedWork

2.1 U-Net and Its Variants

U-Net [Ronneberger et al., 2015] has become a cornerstone in medical image segmentation due to its symmetric encoderdecoder structure and skip connections that enhance detail preservation. However, U-Net and its variants face limitations in modeling explicit long-range dependencies within images, which is critical for precise segmentation. To address the limitations of U-Net, researchers explored the integration of Transformers into U-Net. The integration of U-Net and Transformer, e.g., TransUNet [Chen et al., 2024a], leverages the strengths of both architectures to enhance segmentation performance: restoring local spatial information through UNet and modeling global context by Transformer.

2.2 Mamba inMedical Image Segmentation

Most recently, Mamba [Gu and Dao, 2024], a state space model, has emerged as a promising alternative to CNNs and Transformers in medical image segmentation. Mamba excels in modeling long-range interactions while maintaining linear computational complexity, which is particularly advantageous for handling large medical images. MambaUNet [Wang et al., 2024d] further synergizes U-Net with Mamba's capabilities by adopting a pure Visual Mambabased encoder-decoder structure, which facilitates comprehensive feature learning and captures both intricate details and broader semantic contexts within medical images. Mamba-UNet [Wang et al., 2024d] has been shown to outperform various types of UNet in medical image segmentation under the same hyperparameter setting.

2.3 Perception of Blurry Information

A variety of methods have been proposed to deal with the issue of blurry medical images. An iterative edge attention network was introduced in [Wang et al., 2022b], which employs an edge attention preservation module to reduce noise and help the edge flow focus on boundary-related information. Ma et al. [Ma et al., 2024c] developed an edge feature fusion module to integrate the spatial correlation of the edges of the injury in adjacent images, achieving more precise edge segmentation. Qiao et al. [Qiao et al., 2022] proposed an edge extraction method for medical images based on improved Local Binary Pattern combined with edge-aware filtering to suppress noise and enhance contrast while preserving edges.However, these methods typically improve edge detection capabilities by eliminating noise or incorporating additional information but do not fundamentally change the visual contamination issues caused by imaging blurriness.

2.4 Segmentation and Detection in the Dark

Low illumination can lead to degraded image quality including image blur and increased noise due to the low signal-tonoise ratio and reduced contrast between targets and backgrounds [Chen et al., 2023]. To overcome these issues, a DNF model [Jin et al., 2023] was proposed to decouple domainspecific subtasks, fully utilize the unique properties of the RAW and sRGB domains, and employ a feedback mechanism for feature propagation across stages, avoiding information loss caused by image-level data flow. Alternatively, an improved SFNet algorithm, i.e., SFNet-N [Wang et al., 2022a], was proposed specifically for semantic segmentation of low-light autonomous driving road scenes, which includes a light enhancement network and a segmentation network with strong feature extraction capabilities to identify objects in dark environments. Wang et al. [Wang et al., 2024b] proposed a method for multi-object tracking under low-light conditions by utilizing an adaptive low-pass downsampling module and a degradation suppression learning strategy to learn illumination-invariant features from low-light videos. Although these methods achieved success in different tasks under low-light natural scenes, they have not yet been applied to medical image analysis, which has similar conditions.

3 Methodology

DVNet represents a novel approach to mimicking the principles of dark vision by improving the sensitivity of the model to subtle image features and addressing the inherent blurriness in medical images. In this section, we elaborate on the three major components of the proposed DVNet: (1) WDPM, (2) MEFM, and (3) Spatial-Guided Refinement Module (SGRM). These components work synergistically to emulate the remarkable ability of nocturnal animals to perceive and process visual information in low light environments. Fig. 2 illustrates an overview of the proposed model.

3.1 Wavelet-Driven PerceptionModule

In dark vision, the ability of rod cells to perceive low contrast regions relies mainly on enhancing signals of different frequencies, which is critical to capture structural details in dim environments. However, traditional skip connections, commonly used in U-shaped networks, simply propagate lowlevel semantic information, limiting their ability to effectively process different frequency signals in blurry images. To address this limitation, the WDPM is developed by integrating a 3D stationary wavelet transform (SWT) module [Chen et al., 2012] into each branch of our U-shaped network. SWT preserves spatial resolution while decomposing feature maps into distinct frequency subbands, enabling the network to distinguish between structural details and noise. As shown in Fig. 3, this enhances feature representation at a deeper semantic level and aligns with the principles of dark vision.

As illustrated in Fig. 2, given a 3D input volume I ∈ RC0×D×H×W with C0 input channels, D slices (depth), spatial dimensions H (height) and W (width), the U-shaped architecture processes the input through its encoder structure, generating embedding Ee ∈ RCe× D 2e-1 × H 2e-1 × W 2e-1 at different layers (e ∈ {1, ..., 4}) corresponding to varying levels and resolutions of features.

Then, in each branch, the input feature maps are first processed through a 3D SWT, which decomposes the volumetric data into multiple subbands, capturing both high-frequency edge details and low-frequency structural information. The decomposed subbands are concatenated with the original input feature maps. This operation combines the complementary information from the frequency domain (SWT subbands) with the spatial features of the raw input, allowing the network to leverage both perspectives for improved feature representation. The concatenated features are passed through a 3D convolution layer to integrate spatial and frequencydomain information, extracting more discriminative features relevant for lesion segmentation. To enhance channel-wise feature dependencies, the convolution output is refined using a 3D Squeeze and Excitation (SE) block [Hu et al., 2018]. This recalibrates the feature maps by dynamically weighting the channels based on their importance, ensuring that critical features are emphasized. Finally, to enable patches frommultiple stages to focus on the corresponding regions in the original embeddings, the embeddings are tokenized by reshaping features from different scales into token sequences with patch sizes P 2e-1 [Wang et al., 2024a]. The channel dimensions are all changed to C = 128. Then, a spatial embedding module is used to capture positional information, improve spatial awareness of the model, and output the token sequences of embeddings Se ∈ RV ×C , V = DHW P 3 .

3.2 Mamba-Enhanced FusionModule

As shown in Fig. 4, MEFM is inspired by the working mechanisms of horizontal cells and bipolar cells in the visual neural system, where the Center-Surround Antagonism [Ankri et al., 2020] enables edge detection and contrast enhancement within the visual cortex. By integrating attention mechanisms through Mamba, MEFM achieves global information correlation and enhances the contrast between lesions and surrounding healthy tissues.

MEFM integrates multi-branch embeddings effectively while preserving their unique characteristics. As illustrated in Fig. 2, MEFM consists of L stacked layers, each performing a sequence of operations to fuse and refine features from all branches iteratively. For each layer l ∈ [1, 2, ...,L], the embeddings from all branches are concatenated along the channel axis to form a unified feature representation.

where T l c ∈ RV×4C and e is the index of branches. This operation aggregates complementary information across branches, preparing the features for subsequent fusion.

The concatenated embeddings are passed through a Mamba layer, which plays a crucial role in dynamically learning and balancing frequency-specific weights to achieve complementary integration of features, enhancing the model's adaptability to diverse input characteristics. After fusion, the combined features are split and grouped to ensure that each branch ˆ T l i focuses on its specific representation while maintaining inter-branch relationships.

For each branch, its embedding ˆ T l i ∈ Rl×C is treated as the query (Q), while the embeddings from the other branches ˆ T l j = concat( ˆ T lk |k ̸= i) ∈ Rl×3C serve as the key (K) and value (V). This attention mechanism allows each branch to focus selectively on relevant features from the other branches, enriching its representation through inter-branch interactions. The process is repeated for all branches, ensuring comprehensive attention-based feature refinement.

where W lQ,i , W lK,i , and W lV,i are learnable weight matrices for each lth layer and ith branch, and dli is the feature dimension for scaling.

Each branch's output from the attention mechanism is further refined using a SE block to recalibrate channel-wise dependencies dynamically and a Multi-Layer Perceptron (MLP) to enhance non-linear feature representation and improve discriminative ability.

This process is repeated for L layers, with the output of each layer serving as input to the next. The iterative refinement allows MEFM to progressively improve feature quality, ensuring robust fusion and representation.

3.3 Spatial-Guided Refinement Module

The SGRM is designed to restore the dimensions of the unified embeddings of each branch to the original input size, allowing seamless integration with the final reconstruction pipeline. As illustrated in Fig. 2, SGRM employs an MLP to project the feature representation of each branch, T L e . Then, using a 3D transposed convolution operation, the refined embeddings are reshaped and expanded to their original spatial dimensions. This operation reconstructs the volumetric structure of the input image by reversing the patch embedding and tokenization process.

where Oe represents the final output of the eth branch.

This module ensures that the spatial integrity of the original input image is preserved while maintaining the enhanced feature representation learned during the earlier stages. By directly coupling embedding refinement with spatial reconstruction, SGRM effectively bridges the gap between the unified branch embeddings and the final output.

4 Experiments

4.1 Datasets

This study conducted an in-depth analysis using multiple datasets across diverse domains to evaluate DVNet's robustness and adaptability. These datasets include both 2D and 3D

imaging modalities, ensuring a comprehensive assessment of our model's performance.

The BraTS 2020 and BraTS 2023 datasets [Menze et al., 2014; Kazerooni et al., 2024; Henry et al., 2021] are widely recognized benchmarks for brain tumor segmentation tasks. These datasets provide 3D brain MRI volumes with precise ground truth annotations, including four modalities for each volume: T1-weighted, T1-contrast enhanced, T2-weighted, and FLAIR. Ground truth annotations include three segmentation targets: Whole Tumor (WT), Enhancing Tumor (ET), and Tumor Core (TC). The BraTS 2020 & 2023 datasets provide foundational and diversity benchmarks, challenging the model's adaptability to varying imaging conditions and glioma types. For a fair comparison, we follow the evaluation protocol outlined in SegMamba [Xing et al., 2024] and utilize public implementations of the baseline methods in comparison to retrain their networks under the same settings for BraTS 2020 & 2023 datasets, respectively. The data is split randomly: 70% of the 3D volumes for training, 10% for validation, and the remaining 20% for testing.

In addition to 3D segmentation, DVNet was also evaluated on the Abdomen MRI dataset [Ji et al., 2022] and Microscopy Cell Dataset [Ma et al., 2024b] for 2D segmentation tasks. The Abdomen MRI dataset was used for the MICCAI 2022 AMOS Challenge [Ji et al., 2022], designed for segmentation of abdominal organs. Microscopy Cell dataset was used for the NeurIPS 2022 Cell Segmentation Challenge [Ma et al., 2024b]. In accordance with the settings of xLSTMUnet [Chen et al., 2024b], we employed 60 labeled MRI scans for training and 50 scans for testing in the Abdomen MRI dataset and 1,000 images for training and 101 images for evaluation in the Microscopy Cell dataset.

4.2 Implementation Details

DVNet is implemented based on SegMamba [Xing et al., 2024] for 3D tasks and xLSTM [Chen et al., 2024b] for 2D tasks. For 3D tasks, the input crop size is set to (64×64×64) with a batch size of 2. The optimization process uses crossentropy loss with a stochastic gradient descent optimizer and a polynomial learning rate scheduler (initial learning rate of 0.01 and decay of 1 × 10-5 ). Training runs for 1,000 epochs with data augmentations including brightness, gamma, rotation, scaling, mirror, and elastic deformation. For 2D tasks, the loss function is the sum of Dice loss and cross-entropy loss, optimized with the AdamW optimizer (weight decay:

0.05). Learning rates are set empirically for each dataset: Abdomen MRI (0.005) and Microscopy (0.0015). Batch size is set to 30 for Abdomen MRI and 12 for Microscopy. All models are trained for 1,000 epochs for 2D tasks.

4.3 EvaluationMetrics

The Dice Similarity Coefficient (DSC) and 95% Hausdorff Distance (HD95) are adopted for quantitative comparison on the BraTS 2020 & 2023 datasets. The DSC and Normalized Surface Distance (NSD) are used as evaluation metrics for Abdomen MRI dataset. The F1 score is used for cell segmentation on the Microscopy dataset.

4.4 Experimental Results

To evaluate the effectiveness of DVNet, we conducted comparisons in different scenarios with appropriate baselines.

Plug-and-play: To demonstrate the effectiveness of DVNet as a plug-and-play module, DVNet was plugged into four state-of-the-art (SOTA) models (SegResNet [Myronenko, 2019], UNETR [Hatamizadeh et al., 2022], SwinUNETR [Hatamizadeh et al., 2021] and SegMamba [Xing et al., 2024]). Their performance with and without DVNet was compared on the BraTS 2020 and BraTS 2023 datasets. As shown in Table 1, DVNet consistently improves segmentation accuracy (DSC) and boundary precision (HD95) over baseline models by improving feature extraction and addressing challenges such as blurry and low-contrast regions. For example, on BraTS 2023 dataset, DVNet improves the DSC score of SegResNet [Myronenko, 2019] from 92.02% to 93.82% for WT and reduces its HD95 from 4.07 to 3.64, demonstrating its ability to refine predictions and improve boundary delineation. Similarly, when integrated with transformer-based models like SwinUNETR [Hatamizadeh et al., 2021], DVNet achieves notable gains, with DSC of TC increasing from 87.79% to 92.22% and HD95 decreasing from 4.42 to 2.72. These findings validate the robustness and adaptability of DVNet, proving its capability to deliver SOTA performance across diverse datasets and clinical scenarios.

Comparison with SOTA methods for the 3D segmentation task: We also evaluated the performance of DVNet against other SOTA models reported in SegMamba [Xing et al., 2024] on the BraTS 2023 dataset. The results in Table 2 demonstrate the superiority of DVNet over all baseline models for brain tumor segmentation as the DVNet achieves the highest DSC scores and the lowest HD95 values among all methods compared, especially excelling in challenging subregions like ET and TC. Compared to the strongest baseline, SegMamba [Xing et al., 2024], DVNet further enhances segmentation accuracy and improves boundary delineation, particularly for the challenging ET subregion.

Comparison with SOTA methods for 2D segmentation tasks: Table 3 compares the segmentation performance of DVNet with SOTA models for 2D medical images. On the Abdomen MRI dataset, the DVNet integrated with the xLSTM enc architecture [Chen et al., 2024b], slightly exceeds xLSTM enc and achieves the highest performance, with a DSC of 77.58% and an NSD of 83.84%, demonstrating the effectiveness of our plugin-based approach in enhancing segmentation accuracy and boundary precision. For the Microscopy dataset, DVNet again outperforms other SOTA models, achieving an F1 score of 65.54%, a more impressive improvement over xLSTM enc (60.36%). This demonstrates the robustness of DVNet in handling complex instance segmentation tasks.

In summary, experimental results have demonstrated the effectiveness of DVNet as a plug-and-play module, consistently enhancing segmentation accuracy and boundary precision for both 3D and 2D tasks.

4.5 Ablation Study

Ablation studies have been conducted to demonstrate the effectiveness of different modules in DVNet. Table 4 compares the contributions of the WDPM and MEFM in the proposed DVNet (plug-in SegResNet [Myronenko, 2019]) on the BraTS 2023 dataset.

Removal of WDPM or MEFM results in notable performance degradation with a decrease in DSC score and an increase in HD95, particularly for the more challenging TC and ET subregions. The results indicate reduced segmentation precision and boundary accuracy, underscoring the critical roles of both WDPM and MEFM in boundary refinement and multi-region interaction. In contrast, the full DVNet achieves the best overall performance.

To further explore the contributions of different frequency signals in medical image segmentation, an ablation study was conducted to study the effect of different SWT components. The results shown in Table 5 demonstrate that neither lowfrequency nor high-frequency components alone are sufficient for optimal segmentation performance. Low-frequency components (LLL) excel in capturing global structural information but lack the details required for precise boundary delineation. In contrast, high-frequency components (HHH) focus on local details but fail to provide the broader context necessary to segment complex structures and are also susceptible to noise.

Overall, the ablation study highlights the importance of both WDPM and MEFM in enhancing the segmentation performance of DVNet. WDPM strengthens feature representation and target structure recognition; while MEFM improves boundary precision and inter-region consistency. In addition, the full SWT-based DVNet effectively integrates both lowfrequency and high-frequency information, leveraging their complementary strengths to achieve superior performance. This highlights the importance of multi-frequency analysis in medical image segmentation tasks, demonstrating DVNet's efficacy in tackling complex medical image segmentation tasks.

4.6 Visualization

To explore the effectiveness of DVNet, we visualize the outputs of DVNet. We randomly selected some images and performed visualization in the inference mode on the BraTS 2023 test dataset. Fig. 5 illustrates the intermediate and final segmentation results of the proposed DVNet for some representative cases. Each row corresponds to a single case, and the columns represent (from left to right): the input image, the outputs of the four branches (O1 to O4), the final prediction by DVNet, and the ground truth (GT) segmentation mask. The branch outputs highlight the model's capability to capture features at multiple levels, providing complementary information for the final prediction. The final DVNet predictions demonstrate strong alignment with the ground truth, effectively capturing the boundaries and internal details of the target regions. However, subtle discrepancies in tiny structures indicate potential areas for further improvement. Overall, the visualization results validate the robustness and precision of the proposed multi-branch segmentation approach in addressing complex segmentation tasks.

5 Conclusion

In this work, we are motivated to incorporate the principles of scotopic vision into noninvasive medical image processing and analysis tasks, with the aim of inspiring future research endeavors. Specifically, we introduced DVNet, a novel plug-and-play dark vision enhancement method that teaches models to fully capture and learn key visual features in blurry medical images. By utilizing the multi-scale frequency information obtained from wavelet transform, DVNet can identify lesions in low-contrast areas, using them as attention indices for dense prediction tasks. Our approach refines these indices, constructing global dependencies of visual features in the manner of potential conduction in the optic nerve, reactivating blurred areas and thereby achieving precise segmentation of lesion regions. Extensive experiments demonstrate that DVNet achieves SOTA performance in various medical image segmentation tasks, highlighting its effectiveness and generalizability.

Acknowledgments

This work is supported by grants from the National Natural Science Foundation of China (NSFC 62322215, 62402488), National Science Foundation (MRI-2018966) and South Carolina Research Authority (2331-241-2024811). This study was supported in part by the high-performance computing center of Central South University.