CMFS: CLIP-Guided Modality Interaction for Mitigating Noise in Multi-Modal Image Fusion and Segmentation

1 Introduction

Infrared-visible image fusion (IVF) and multi-modal semantic segmentation (MMSS) are critical for robust scene understanding, especially under challenging conditions such as low light or adverse weather. IVF combines thermal and visible images to provide a comprehensive view, enhancing applications like night-time driving assistance and safety surveillance [Zhao et al., 2023; Yi et al., 2024]. Concurrently, MMSS assigns semantic labels to each pixel by leveraging information from multiple sensors, thereby improving contextual and structural scene interpretation essential for autonomous driving, robotics, and surveillance [Lv et al., 2023].

∗Corresponding author

Figure 1: Visual comparison of image fusion and semantic segmentation under low-light conditions. Our method effectively perceives faint thermal objects with the guidance of CLIP, while the interaction between modal features profoundly suppresses background noise, highlights salient objects, and maintains robust performance.

Despite their importance, current approaches often treat IVF and MMSS as separate tasks, resulting in challenges related to high noise, modality inconsistencies, and inefficient cross-modal interactions. These limitations compromise both the visual quality of fused images and the accuracy of semantic segmentation. Even joint modeling attempts [Zhang et al., 2024a] that use attention-based frameworks tend to be computationally intensive and sensitive to noise or imbalanced inputs. As a result, they may struggle with fine structures, elongated object shapes, and complex backgrounds, leading to incomplete scene interpretations and suboptimal fusion.

Jointly addressing IVF and MMSS remains challenging due to several factors. First, the domain gap between thermal and visible images can introduce significant noise, particularly under adverse conditions. At the same time, simple fusion strategies might propagate irrelevant or distorted information by treating all regions equally. Second, semantic segmentation requires precise pixel- and region-level cues. However, noise and inconsistencies arising during fusion can hinder accurate pixel categorization. Finally, the high computational complexity inherent in many attention-based methods limits their real-time applicability. These challenges underscore the need for a robust approach to adaptively isolate inter-modal interference, preserve critical features, and efficiently model long-range dependencies.

To address these challenges, we propose a unified multitask learning framework that simultaneously performs IVF and MMSS. Our approach begins with a Multi-modal Interaction Alignment (MIA) module, which employs a bidirectional VMamba-based encoder with an additional shuffle layer to obtain more robust features, enhancing unimodal representations and foreground saliency. This encoder partitions features and exploits CLIP's powerful classification priors using category-specific text prompts to dynamically guide region localization, as Figure 1 shows. Building on this, we introduce a regionally constrained hybrid attention (Hybrid-Att) module that selectively suppresses background noise and facilitates precise cross-modal feature interaction. Furthermore, we develop a Frequency-Space Collaboration (FSC) module to achieve joint multi-domain feature enhancement and denoising. The FSC module integrates waveletand spatial-domain features and leverages global Fourierdomain modeling to provide adaptive noise reduction and balanced feature allocation. Additionally, we employ a low-rank mixture-of-experts (MoE) with dynamic routing to enhance region-specific fusion and improve pixel-level accuracy while maintaining computational efficiency. The main contributions of this work can be summarized as follows:

• We propose a region-aware Modal Interaction Alignment module that combines a bidirectional VMambabased encoder-with an additional shuffle layer and a CLIP-guided multi-modal interaction block to emphasize target regions dynamically, enhance foreground saliency, and mitigate noise.
• We develop a Frequency-Spatial Collaboration module that leverages selective scanning and integrates wavelet-, spatial-, and Fourier-domain features to achieve adaptive denoising and balanced feature allocation.
• We introduce a low-rank MoE with dynamic to improve region-specific fusion and spatial pixel accuracy.

2 RelatedWork

2.1 Multi-modal Learning

Multi-modal learning integrates information from diverse modalities to improve performance, overcoming the limitations of single-modal approaches, such as context sensitivity and incomplete scene understanding. In this work, we mainly focus on RGB-T tasks, where collaboration between visible and thermal images may improve robustness in challenging conditions such as low light and adverse weather.

Multi-modal Image Fusion is a fundamental task that integrates relevant and informative features from multiple modalities to generate comprehensive fused images. Autoencoders pioneered the use of deep learning techniques in this field, with methods such as DenseFuse [Li and Wu, 2018] leveraging autoencoders for feature extraction and fusion through predefined strategies. To enhance fusion effectiveness, recent approaches [Liu et al., 2023; Zhang et al., 2024b] incorporate high-level visual tasks to provide richer semantic guidance. TextIF [Yi et al., 2024] further exploit textual cues to improve visual fidelity and perceptual robustness. However, existing multi-modal fusion methods [Zhao et al., 2024c] often fail to produce visually coherent images in low-light and noisy environments, revealing significant potential for improvement.

Multi-modal Semantic Segmentation is a dense prediction task that assigns category labels to each pixel by integrating information from multiple modalities to improve accuracy and contextual understanding. Early methods like MFNet [Ha et al., 2017a] used element-wise summation or concatenation for cross-modal feature fusion, often leading to redundancy and overlooking modality differences. To address these limitations, researchers introduced specialized fusion operations. For example, [Guo et al., 2021] used multi-level skip connections to enhance feature flow, while [Zhang et al., 2021] proposed a bridging-then-fusing strategy for multiscale feature fusion. Attention mechanisms later became pivotal, with [Deng et al., 2021] incorporating channel and spatial attention in the encoder and [Zhou et al., 2022] combining cross-modal features with boundary-aware supervision for refined outputs. Recent advances emphasize exploiting modality complementarity, such as [Zhou et al., 2023b]'s Transformer-based approach to leverage cross-modal correlations. However, challenges remain in fully exploring feature relationships, and interactions, and preserving spatial details.

2.2 Frequency Exploration

The Fast Fourier Transform (FFT) is a fundamental tool in frequency domain analysis, enabling efficient conversion of signals to a domain where global statistical properties are more accessible. FECNet [Huang et al., 2022] leverages Fourier feature amplitudes to isolate global lightness components, improving image clarity and aesthetics. Similarly, FSDGN [Yu et al., 2022] uses Fourier amplitudes as global haze indicators for image dehazing. [Kong et al., 2023] introduced a frequency-domain self-attention mechanism for efficient image deblurring, and [Zhou et al., 2023a] utilized the Fourier transform to model global dependencies in a novel backbone network for image degradation. While FFT has advanced many applications, its signal processing limitations highlight improvement areas. Complementing FFT, the Wavelet Transform offers multi-resolution analysis, effectively capturing localized signal variations.

2.3 State Space Models

State space models (SSMs) have gained attention for modeling long-range dependencies in sequences. The structured state-space sequence model (S4) [Gu et al., 2021] was introduced to capture these dependencies with linear complexity, followed by improvements in S5 [Smith et al., 2022], H3 [Fu et al., 2023], and Mamba [Gu and Dao, 2023]. Notably, Mamba outperforms Transformers in linear scalability on long-sequence NLP tasks due to its data-dependent selective state space mechanism and hardware optimization. The applicability of SSMs extends beyond NLP, with pioneering works applying Mamba to various tasks, including image classification [Zhu et al., 2024; Liu et al., 2024], image restoration [Guo et al., 2025] and others [Li et al., 2025].

3 Method

3.1 Preliminaries

State Space Models. State Space Models (SSMs) describe the dynamics of continuous systems by employing a method based on linear ordinary differential equations. They model the relationship between the input x(t) ∈ R and the output y(t) ∈ R through a latent state h(t) ∈ RN , which can be formulated as follows:

where A, B, C are state transition matrices. The discrete variants of SSMs, such as Mamba, incorporate a discretization step parameterized by a timescale ∆, transforming the continuous parameters A,B into their discrete counterparts A, B through the zero-order hold (ZOH) method, expressed as:

In addition, Mamba [Gu and Dao, 2023] incorporates a selective scanning mechanism that dynamically adjusts to the input, enabling it to capture long-range contextual dependencies across different regions. This capability makes it particularly valuable for tackling complex computer vision tasks.

3.2 Overall Framework

As shown in Figure 2, CMFS is a multi-task framework that jointly tackles infrared-visible image fusion and semantic segmentation. It takes a visible image I vi ∈ RH×W×3 and thermal image I t ∈ RH×W×3 , where H, W denote the height and width, respectively. To enhance the feature representations, CMFS progressively extracts and refines multi-modal features over n scales. At each i-th scale, VMamba-based encoders [Liu et al., 2024] Evi i and Et i , which incorporate a bidirectional Mamba module with an additional shuffle layer to obtain more robust features, extract and refine unimodal features Φvi i and Φti .

In extremely dark environments, both modalities often fail to capture meaningful details for objects lacking thermal radiation, resulting in severe noise and potentially introducing additional interference after interaction. To address this, the Multi-Modal Feature Interaction (MMFI) block leverages CLIP's [Radford et al., 2021] category prompts and a hybrid pooling-based attention mechanism to assign regionspecific weights, thereby suppressing background noise while facilitating robust inter-modal interactions. Subsequently, the Frequency-Spatial Collaboration (FSC) module further eliminates residual and amplified noise introduced during crossmodal interaction by performing long-range modeling in both the spatial and wavelet domains. Afterward, the features are sent to the Multi-Modal Feature Enhancement Module for pixel-level feature representation learning. The resulting denoised features are then fed into the fusion head to produce the final fused image F. Simultaneously, these fused features pass through a ResMoE block to extract fine-grained details, which are then fed into the segmentation head to generate the final semantic segmentation results Is.

3.3 Multi-modal Interaction Alignment Module

RGB images provide texture information, while thermal images highlight crucial boundary cues. Fully leveraging intramodal self-reinforcement and aligning inter-modal interactions is essential. To this end, the Multi-modal Interaction Alignment (MIA) module achieves this by combining bidirectional Mamba (BM) blocks and a MMFI block [Zhang et al., 2024c] shown in Figure 3. Each BM block first updates the unimodal feature {Φvi ,Φt } to {Υvi ,Υt } through unimodal self-reinforcement and contextual modeling. The MMFI block then uses CLIP-based category prompts to guide region-level weighting, ensuring discriminative cross-modal interactions while suppressing background noise. Formally,

(3)

where Θi represents the output of related features. Bidirectional Mamba Block. To enhance the intrinsic quality of unimodal features, we adopt a ShuffleNet-inspired approach [Zhang et al., 2018] that reorders channel arrangements. This method improves feature diversity and strengthens intra-modal representations. Additionally, spatial perceptual understanding is achieved through sequential forward and backward scanning, leveraging SSM to model intramodal relations and effectively refine and integrate critical contextual information within each modality. This process can be formulated as:

(4)

Multi-Modal Feature Interaction (MMFI) Block. The MMFI block is designed to integrate visible and thermal features under challenging conditions. First, each unimodal feature is partitioned into non-overlapping patches to allow localized processing. Within each patch, channel attention rescales channel responses based on global pooling descriptors, while spatial attention pinpoints crucial pixel-level locations. A learnable gating mechanism then adaptively fuses signals from both modalities, preserving complementary cues while mitigating noise. To further reinforce fusion, CLIP's

text encoder provides category guidance by transforming a prompt (e.g., The photo of a person) into text features. Each patch is compared with these text features, and patches with higher similarity scores, likely containing the target, receive stronger cross-modal attention. Finally, channel-spatial attention weights, gating outputs, and the CLIP-based attention map are combined to yield the updated unimodal features. The final update for each modality is defined as:

where Re(·) indicates a reshape operation, and Wi are learnable attention weights, and M denotes the CLIP-based attention map. This synergy between localized attention, crossmodal gating, and category prompts enables robust fusion of visible and thermal streams, effectively suppressing noise and emphasizing patches most likely to contain target objects.

3.4 Frequency-Spatial CollaborationModule

Multi-modal interactions may introduce inconsistencies and noise that degrade fusion quality. As illustrated in Figure 4, the proposed FSC module addresses these by jointly modeling feature representations in both the wavelet and spatial domains, thus mitigating noise and enhancing the overall quality of the fused output. The FSC module comprises two main components: the Hybrid Mamba Compensation Conversion (HMCC) Block and the Frequency Denoising (FD) Block.

Hybrid Mamba Compensation Conversion (HMCC) Block. The HMCC block operates in two parallel branches: one for wavelet-domain modeling and the other for spatialdomain modeling. Θx (with x ∈ {vi , t}) denote the output of the related features. We first modulate Θx with a 1×1 convolution, then apply a second-order discrete wavelet transform (DWT) to decompose the features into multiple frequency bands (LL, LH, HL, HH). A state space model with local 2D scanning [Huang et al., 2024] refines these frequency bands. Finally, an inverse wavelet transform (IWT) reconstructs the wavelet-domain features [Zou et al., 2024]:

where LN(·) denotes LayerNorm. In the spatial branch, we capture both local and global relations. Specifically, we first apply a vanilla 2D scanning within SSM to model long-range spatial correlations, then a channel attention (CA) operation to enhance local consistency. Formally,

where s1, s2 ∈ RC are learnable scale factors controlling the flow of skip information. Both outputs, Θxwav and Θxsp , are then concatenated as follows:

Frequency Denoising (FD) Block. To further suppress mixed noise and refine the multi-domain features, we employ a Frequency Denoising (FD) block. After concatenation, we apply a Fast Fourier Transform (its degeneration-separation characteristics were validated in the supplementary materials.) to Θxfreq-init , decomposing it into amplitude and phase components, Ampx and Phax . A denoising convolution filters out noise in the amplitude domain, then an inverse FFT (IFFT) reconstructs the signal back to the spatial domain:

where DConv(·) is a depth-wise convolution that adjusts the channel count to match the module's input dimension, and DeConv operation involves continuous pixel-wise convolution followed by the LeakyReLU activation function. Finally, we apply a simple channel attention (SCA) mechanism:

where Θxfreq-mid indicates an intermediate representation used for channel-attention recalibration. The result Θxfreq preserves both global structure and local detail, ensuring a robust feature representation suitable for subsequent tasks. This wavelet-spatial modeling and frequency-domain denoising fully leverage multi-domain information to suppress noise, retain crucial details, and yield semantically fused features. The denoising capability of the FSC module was further validated through the Gaussian color denoising task in the supplementary materials, demonstrating encouraging performance.

3.5 Multi-Modal Feature Enhancement Module

Semantic segmentation demands precise pixel-level classification, with different regions requiring distinct semantic cues. Prior approaches [Zhang et al., 2024a] often employ selfand cross-attention mechanisms for cross-modal fusion, incurring high computational costs. To mitigate this, we propose a low-rank mixture-of-experts (MoE) mechanism that efficiently encodes fused features while eliminating redundancy. First, frequency-domain features Θvi freq and Θtfreq are fused in a Channel Embed Block via residual convolution to produce the initial fused feature ˆofused:

where [·] is concatenation, CE(·) is a channel embedding module composed of three convolution layers and a normalization layer, and BN(·) denotes BatchNorm. Next, low-rank experts refine the fused features by compressing the feature space via low-rank decomposition, reducing computational complexity and filtering out redundancy. Jointly leveraging diverse subspace expressions to simplify the fused feature representation [Zamfir et al., 2024]. Dynamic routing activates the most relevant experts for specific regions, enhancing semantic representations. The detailed refinement process is provided in Algorithm 1 in the supplementary materials, with the final output denoted as ofused. Finally, for object categories with elongated shapes or intricate spatial continuity, large-kernel strip convolutions (following Hou et al. [Hou et al., 2024]) are applied to refine fine-grained features, thereby improving the segmentation of strip-shaped targets.

3.6 Training Objective

To generate a high-quality fused image If , we integrate a Gradient Residual Dense Block into the fusion head to directly produce a three-channel image with enhanced textures Figure 5: Qualitative segmentation results on the MFNet dataset.

and without YCbCr distortions. A color loss Lcolor aligns the fused CbF and CrF with the gamma-corrected channels of Ivi, while contrast stretching β on the thermal image improves gradient and intensity alignment. The fusion loss is defined as

For semantic segmentation, we use the cross-entropy loss,

where P is the ground truth. The overall loss is

4 Experiments

4.1 Experimental Setting

Datasets. We evaluate the performance of CMFS on semantic segmentation and image fusion tasks using the MFNet [Ha et al., 2017b], PST900 [Shivakumar et al., 2020], and FMB [Liu et al., 2023] datasets. Specifically, these datasets comprise 1569, 1038, and 1500 paired infrared and visible images, with resolutions of 480 × 640, 720 × 1280, and 600×800, respectively. Among these, 393, 288, and 280 image pairs are used for testing.

Implementation Details. The semantic segmentation and image fusion tasks are trained jointly for 500 epochs to enable effective multi-task learning. The training process adopts an initial learning rate of 6 × 10-5 , a batch size of 4, and the Adam optimizer with a weight decay coefficient of 0.01. During training, standard data augmentation techniques such as horizontal flipping, random scaling, and cropping are applied. All training images are cropped to a uniform size of 480×640. Refining coefficient value settings in the loss function: λint = 0.5, λgrad = 0.2, λcolor = 1.0, λfus = 0.1, λseg = 1.0. As shown in Figure 2, we adopt a four-stage Vmamba-tiny encoder, which operates across four distinct scales. All experiments are conducted on the NVIDIA L40 GPU with 46GB of memory, coupled with an Intel Xeon Silver 4310 10-Core Processor CPU.

Figure 7: Visual comparison of image fusion on the MFNet dataset.

4.2 Multi-modal Semantic Segmentation

To ensure a fair comparison, we primarily used data from [Zhang et al., 2024a] and quantitative results reported in the respective papers, with visualizations sourced from their open-source code. Since most methods were not trained on the FMB dataset, we compared only models with publicly available optimal weights. We used mean Intersection over Union (mIoU) as the evaluation metric, which effectively reflects segmentation performance in complex scenes while mitigating the impact of class imbalance. As Figure 5 and Figure 6 shown, our method demonstrates precise segmentation of target categories in both day and night conditions, with accurate classification of edge pixels, aided by the MMFE module. It performs consistently across categories and handles ambiguous pixels effectively, even in low-light settings. For example, in Figure 5, our method delineates the contours of objects like bikes and bumps with high accuracy, achieving realistic visual results. In contrast, other methods either approximate the regions or miss features entirely. The quantitative results in Table 1 demonstrate the superiority of our method. It achieves the highest mIoU on all three datasets, with a maximum gain of 5.1% over the second-best method. To this end, we substituted the PC-Att module in the stateof-the-art MRFS model with our MMFE module, adhering strictly to its original training settings. After training for the same number of epochs on the FMB dataset, we observed a significant improvement in segmentation performance, with the mIoU increasing from 61.2 to 63.7. Additionally, the GFLOPs were reduced from 219.13 to 177.65, demonstrating a substantial reduction in computational complexity. Overall, quantitative and qualitative evaluations indicate that CMFS performs well in semantic segmentation.

4.3 Multi-modal Image Fusion

We evaluate the image fusion performance of CMFS on the MFNet and FMB datasets using metrics such as standard deviation (SD) [Aslantas and Bendes, 2015], entropy (EN), and mutual information (MI) [Qu et al., 2002] to assess image detail preservation, multi-modal information integration, and retention of complementary information. Visual results in Figure 7 and 8 show that CMFS achieves high visual fidelity, effectively removing noise even in low-light conditions. As shown in Figure 7, CMFS avoids the style bias in nighttime infrared images and effectively restores texture details like road traffic signs. Figure 8 highlights its ability to eliminate modality misalignment artifacts, ensuring sharper object boundaries. The FSC module further enhances texture, sharpens edges, and aligns the output with human perception. Quantitative results in Table 2 confirm CMFS's good performance, achieving encouraging scores across most metrics compared to state-of-the-art methods.

4.4 Ablation Studies

We conduct comprehensive ablation studies to assess the effectiveness of the specific design choices in our method, systematically evaluating six variants. Baseline Model: removes all components of our design, retaining only the fusion and semantic segmentation heads, and uses residual convolutional networks (Channel Embed) to fuse features from the two modalities. Model I: removes the Bimamba block to evaluate the impact of intra-modal feature self-reinforcement. Model II: omits the feature partitioning step and CLIP regional importance guidance in the MMFI module, keeping only the hybrid attention mechanism with gated correction. Model III: discards the FSC module to highlight the contribution of multi-domain joint modeling for denoising. Model IV: replaces the MMFE module with a simple single-layer 1x1 convolution to demonstrate the necessity of feature refinement and spatial modulation. Model V: removes the immodel consistently achieves accurate segmentation of objects across different categories, even in smoke-filled environments.

age fusion head to assess the mutual enhancement between the two tasks. Based on the controlled variable analysis presented in Table 3, it is evident that removing any individual module from the full model results in a decrease in segmentation performance, indicating that each of the designed modules contributes positively to overall segmentation accuracy. In comparison to the baseline, the Model IV and V configurations show lower performance, highlighting the importance of initial fusion via the residual network (Channel Embed Block) and the synergistic interaction between multimodal fusion and segmentation. Notably, the performance drop is most pronounced upon removal of the ResMoE Block, reinforcing the value of utilizing a MoE mechanism to refine features and eliminate redundancy, which provides a novel approach for enhancing segmentation performance. Visual comparisons in Figure 9 support these findings by introducing another extreme environment of smoke, confirming that our framework effectively meets the objectives of multi-modal segmentation and fusion tasks and demonstrates broad applicability. Additionally, the visual results in Figure 10 also substantiate the positive contribution of the designed modules to image fusion fidelity. The conclusion can be drawn by comparing the distribution and contrast of the surrounding artifacts of the person in the image. These results emphasize the effectiveness of these designs in improving image fusion and highlight the positive impact of semantic segmentation.

5 Conclusion

In this work, we presented a unified framework for infraredvisible image fusion and multi-modal semantic segmentation that effectively tackles noise, modality inconsistencies, and inefficient cross-modal interactions. Our approach improves both fusion quality and segmentation accuracy through robust feature extraction, CLIP-guided alignment, and adaptive denoising, as validated by extensive experiments that outperform state-of-the-art methods.

Acknowledgements

We thank the reviewers for their insightful comments. This work was supported by the National Natural Science Foundation of China (No. 62172126), and the Shenzhen Research Council (No. JCYJ20210324120202006).