SpeciFuse: Learning Degradation-Type Specificity for Robust Infrared and Visible Image Fusion Under Composite Degradations

1 Introduction

Infrared-visible image fusion (IVIF) is a prominent subfield in multimodal image fusion research [Zhang and Demiris, 2023]. While visible images provide texturally rich details consistent with human vision, they are vulnerable to poor lighting and camouflage. Infrared images highlight thermal targets and penetrate obscurants like smoke and darkness, but they typically suffer from low spatial resolution and are more susceptible to noise. Therefore, IVIF aims to generate a single image that combines these complementary information to achieve more robust visual perception [Ma et al., 2019a]. Furthermore, the fused images significantly improve the performance of downstream computer vision tasks such as object detection, tracking, and segmentation [Li et al., 2022; Chen et al., 2025; Zhao et al., 2024; Li et al., 2025c].

Although deep learning has revolutionized IVIF by offering superior performance, most existing methods assume ideal input conditions. Their performance thus significantly deteriorates when source images are degraded. This limitation has motivated growing interest in integrating image restoration with fusion processes [Guo et al., 2024; Wang et al., 2025] to enhance robustness in real-world scenarios, as shown in Figure 1. Single-degradation data refers to images that are corrupted by only one type of degradation (e.g., low light, blur or noise). Composite-degradation data consists of images that are affected by multiple degradation types. Methods trained on single-degradation data [Yi et al., 2024; Tang et al., 2025b] simply aggregate the capacity to address each isolated degradation, and thus fail to handle images with composite degradation. Methods trained on compositedegradation data [Xue et al., 2024; Tang et al., 2024; Zhang et al., 2025] learn holistic mapping from fixed degradation combinations, resulting in limited generalization to unseen composite degradation scenarios. The neglect of the intricate coupling and mutual interference among distinct degradation types hinders aforementioned methods from discerning degradation types and correspondingly suppressing them in the fusion process, thereby inevitably leading to suboptimal fusion results under diverse composite degradations.

To overcome the above limitations, we propose SpeciFuse, an IVIF network designed to explicitly model and address conflicts arising from composite degradations, thereby achieving robust image fusion that generalizes effectively to various combinations of multiple degradations. As shown in Figure 2, SpeciFuse employs a siamese architecture that has multiple parallel fusion pipelines to learn the specificity of individual degradation type and to adaptively modulate fusion strategy. During training, degraded images are first generated from paired high-quality source images by using distinct single-degradation operations. Each fusion pipeline takes one degraded image and the high-quality image from the other modality as the input. All fusion pipelines share the same network architecture and weights, and their outputs are jointly optimized to train the SpeciFuse model.

To capture the specificity of individual degradations, we propose degradation-type specificity decoupling modules (DSDM), which establish interactions between the encoders of degraded modality in two pipelines, as shown in Figure 3. The DSDM projects the intermediate features of encoders into a latent space and computes their cross-correlation matrix. It constrains these matrices to an approximately diagonal form, where larger diagonal elements ensure the robust representation of shared content, and smaller off-diagonal elements compel each feature channel to specialize in encoding characteristics unique to individual degradation. Through this strategy, SpeciFuse learns to disentangle degradation-specific features, thereby distinguishing the existing types of degradation from intricate coupling while preserving content fidelity under diverse composite degradations.

Furthermore, we propose a degradation-aware gated fusion module (DGFM) to dynamically modulate the fusion process according to the decoupled degradation-type-specific features, as shown in Figure 3. Specifically, the DGFM first derives a degradation attention map from the encoded features. Capturing the characteristics of the prevailing degradation types, this map is then used to generate channel-wise and spatial-wise gating weights that directly modulate the fusion of cross-modal features. Through this degradationconditional gating mechanism, SpeciFuse reliably generates high-quality fused images even under diverse and challenging degradation conditions. The contributions of this paper are summarized as follow:

• We propose SpeciFuse, an infrared-visible image fusion network that consists of multiple fusion pipelines based on a siamese architecture. It is designed to explicitly learn the specificity of individual degradations and adaptively modulate the fusion process, enabling robust generalization to composite degradation scenarios while being trained on single-degradation conditions.
• Degradation-type specificity decoupling modules are designed between two pipelines to drive the crosscorrelation matrix of their encoded features toward a diagonal form. This strategy facilitates clear discernment of degradation-type-specific features from composite degradations, thus eliminating mutual interference while preserving content fidelity.
• A degradation-aware gated fusion module is proposed to dynamically modulate the fusion strategy in both channel and spatial dimensions based on the decoupled degradation-type-specific features. It establishes a dynamic fusion strategy that generates high-quality fusion results under diverse scenarios.

2 RelatedWorks

2.1 Deep Learning-based Image FusionMethods

With the advancement of deep learning, deep learning-based methods [Tang et al., 2025a; Li et al., 2025a; Liu et al., 2024;

Tang et al., 2025c; Li et al., 2025b] have become the mainstream in IVIF. They can be broadly categorized into autoencoder-based, GAN-based, and end-to-end approaches.

Auto-encoder-based methods typically employ an encoderdecoder framework. DenseFuse [Li and Wu, 2018] pioneers this direction by extracting multi-scale features, but its fusion strategy is oversimplified. Xu et al. [Xu et al., 2021] later proposed a learnable adaptive fusion strategy to overcome handcrafted rules, though at increased computational cost. GAN-based methods formulate fusion as an adversarial game. FusionGAN [Ma et al., 2019b] is a pioneering work in this field. Ma et al. [Ma et al., 2020] introduced a dual-discriminator architecture to better balance multimodal information. However, these methods often suffer from training instability and mode collapse. End-to-end methods have evolved to implicitly handle the entire fusion pipeline. Zhang et al. [Zhang et al., 2020] refined the loss function to guide information distribution. PIAFusion [Tang et al., 2022a] introduced illumination-aware and differentialaware mechanisms. CDDFuse [Zhao et al., 2023] proposed a correlation-driven dual-branch architecture. SwinFusion [Ma et al., 2022] utilized a Swin Transformer to model long-range dependencies. YDTR [Tang et al., 2022b] designed a Y-shape dynamic Transformer for multi-task generalization.

2.2 Degradation-Resistant Image Fusion

To address information loss in complex scenes, degradationresistant IVIF methods have been developed. Initially, researches focused on low-light conditions. Lei et al. [Lei et al., 2024] developed a method specifically for nighttime scenarios. DIVFusion [Tang et al., 2023] jointly addresses lowlight enhancement and fusion in an end-to-end framework. LENFusion [Chen et al., 2024] integrates a feedback mechanism for joint low-light enhancement and fusion.

As real-world degradations extend beyond low-light, research shifted toward handling multiple degradations. TextIF [Yi et al., 2024] employs a text-interactive approach to dynamically adjust fusion based on degradation descriptions. ControlFusion [Tang et al., 2025b] integrates language-vision prompts for controllable and adaptive fusion.

More recent works aim for automated adaptation to composite degradations. TSDM [Xue et al., 2024] employs a teacher-student framework using synthesized degraded images. DRMF [Tang et al., 2024] leverages diffusion models to aggregate cross-modal priors during the restoration and fusion process. DAF [Wang et al., 2025] learns degradation recovery implicitly by constructing dynamic negative and invariant positive samples. OmniFuse [Zhang et al., 2025] couples restoration and fusion in a diffusion model and enables language control. However, the neglect of the intricate coupling and mutual interference among distinct degradation types hinders these methods from generate high-quality results under diverse composite degradations.

3 Method

The architecture of the proposed SpeciFuse is shown in Figure 2. It employs multiple fusion pipelines within a siamese architecture, each dedicated to processing images of one specific degradation. For example, in a pipeline processing a low-light degraded visible image, the corresponding infrared input remains in its original high-quality state. The total number of pipelines N is defined as the sum of the degradation types from both modalities (i.e., N = Nvi + Nir). Each pipeline contains feature encoders for both modalities, a degradation-aware gated fusion module, and a decoder for reconstructing the fused image. The corresponding components share weights across all pipelines.

3.1 Degradation-Type Specificity Decoupling Module

Existing degradation-resistant image fusion methods suffer from limited generalization capabilities due to the intricate coupling and mutual interference among distinct degradations. To address this, we propose degradation-type specificity decoupling module (DSDM) to enforces the separation of feature representations from different types of degradation during encoding, thereby preventing feature interference that would otherwise compromise the fusion quality, as demonstrated in Figure 3.

Inspired by the Barlow Twins [Zbontar et al., 2021], we propose a novel degradation-type specificity decoupling module to address this fundamental limitation. It explicitly establishes interaction between differently degraded images with same content information, and guides the network to decouple degradation features by decorrelating their feature representations in the latent space (Figure 4). This strategy enforces each degradation type to be encoded in distinct feature dimensions. It enables clear discernment of degradations from mutual interference and preserves high-fidelity content within the encoded features, thus facilitating the capability of handling composite degradations.

Specifically, during the training phase, for a paired of differently degraded images derived in same modality, the output features of instance normalization [Huang and Belongie, 2017] blocks Fd are processed through two successive projection operations. The flattened features of the i-th degradation F i flat are obtained by two projectors and a convolution layer.

For a pair of flattened features of differently degraded images, they describe identical scene content but carry distinct types of degradation. Thus a degradation decoupling loss is computed based on the cross-correlation matrix of two flattened features. The diagonal elements are maximized to ensure consistent representation of objects and scene content across varying degradation conditions. Simultaneously, the off-diagonal elements are minimized to force each dimension to specifically encode features of particular degradation and thus obtain their distinctive features. The total degradationtype specificity decoupling loss is expressed as:

where Lspe denote the decoupling loss in single modality. It can be described as:

where i and j represent the i-th and j-th degradation, respectively. C is the cross-correlation matrix and Cm,m denotes the matrix element at position (m,m). λ is a weighting hyperparameter that balances the contributions of the two loss components. We set λ = 0.0051, which is consistent with the Barlow Twins implementation.

3.2 Fusion Pipline

Image Encoder

The proposed network employs separate infrared and visible encoders with identical architectures but non-shared weights, as shown in Figure 5. Due to the statistical characteristics of different degradations may exhibit significant variations, which makes optimization difficult, an instance normalization module is utilized to perform rough alignment of lowlevel statistical features for input images. Meanwhile, since the normalization process inevitably causes information loss that may lead to information loss, a parallel feature extraction branch is constructed based on residual blocks [He et al., 2016] to ensure complete transmission of source image information to the fusion module. The features extracted from two paths are concatenated and sent to the fusion module.

Specifically, the instance normalization block (INB) is performed as follows:

where µ (·) and σ (·) represent the mean and standard deviation of each channel, γ and β are learnable parameters. The output of INB is degradation feature Fd.

The residual feature extraction branch (REB) comprises three sequentially connected residual blocks. The encoded feature Fe are the concatenation of Fd and output features of the REB. The degradation-aware attention map A is calculated from Fe.

Degradation-Aware Gated Fusion Module

Furthermore, to adaptively adjust fusion strategies according to decoupled degradation-type-specific feature, we propose a degradation-aware gated fusion module that adaptively adjusts the fusion strategy in both spatial and channel dimensions based on the previously computed degradation-aware attention maps, as shown in Figure 5.

Specifically, it first compute degradation attention maps from the encoded feature in DAB. And we concatenate the DRMF

DAFusion OmniFuse Ours

Figure 6: Qualitative comparison results in practical scenario.

encoded features Fe of both modalities along the channel dimension and sum their corresponding degradation-aware attention maps A to obtain FE and A. Subsequently, channel attention and spatial attention are computed from FE, A and the inversion of A. Then element-wise multiplication is preformed on these components to generate the gating weights G for FE. Finally, the fused feature Ff is obtained by elementwise multiplication between G and FE.

Fusion Decoder and Loss

The fusion decoder D consists of upsampling and downsampling operations with a residual block interleaved between them, which transforms the fused feature representation Ff into the output fused image If . The loss functions are computed between If and high-quality source images I gt vi and I gt ir .

The max intensity loss Lmax calculates the intensity discrepancy between fused image and the supremum of corresponding source images. It is expressed as:

where H and W represent the height and width of the image, max (·) denotes the element-wise maximum selection.

The structural similarity (SSIM) [Wang et al., 2004] loss function quantifies structural similarity between the fused image and source images by evaluating luminance, contrast, and structural patterns. It is defined as:

The gradient loss computes the loss of higher-intensity edges between the source images, which can be expressed as follows, where ∇ denotes the Sobel operator:

The overall loss function consists of degradation-type specificity decoupling loss and fusion loss:

where α, β, γ and δ are respectively set to 1e-5, 4, 1 and 1 for the balance between loss terms.

4 Experiment

4.1 Experimental Details

The proposed SpeciFuse needs high-quality infrared-visible image pairs as benchmarks due to its capability of learning degradation-type specificity between different individual degradations in same scene. For this purpose, we selected 718 precisely aligned infrared-visible image pairs from the EMS [Yi et al., 2024] dataset to construct the training set. And we randomly extracted 128×128 co-located patches from each source image pair as network inputs. The entire network was implemented in PyTorch and trained on an NVIDIA 3090 GPU with empirically determined hyperparameters: the infrared and visible encoders were optimized with a learning rate of 1e-5, while the gated fusion module and fusion decoder employed a learning rate of 1e-4. All components were jointly trained for 60 epochs using the AdamW [Loshchilov and Hutter, 2017] optimizer with a batch size of 4.

4.2 Model Comparisons

To demonstrate the effectiveness of the proposed method, we conducted comprehensive comparisons with nine stateof-the-art (SOTA) fusion approaches. These include four fusion methods for the normal scenario (YDTR [Tang et al., 2022b], CDDFuse [Zhao et al., 2023], LRRNet [Li et al., 2023] and PIAFusion [Tang et al., 2022a]), and five degradation-resistant fusion algorithms (TextIF [Yi et al., 2024], TSDM [Xue et al., 2024], DRMF [Tang et al., 2024], DAF [Wang et al., 2025], and OmniFuse [Zhang et al., 2025]). For evaluating practical fusion performance, we employ benchmark datasets including M3FD [Liu et al., 2022], LLVIP [Jia et al., 2021], and MSRS [Tang et al., 2022a]. Based EMS dataset, we synthesize low-quality images for assessing fusion capabilities under composite degradations.

The quantitative metrics employed in this study include Peak Signal-to-Noise Ratio (PSNR), Cross-Correlation coefficient (CC), Spatial Frequency (SF) [Eskicioglu and Fisher, 2002], Mean Square Error (MSE), Spectral Correlation Distortion (SCD) [Aslantas and Bendes, 2015], and Quality Assessment of image Fusion based on edge information (Qabf) [Xydeas and Petrovic, 2000].

Image Fusion in Practical Scenario

The qualitative results under practical scenario is shown in Figure 6. Specifically, YDTR, LRRNet, and TSDM produce insufficient brightness in target regions and background contrast. CDDFuse, PIAFusion, and TextIF maintain basic thermal intensity and details but lack target prominence and color fidelity, with suboptimal brightness. OmniFuse yields overly smooth results, losing texture sharpness. Although DRMF and DAF achieve higher brightness, both lose intrinsic contrast and internal detail within thermal targets. Additionally, DRMF weakens thermal targets, while DAF introduces border expansion artifacts. In contrast, SpeciFuse achieves the most balanced and informative fusion.

Quantitative results are presented in Table 1. SpeciFuse achieves strong overall performance across PSNR, CC, and SF, reflecting its superiority in preserving structural detail, maintaining source consistency, and enhancing visual clarity

Image Fusion in Composite Degradation Scenario

To further assess fusion performance under complex degradations, we constructed a composite degradation test set based on the EMS dataset, with separate sets for visible and infrared modalities. For visible images, the set includes three composite degradations: low-light + noise (Degradation I:), lowlight + haze (II), and low-light + random noise + haze (III). For infrared images, it includes three composite degradations: low-contrast + random noise + stripe noise (IV), blur + lowcontrast (V), and low-contrast + random noise + stripe noise + blur (VI). For TextIF, we used "low-light" and "low-contrast" text prompts corresponding to the visible and infrared degradations, respectively.

Qualitative results in composite degradations are shown in Figures 7 and 8. TextIF reduces degradation impact via text guidance but is limited to predefined categories and declines on unseen degradations. TSDM denoises well but underperforms in brightness and contrast restoration. DRMF generates generally good quality but fails under some conditions, and it over-relies on visible information, covering the infrared information. DAF handles unknown degradations but amplifies noise and causes over-enhancement, and it loses internal contrast of objects. OmniFuse suffers from overall blurring and weak thermal objects. In contrast, SpeciFuse preserves both object saliency and scene details more comprehensively.

Quantitative results under six degradation scenarios are summarized in Table 2, where our method achieves strong overall performance. The superior PSNR and lower MSE reflect high pixel-level fidelity and effective detail recovery across degradations. Outstanding SCD and Qabf scores indicate a superior balance between thermal contrast retention and visible texture preservation, along with strong edge and directional feature maintenance. These results align with qualitative observations and demonstrate that SpeciFuse robustly recovers both structural and textural information under composite degradations.

Futhermore, SpeciFuse achieves a highly competitive balance between efficiency and performance, with 1.170M trainable parameters and an average runtime of 0.093 seconds per image (640 × 480 px) on an NVIDIA RTX3090 GPU. In comparison, the corresponding performance for other degradation-resistant methods are: TextIF (215.110M, 0.387s), TSDM (0.635M, 0.965s), DRMF (170.90M, 9.782s), DAF (0.278M, 0.085s), and OmniFuse (173.34M, 5.372s).

4.3 Ablation Study

In order to validate DSDM, DGFM and loss functions, we perform five set of ablation experiments on them respectively. Specifically, "w/o DSDM" represents the variant that remove degradation-type specificity decoupling module and decoupling loss in training stage. In "w/o DGFM", the visible and infrared encoded features were simply concatenated before being fed into the decoder. "w/o LSSIM", "w/o Lmax" and "w/o Lgrad" respectively denote the variant that remove the corresponding loss function in training stage. Figure 9 shows ablation results under a practical scene. Removing the DSDM and DGFM leads to failure in restoring color and details from darkness, reduced target contrast, and blurred details. Omitting the intensity loss causes severe color distortion. Removing the SSIM loss slightly degrades brightness and details, while dropping the gradient loss blurs textures. These findings confirm that both modules are essential for adaptive restoration and preservation, and each loss is critical for visual fidelity. Furthermore, we conducted hyperparameter sensitivity analysis on λ, α, β, γ and δ. Quantitative results in Table 3 and 4 show the full model significantly outperforms all ablated variants, verifying that every component is indispensable for optimal performance.

5 Conclusion

This study proposes SpeciFuse, a robust infrared-visible image fusion network that learns the specificity of individual degradations via a siamese architecture, thus modeling and addressing the conflicts arising from composite degradations in the fusion process. In SpeciFuse, the degradationtype specificity decoupling module is designed between the encoders of two pipelines to decorrelate the feature representations of different degradation types, thereby reducing their mutual interference and preserving structural information in the feature encoding. Furthermore, a degradationaware gated fusion module is proposed to adaptively learn degradation-conditional fusion strategies based on the decoupled features. This module dynamically modulates fusion process via channel-wise and spatial-wise gating mechanisms, significantly enhancing the fusion robustness and quality across diverse conditions. Remarkably, although trained on single-degradation conditions, comprehensive experiments demonstrate the superiority of SpeciFuse in handling diverse composite degradations.

Acknowledgments

This work is supported by National Natural Science Foundation of China under Grant No.62401411. This work is also supported by Natural Science Foundation of Hubei Province under Grant No.2026AFB616.