Progressive Modality-Adaptive Interactive Network for Multi-Modality Image Fusion

1 Introduction

The increasing demand for image processing technologies has driven the widespread application of image fusion, particularly in complex environments [Sun et al., 2022; Zhang et al., 2021] where single-modality images often fail to provide sufficient information. To address this limitation, MMIF [Xu et al., 2020; Zhao et al., 2020] combines images from different modalities, leveraging their strengths to enhance information representation and capture fine-grained details. Within MMIF, infrared and visible image fusion (IVF) [Ma et al., 2023] plays a critical role by combining the complementary information from both modalities. It leverages the infrared's advantage in harsh conditions while benefiting from the finer details provided by visible images. This fusion technique generates more comprehensive and insightful images, with significant applications [Zhang et al., 2020] in object detection and surveillance. Furthermore, MMIF also encompasses medical image fusion (MIF) [James and Dasarathy, 2014], where modalities such as CT and MRI are integrated to offer more precise diagnostic insights, thereby facilitating advanced medical decision-making.

In recent years, numerous methods [Deng and Dragotti, 2020; Zhang and Ma, 2021] have been proposed in the field of MMIF. Although these methods have achieved some success, some still fail to address the distinct information distribution characteristics of different modalities. For instance, infrared images often contain sparse and localized information, while visible images provide richer, more detailed content. Furthermore, the correlation between modalities is not static but varies across different scenes, a dynamic relationship that existing methods have yet to adequately explore. This dynamic relationship, along with the modality-specific characteristics, plays a crucial role in multimodal image fusion, as evidenced by quantitative analysis.

To facilitate a deeper comparison between infrared (I) and visible (V) image modalities, we utilize paired images from the MSRS [Tang et al., 2022] dataset. In this study, we compute several evaluation metrics to quantitatively assess the relationship between these two modalities. The visual results, as illustrated in Figure 1, consist of bar charts that depict the differences in information sparsity across various metrics, along with a scatter plot of JS divergence obtained via K-Means clustering, which captures the dynamic correlation across different scenes. As for the bar chart, it illustrates the distribution of P values across different metrics, with each bar representing a specific metric. Given an input pair Ij and Vj (representing the j-th image pair), we compute their feature values F(Ij) and F(Vj) based on various metrics, including gradient change rate (GCR) , Laplacian variance (Lap) , entropy (H) and L1 norm (L1) . The difference F(Ij) - F(Vj) is then calculated, and the statistic P is defined as the proportion of instances where this difference is positive or negative. This provides a quantitative measure of the distribution of differences for each metric. The bar chart demonstrates that infrared images commonly exhibit higher sparsity compared to visible images, clearly highlighting the differences between the two modalities. Meanwhile, along with the scatter plot derived from the JS divergence analysis of the Ij, Vj image pairs, K-Means clustering is applied to categorize the image pairs into three distinct groups. This clustering captures the dynamic correlation across different scenes, revealing that the correlation between the image pairs varies with the scene, highlighting how the image pairs complement each other under different conditions. These findings highlight the complex relationship between modality differences and correlations, offering key insights for designing the subsequent fusion approach.

Motivated by the critical insights into the intricate interplay between modality differences and correlations, we introduce a novel two-stage framework designed to comprehensively tackle these challenges. In the first stage, to accommodate the unique characteristics of infrared and visible images, we design two distinct modules: the Neighbor-Group Matching Model (NGMM) and the Context-Aware Modeling Network (CAMN), enabling modality-adaptive feature extraction. Unlike traditional methods that apply uniform processing across modalities, our approach tailors the processing to the intrinsic differences of each modality, enabling finer-grained feature extraction and effectively addressing the challenges posed by modality disparities. Building on this, the second stage introduces the Modality-Interactive Compensation Module (MICM), trained with frozen first-stage modules to capture deep inter-modality relationships through fine-grained interactions. The module employs a gated-weight fusion mechanism to dynamically adapt to the varying correlations between image pairs, effectively addressing the evolving interactions and ensuring the precise integration of modalityspecific information.

To summarize, our contributions are as follows:

• We propose an innovative asymmetric two-stage framework to effectively capture the variations and correlations between different modalities, enhancing the robustness and adaptability of our fusion model.
• We design two distinct, modality-adaptive feature extraction modules that dynamically adapt to the varying sparsity levels and information distributions specific to infrared and visible images, ensuring a more precise and effective fusion process.
• We propose a modality-interactive compensation mechanism that adeptly captures inter-modality correlations through fine-grained interactions, enhancing fusion performance by leveraging a gated-weight fusion strategy.

2 RelatedWorks

Recent years have witnessed significant progress in deep learning-based multimodal image fusion methods. Convolutional networks, known for their ability to capture spatial patterns, have been widely applied to image fusion tasks [Li et al., 2018; Liu et al., 2018]. However, convolutionbased methods often struggle to capture long-range dependencies due to their inherently local receptive fields. As vision attention mechanisms [Dosovitskiy, 2020; Zamir et al., 2022] have gained prominence, Transformer-based methods have emerged as powerful tools for capturing global dependencies, greatly enhancing performance in complex fusion tasks [Zhao and Nie, 2021; Tang et al., 2024]. Generative Adversarial Networks (GANs) [Creswell et al., 2018; Goodfellow et al., 2020] use a game-theoretic interaction between a generator and a discriminator to generate highquality fused images. Unlike convolutional and Transformerbased methods, which focus on feature extraction, GANs use adversarial training to enhance fusion, effectively preserving thermal target information and retaining key details [Li et al., 2019; Chakraborty et al., 2024]. Autoencoders (AE) [Michelucci, 2022; Berahmand et al., 2024] have also been widely applied to multi-modal image fusion tasks [Zhao et al., 2020; Li et al., 2021]. AE-based methods employ an encoder-decoder architecture to map input images into lowdimensional representations, which are then fused in a highdimensional space to extract key features and produce highquality fused images. With the growing demand for practical applications, image fusion methods that integrate downstream tasks have seen rapid development [Liu et al., 2023a; Liu et al., 2023b]. By exploiting the synergy between upstream and downstream tasks, these methods effectively balance fusion quality and performance for the specific target task. While some existing methods may not fully account for the distinct information distribution characteristics of different modalities, our approach introduces a tailored processing strategy to enhance modality-specific feature extraction. By incorporating dynamic weighting and gating mechanisms, our method captures the dynamic correlations between modalities, enabling effective interaction and compensation across them. Together, these strategies effectively handle the coexistence of modality differences and correlations, ultimately enhancing the quality of the fused images.

3 Method

3.1 Overall Architecture

In this paper, we propose a progressive modality-adaptive interactive network (PoMAI) for MMIF, designed to address the challenges of modality-specific sparsity variations and dynamic correlations. As shown in Figure 2, PoMAI comprises five key modules and two training stages. The first stage focuses on learning modality-specific features, while the second stage builds upon the first by fine-tuning the model, with selected modules frozen, to better capture dynamic correlations. To further clarify the operation of PoMAI, we use the infrared and visible image fusion (IVF) task as a case study.

3.2 Shared Encoder

We employ a shared encoder E(·), consisting of four cascaded Transformer blocks [Zamir et al., 2022], to extract features from the input images II ∈ RH×W×1 , IV ∈ RH×W×3 , yielding consistent feature representations:

The extracted features xIenc , xVenc ∈ RH×W×C maintain consistent dimensionality across modalities, facilitating efficient multimodal fusion.

3.3 Neighbor-GroupMatchingModel

Given the feature xIenc , NGMM integrates the Hash-based Feature Neighboring (HFN) and Group Attention Mechanism (GAM) to better align with the sparsity characteristics of infrared images, as shown in Figure 2.

Specifically, xIenc is processed by a dual-branch structure, where separate convolutions with identical parameters are applied in each branch. The detailed formulation is as follows:

Following the dual-branch processing, xIleft , xIright ∈ RH×W×C are flattened into fleft, fright ∈ RL×C , where L = H×W. Leveraging the locality-sensitive hashing mechanism introduced in [Andoni et al., 2015], The HFN module begins by projecting the input features xIright onto a spherical space. Subsequently, the indices S ∈ RH×W×1 , which identify the top K most relevant elements within the feature space, are generated through hash mapping. These indices S are used to perform hash-based grouping of xIleft , xIright , resulting in gleft, gright ∈ RL×C . Note that the indices S are computed in the right branch and shared with the left branch. The grouping process is formalized as follows:

Building on HFN's feature grouping, we introduce the GAM to capture the relationships among grouped features. This process ultimately produces M aggregated feature groups, where M is the total number of groups, calculated as M = LK . The attention scores for each group are reshaped and added to xIenc via a residual connection, yielding the NGMM's output and enhancing the features.

3.4 Context-Aware Modeling Network

Unlike infrared images that exhibit high sparsity, visible images contain a wealth of details and intricate structures. To address this, we propose the Context-Aware Modeling Network (CAMN), which integrates the Spatial-Context Block

(SCB) and Channel-Context Block (CCB) to effectively extract and encode the rich information in visible images.

As shown in Figure 2, the processing of xVenc begins with the SCB, designed to extract spatial contextual information. Within the SCB, the input feature passes through a 1×1 conv for initial processing, followed by a 7 × 7 dilated conv (DC) to capture spatial dependencies. A subsequent 1×1 conv refines the representation, with residual connections integrating the original features to produce xVscb . The SCB module emphasizes spatial representation, formulated as follows:

The output of the SCB, xVscb , is fed into the CCB for channel refinement. The CCB applies a normalization layer, followed by two 1×1 pointwise conv (PC) to adjust channel dimensions. A residual connection combines xVscb with the intermediate features, yielding xVccb . The CCB module emphasizes channel-wise interaction, formulated as follows:

Finally, a residual connection combines xVccb with xVenc , helping preserve the rich information and intricate details of visible images, and yielding the final output xVcamn .

3.5 Modality-Interactive CompensationModule

Building on previous observations, we find that certain image pairs exhibit significant modal correlations, providing a foundation for the interaction module. Inspired by [Dai et al., 2021], we propose the Modal Interaction Compensation Module (MICM), illustrated in Figure 3, which integrates the Feature Perceptual Module (FPM) and Gated Feature Fusio (GFF) to dynamically capture cross-modal correlations and compensate for modal discrepancies.

Given the features xIenc and xVenc , we fuse the two modalities through element-wise addition to obtain xsum. Then, xsum is fed into both the channel attention (CA) and spatial attention (SA) modules. Next, the attention scores from CA and SA are combined via element-wise addition. Finally, the fused attention features are concatenated (denoted as Cat(·, ·)) with xsum along the channel dimension to produce the output of the FPM, xfpm. This process is succinctly expressed by the following formula:

The result feature xfpm from the FPM is further processed by the GFF module. First, a 7 × 7 convolution is applied to xfpm, and the result is passed through a sigmoid activation σ(·) to generate the gating map W. This gating map modulates xIenc and xVenc , producing weighted features. These weighted features are then added to xIngmm and xVcamn , respectively. Finally, the resulting features are concatenated along the channel dimension to form xgff , which also serves as the output of the MICM. This process can be described as:

3.6 Decoder

The decoder D(·) is used in both stages, mirroring the encoder's structure to ensure consistent feature processing. In the first stage, it takes concatenated features from both modalities as input, while in the second stage, it processes the fused features from the MICM module, ensuring visual coherence and effective multimodal integration. Formally:

3.7 Progressive Training

As illustrated in Figure 2, our framework employs a progressive two-stage training strategy. Specifically, in the first stage, we jointly train the encoder E(·), NGMM, and CAMN to capture modality-specific feature from paired images {II, IV}. These features are then decoded by D(·) to generate a fused image that integrates the essential information from both modalities. In the second stage, we freeze the first-stage modules and introduce MICM to build the dynamic correlation between the two modalities. The resulting features, refined through enhanced cross-modal interactions, are decoded by D(·) to produce the final fused image, reflecting the dynamic correlation between the modalities. Since the objective of both stages is to synthesize a fused image that effectively integrates information from the two modalities, we employ the same loss function L for consistency and training efficiency. The loss function is formulated as follows:

The coefficients α1, α2, α3, α4 in the total loss function are weighting factors that balance the contributions of the different loss components to the overall objective. Each term in the overall loss function is computed as follows:

where SSIM(·, ·) represents the structural similarity index [Wang et al., 2004]. The operator ∇ denotes the Sobel gradient operator, used to capture edge information. The max(·, ·) function selects the pixel-wise maximum values between II and IV .

4 Infrared and Visible Image Fusion

4.1 Setup

Datasets. We conduct IVF experiments on four popular datasets: TNO [Toet and Hogervorst, 2012], RoadScene [Xu et al., 2020], M3FD [Liu et al., 2022], and MSRS [Tang et al., 2022]. Our proposed method, PoMAI, is trained on the MSRS training set (1083 pairs) and tested on TNO (20 pairs), RoadScene (70 pairs), M3FD (100 pairs), and the MSRS test set (361 pairs). To evaluate the generalization capability of the model, no fine-tuning was performed on TNO, RoadScene, or M3FD during testing.

Metrics. We use four metrics to evaluate the fusion results: standard deviation (SD), spatial frequency (SF), average gradient (AG) and visual information fidelity (VIF). A higher value of these metrics indicates better fusion quality, with details available in [Ma et al., 2019].

Implement details. Our experiments are implemented with the Pytorch framework and on a machine with four NVIDIA Tesla P100 GPUs. The training images are initially processed by randomly cropping them into 128×128 patches, ensuring a diverse set of training samples for the model. The training process is structured into two phases: the first phase comprises 80 epochs, followed by the second phase with 20 epochs, both utilizing a batch size of 16. The Adam optimizer is employed to update the parameters of each module, with an initial learning rate of 1 × 10-4 that decreases by a factor of 0.5 every 20 epochs. In Eq. 14, the values of α1, α2, α3, and α4 are set to 5, 1, 1, and 10.

4.2 Comparison with SOTAMethods

We conduct comprehensive qualitative and quantitative analyses on four popular datasets, comparing our approach with 12 state-of-the-art (SOTA) methods, including DeFusion [Liang et al., 2022], UMF [Wang et al., 2022], CDDFuse [Zhao et al., 2023], SegMiF [Liu et al., 2023a], BDLFusion [Liu et al., 2023b], IGNet [Li et al., 2023b], LRRNet [Li et al., 2023a], EMMA [Zhao et al., 2024], CAF [Liu et al., 2024a], MMDRFuse [Deng et al., 2024], ITFuse [Tang et al., 2024] and TIM [Liu et al., 2024b].

Qualitative Comparisons. Figure 4 presents visual comparisons on MSRS dataset. Compared with other methods, the proposed approach better preserves the rich details from the visible modality (highlighted by the green-colored box) while simultaneously retaining the thermal information from the infrared image (highlighted by the red-colored box) to the greatest extent, achieving superior fusion results.

Quantitative Comparisons. Beyond visual comparisons, we evaluate our method quantitatively on four datasets using four metrics, as shown in Table 1. Our method achieves significant performance improvements, with superior SD values ensuring optimal contrast and competitive VIF scores indicating effective correlation preservation.

4.3 Ablation Study

We conduct a series of ablation studies on the MSRS dataset, using the same evaluation metrics as in the IVF experiments, to evaluate the individual contributions of the three proposed modules and the effectiveness of the progressive training strategy. The results are summarized in Table 2.

NGMM. To evaluate the effectiveness of the NGMMmodule, we conduct an ablation study by removing it from the PoMAI framework, while keeping all other components intact. This setup, denoted as Exp.I, demonstrates that the removal of the NGMMmodule leads to a significant performance decline across most evaluation metrics, failing to match the performance of the full PoMAI model. This highlights the crucial role of the NGMM module in enhancing system performance.

CAMN. To assess the contribution of CAMN, we perform an ablation study by removing CAMN from the PoMAI framework while retaining all other components. This experimental setup is denoted as Exp.II. The result consistently demonstrates that the removal of CAMN resulted in a notable decline in performance across the majority of evaluation metrics, highlighting its critical role in augmenting the model's overall effectiveness.

MICM and Training Strategy. Finally, to evaluate the effectiveness of incorporating the MICM module and adopting the two-stage training strategy, in which the weights of the first-stage modules (including the Shared Encoder E(·), NGMM, CAMN and Decoder D(·)) are frozen and only the MICM module is trained, we design three ablation experiments. Specifically, Exp.III employs only the first-stage network without introducing the MICM module, equivalent to training without the two-stage strategy. Exp.IV integrates the MICM module's training process directly into the first-stage network, effectively eliminating the two-stage architecture. Finally, Exp.V introduces the MICM module while allowing the first-stage network's weights to be updated during training, thereby exploring the impact of weight optimization. The results of the three ablation experiments clearly demonstrate the necessity of incorporating the MICM module, as well as the benefits of adopting a two-stage training strategy. Specifically, by freezing the weights of the first-stage modules and progressively training the MICMmodule in the second stage, we observe a significant improvement in the model performance. This approach underscores the importance of both the MICM module and the progressive training strategy in achieving superior results.

5 Medical Image Fusion

Setup. We perform MIF experiments using 50 image pairs from the Harvard Medical dataset [Johnson and Becker, 2005]. Notably, we directly apply the model trained on IVF tasks to the Harvard dataset without fine-tuning. The quantitative metrics were consistent with those used in IVF task.

Comparison with SOTA Methods. We conduct comprehensive qualitative and quantitative analyses with 12 state-ofthe-art (SOTA) competitors, all of which are trained exclusively on IVF-related datasets without fine-tuning on MIF. These methods are consistent with those mentioned in the IVF section. The qualitative results, as illustrated in Figure 5, demonstrate that PoMAI effectively preserves fine-grained texture details, avoids color distortion and accentuates structural information. Quantitatively, as shown in Table 3, PoMAI outperforms most existing methods across the majority of metrics, highlighting its potential for MIF tasks.

6 Downstream IVF Applications

6.1 Evaluation in Object Detection

Setup. The multi-modal object detection (MMOD) is performed on the M3FD dataset, which consists of 4,200 pairs of infrared and visible images, categorized into six labels: people, car, bus, motorcycle, truck, and lamp. The dataset is split into training, validation, and test sets with an 8:1:1 ratio. YOLOv5 [Jocher, 2020] is employed to evaluate detection performance using the mAP@0.5 metric. The training configuration consists of 100 epochs, a batch size of 8, the SGD optimizer and an initial learning rate of 1e-2.

Comparison with SOTA Methods. Table 4 indicates that PoMAI achieves competitive performance across multiple evaluation metrics, with notably higher performance in the mAP@0.5 score compared to existing methods. These results suggest that our approach demonstrates certain effectiveness in improving object detection tasks.

6.2 Evaluation in Semantic Segmentation

Setup. The multi-modal semantic segmentation (MMSS) is performed on the MSRS dataset with pixel-level semantic information of nine object categories(e.g., background, car, person, bike, curve, car stop, guardrail, color cone and bump). DeepLabV3+ [Chen et al., 2018] is utilized to assess semantic segmentation performance based on the intersection-overunion (IoU) metric. The training process is configured with 300 epochs, where the backbone network is frozen for the first 100 epochs. Additional settings include a batch size of 8, the SGD optimizer and an initial learning rate of 7e-3.

Comparison with SOTA Methods. Table 4 summarizes the quantitative semantic segmentation results of various fusion methods on the MSRS dataset. Our method achieves competitive performance across multiple metrics, particularly attaining the highest mIoU score. These results indicate that our approach effectively improves the performance of semantic segmentation tasks.

7 Conclusion

In this paper, we propose a progressive modality-adaptive interactive network (PoMAI) for multi-modality image fusion. We analyze the sparsity differences and dynamic correlations between infrared and visible modalities in the MSRS dataset using diverse metrics. Our analysis reveals that infrared features exhibit sparsity and localization characteristics, while visible features contain richer and more detailed information, with their cross-modal correlations dynamically evolving across different scenes. Based on these findings, we design modality-adaptive feature extraction modules and an modality-interactive compensation module to effectively address these challenges. Our experiments demonstrate PoMAI's superior performance in MMIF, with significant gains in downstream tasks. In future work, we will explore advanced feature integration methods beyond simple channel concatenation to enhance inter-modal fusion.

Acknowledgments

This work was supported in part by the National Key Research and Development Plan of China under Grant 2021YFB3600503, in part by the National Natural Science Foundation of China under Grant 61972097 and U21A20472, in part by the Major Scientific Research Project for Technology Promotes Police under Grant 2024YZ040001, in part by the Natural Science Foundation of Fujian Province under Grant 2021J01612 and 2020J01494, in part by the Major Science and Technology Project of Fujian Province under Grant 2021HZ022007, in part by the Fujian Collaborative Innovation Center for Big Data Application in Governments, and in part by the Fujian Engineering Research Center of Big Data Analysis and Processing.

Contribution Statement

This work was a collaborative effort by all contributing authors. Chaowei Huang and Yaru su made equal contributions to this study and are designated as co-first authors. Xiao Ke, as the corresponding author, is responsible for all communications related to this manuscript.