Where Elegance Meets Precision: Towards a Compact, Automatic, and Flexible Framework for Multi-modality Image Fusion and Applications

1 Introduction

With the increasing complexity of application scenarios in intelligent systems, the drawbacks of single-type sensors have become apparent. They are incapable of providing a precise and comprehensive description of scenes or targets, leading to difficulties in accomplishing tasks in real-world challenging environments [Ma et al., 2019a]. To address this, image fusion technology has emerged, which aims to conduct comprehensive analysis and processing of information from different sources, achieving a consistent description of the observed scenes or targets. Among these, infrared and visible sensors play a crucial role in the perception process of intelligent systems. However, both infrared and visible images have their own limitations in diverse scenes [Zhang and Demiris, 2023]. Therefore, fusing the infrared and visible images can intelligent systems maintain stable perception in dynamically changing environments.

Drawing on strong non-linear fitting ability of Convolutional Neural Networks (CNNs) [Liu et al., 2023c; Liu et al., 2023b], extensive learning-based Infrared and Visible Image Fusion (IVIF) approaches have been proposed and applied well [Ma et al., 2019b; Liu et al., 2022a; Xu et al., 2020a]. A naive way to solve IVIF and its downstream tasks is to cascade dual networks with weighted mean functions, with achieving certain performance improvement [Liu et al., 2022a; Sun et al., 2022; Cao et al., 2023]. However, such a direct alternative optimization process make the dual networks incline to focus on only one task, thus bringing imbalanced performance across the dual tasks. Besides, the joint loss function still relies on empirical designing, and the tuning hypter-parameter is laborious, placing an obstacle to achieve high perception accuracy while generating favorable fused images for human inspection.

In the realm of IVIF, there is no ground truth for generating expected fused images. Consequently, the design of the loss function plays a pivotal role in determining the final outcome. That is to say, devising a formulation that seamlessly integrates the two tasks is a fundamental methodology for achieving superior fusion and perception results. Following this observation, this study addresses these critical issues by consolidating the fusion and perception tasks into a single objective, culminating in a Compact, Automatic and Flexible Framework, dubbed CAF. Specifically, we formulate the fusion and perception tasks from a coupling constraint optimization prospective, wherein each component is tailored to its respective loss function. Then, we optimize the dual tasks by introducing a new loss searching scheme. By allowing the perception task to guide the entire search process, the fusion loss function can be automatically derived, and the key features from both tasks are mutually reinforced and optimized.

Furthermore, the proposed CAF can be seamlessly integrated into existing fusion network architectures, outperforming them in terms of results, thereby showcasing the flexibility and generalization capability of our CAF.

The main contribution can be distilled as three main aspects as follows:

• Compact Formulation. From a new aspect, we embrace the image fusion and perception in a coupling constraint optimization. By constructing corresponding loss functions at different levels, the dual tasks mutually constrain each other, thereby achieving a tightly integration.
• Automatic Optimization. Considering the need to reduce the burden of laborious parameter tuning and optimize the dual tasks, we employ the guidance of higherlevel tasks to automatically optimize and search for suitable fusion loss functions. To the best of our knowledge, this is the first time that loss function searching has been introduced to the IVIF community.
• Flexible Framework. It is worth mentioning that CAF is a "ready-to-use" framework. We have extended it to several existing state-of-the-art fusion network architectures, achieving remarkable progress in both fusion and perception performance compared to their original versions. This demonstrates the flexibility and generalizability of our proposed CAF.

Our method achieves superior fused results on par with existing state-of-the-art methods and yields significant improvements for perception tasks (1.7 higher mAP@.5 in detection and 2.0 higher mIoU in segmentation).

2 RelatedWorks

Deep learning has achieved significant results in the field of IVIF [Liu et al., 2023a; Zhao et al., 2023a; Zhao et al., 2023b; Liu et al., 2023d; Liu et al., 2022b] due to its powerful feature extraction capability based on neural networks. As a task without ground truth, numerous loss functions have been proposed and widely used. In early methods focused on visual effects, Li et al. [Li and Wu, 2018] made a preliminary attempt at deep learning, with a loss function composed of two parts to constrain at both the pixel and structural levels. Ma et al. [Ma et al., 2019b] introduced adversarial constraint loss between the fused image and the visible image; furthermore, they [Ma et al., 2020] adjusted the strategy of adversarial loss to estimate the distribution of the visible and infrared domains simultaneously, achieving significant contrast and rich texture details. However, these methods all involve laborious manual design of losses and adjustment of weights. Xu et al. [Xu et al., 2020b] introduced an image quality detection mechanism to dynamically control the weights between different parts of the loss, creating a versatile fusion framework. Besides, Xu et al. [Xu et al., 2020a] introduced information fidelity to dynamically balance the impact of the two source images on the loss function, which somewhat reduced the burden of parameter tuning. However, these attempts could not eliminate the need for designing the loss function itself.

Nowadays, adjusting the loss functions between different tasks has become an main means of adapting fusion methods [Wu et al., 2024] to perception tasks. For object detection, Liu et al. [Liu et al., 2022a] proposed a dual-level adversarial learning network, while Sun et al. [Sun et al., 2022] proposed a detection-driven network; both of which cascade networks for different tasks to merge their respective loss functions. For semantic segmentation, Tang et al. [Tang et al., 2022] also followed a similar approach, using semantic loss to guide the fusion network to acquire more high-level semantic information. However, this kind of weighting of task losses is difficult to achieve a perfect balance to 'Best of Both Worlds'.

3 The ProposedMethod

In this section, we first given a compact formulation to model the relationship of the dual tasks. Then we employ the highlevel tasks to automatically optimize model in a loss-free manner. In the end, the hierarchical training is developed to achieve "Best of Both Worlds".

3.1 ProblemFormulation

Given a pairs of aligned infrared and visible images, our goal is to generate a visual-appealing fused image, along with high perception accuracy. The infrared, visible and fused images are all gray-scale with the size of m× n, denoted as column vectors Iir, Ivis, and If ∈ Rmn×1 , respectively. We formulate the dual tasks into one goal, in which the respected parameters are mutually interacted. The coupling constraint formulation can be written as:

where Lt is the task-specific training loss, and T denotes a perception task network with learnable parameters ωt. Lf is the fusion loss, and F represents fusion network with learnable parameters ωf. The fusion process can be symbolized as If = F(Ivis, Iir;ωf). For the perception task, the objective is to generate an output based on If, incorporating its own loss function to guide the training process.

Compared to previous methods focused primarily on visual effects, our proposed model delineates a distinct relationship, capturing the inherent correspondence between the two tasks. The performance of If hinges on defining the training loss, shaped by meticulously crafted hyper-parameters. However, selecting the appropriate loss type and adjusting weight hyper-parameters pose considerable challenges. To address this,e present a pioneering strategy to navigate these intricacies associated with loss selection and labor-intensive tuning.

3.2 Automatic Loss Search

As aforementioned, Lf plays a significant role to investigate the complementary modal characteristics for diverse perception tasks. To overcome the core obstacle, we present the automatic design scheme to identify task-specific weights that guide the fusion process. Denoting these weighted parameters as β, the aggregated losses based on Lf can be expressed as Lf = ∑ni=1 βiLi. Specially, we utilize the bi-level formulation to elucidate the automatic search process, which can be written as following:

where "train" and "val" denote the losses on the training and validation datasets, respectively. The upper-level objective seeks to minimize the weighted parameters, gauging them against the measurements of perception tasks derived from the fused images. Conversely, the lower-level optimization aims to ascertain the optimal fusion parameters in the context of the provided β. We contend that this formulation aptly achieves task-specific image fusion. On the one hand, the image fusion constraint aids the perception task in precisely assessing the impact of β. On the other hand, feedback from downstream perception tasks can encourage adjustments in β to further refine the fusion process.

However, solving the bi-level optimization in Eq. (2) has been demonstrated as a challenging task, which acquires the massive computation resource to address the nested optimization [Liu et al., 2021c; Liu et al., 2021d]. In order to effectively solve Eq. (2), inspired by the remarkably success of differentiable architecture search [Liu et al., 2019], we adopt the one-step truncated approximation strategy. To simplify the representation, we utilize the Lt val (β; ω∗ f ) and Lf train (β;ωf) to concisely describe the upper and lower objectives in Eq. (2). In detail, the gradient of upperlevel objective can be approximated as ∇βLt val (β; ω∗ f ) ≈ ∇βLt val (β; ωf - η∇ωf Lf train (β;ωf)), where η denotes the learning rate of fusion. In this way, ω∗ f can be approximated by the one-step gradient of image fusion. Moreover, in order to address the hessian matrix, we further utilize the finite difference approximation to solve this second-order gradient, which can describe the intrinsic latent relationship between β and ωf. Denoted ωf - η∇ωf Lf train (β;ωf) as ω′f , the finite difference approximation can be written as

(3)

where δ is a constant and we define ω± f = ωf ± δ∇ω′f Lt val (β;ω′f ). Thus, we obtain the efficiency approximated gradient update of upper-level objective to perform the automatic search for Lf . We also elaborately illustrate the search procedure in Algorithm 1.

Algorithm 1 Automatic Loss Function Search

Require: The loss functions Lt , Lf , and other necessary hyper-parameters.

Ensure: The optimal parameters β∗ .

• 1: while not converged do
• 2: %Optimizing the perception task weights ωt;
• 3: ωt ←ωt -∇Lt train (If; ωt );
• 4: %Optimizing the image fusion network;
• 5: ωf ←ωf -∇Lf train (Ivis, Iir; ωf , β);
• 6: % Optimizing the hyper-parameter β using first-order approximation;
• 7: β ← β - ∇βLt val (Ivis, Iir; β,ωf - η∇ωf Lf train (β; ωf)));
• 8: end while
• 9: return β∗ .

3.3 Loss Search Space

We introduce 4 distinct loss functions: pixel-level loss (comprising L1 and LMSE), structural similarity loss LSSIM, gradient loss Lgrad, and perceptual loss Lper. Each function is tailored to measure specific image characteristics. Detailed formulations of loss functions are presented in Figure 1.

• Pixel-wise similarity loss, including L1 and LMSE, are introduced, which aim to ensure that the pixel intensity of the fused image is consistent with reference images with p = {1, 2}. L1 has been proven to be effective in previous fusion work[Huang et al., 2022; Liu et al., 2022a; Sun et al., 2022; Ma et al., 2021], while LMSE has also widely applied[Xu et al., 2020a; Liu et al., 2021a]. Specifically, we consider there candidate references, including source images (Ivis and Iir), pixel intensity maximum (max(Ivis, Iir)), and visual saliency guidance (M(Ivis, Iir)), calculated by [Liu et al., 2022a]. M denotes the aggregation operation based on saliency mask, which can be written as Mask ⊗ Iir + (1 - Mask) ⊗ Ivis.
• Structural similarity (SSIM) [Wang et al., 2004] has been widely used in the field of image fusion[Zhao et al., 2020; Xu et al., 2020a; Liu et al., 2022a; Liu et al., 2021b]. It considers the similarity in brightness, contrast, and structure, which can better reflect human perception of image quality.
• Gradient loss is added into the search space, which focuses on the details and local edge information of the image, widely used in [Zhang and Ma, 2021; Wang et al., 2022; Sun et al., 2022; Ma et al., 2021].
• Perceptual loss, compared with others, focuses more on the perceptual quality and visual effects of images[Ledig et al., 2017], which is widely used in the fields of image fusion, generation, and restoration[Wang et al., 2022;

Han et al., 2022]. In the formulation, φl denotes the l-th layer in VGG-16 and L = {1, 3, 5, 9, 13}.

3.4 Hierarchical Training

After obtaining the optimal combination β for the LF , we can further solve the major objective Eq. (1) with the hierarchical training. Due to the huge gradient computation of parameters for this nested formulation, instead of utilizing the exact solutions and above approximation directly, we introduce the stage-wise gradual learning to hierarchically address this objective. In details, we first optimize the constraint to obtain the optimal parameters of fusion (i.e., ω∗ f ). Then we conduct the optimization for the upper-level objective to obtain the desired parameters of perception. Owing to the link and guidance that the proposed CAF established between the task and the fusion network during the searching, we can employ the training for fusion and perception separately. This strategy not only circumvents the potential negative influence of task loss on the fusion network parameters but also obviates the rigid demand for labeled data intrinsic to cascaded networks. Importantly, this approach realizes "Best of Both Worlds", fulfilling the inherent requirements of the image fusion and perception tasks.

Discussion. We argue that our framework has two significant characteristics, including model-irrelevant generalization and task-related flexibility. Firstly, our framework is generic enough to replace the arbitrary fusion networks and discover suitable loss combination for improve perception and visual quality. Secondly, due to the flexible formulation (Eq. (1)) and effective task-guided loss search (Eq. (2)), our framework is effective to address various multi-modality semantic perception tasks, without introducing redundant auxiliary learning mechanisms (i.e., feature interaction). The related experimental details are reported below.

4 Experiments

4.1 Implementation Details

Search configurations. We employed labeled datasets M3FD [Liu et al., 2022a] and MFNet [Ha et al., 2017] for searching, with the former being randomly partitioned and the latter using the standard division. The fusion network F is simply composed of three dense blocks, without any specific design. We selected state-of-the-art perceptual task methods (YOLOv5 1 for detection, and Segformer [Xie et al., 2021] for segmentation) as the task network T to integrate into the framework, adopting the provided s/mb1 models and maintaining their original loss functions. During the search for the fusion hyper-parameters matrix β, we used the Adam optimizer with an initial learning rate of 5e-2 for 10 epochs of iterations, and adopted a step learning rate decay strategy at a rate of 0.998.

Training configurations. With the searched loss function Lf, we optimize ωf on 1k image blocks collected from M3FD, TNO [Toet, 2017], and RoadScene [Xu et al., 2020a], using the Adam optimizer with a learning rate of 1e-4 for 100

1 https://github.com/ultralytics/yolov5

epochs. After training the fusion network F, the generated fusion images are fed into the task network for fine-tuning to test task performance. Except for the training rounds (100 epochs for detection, 4k iterations for segmentation), all other settings remain unchanged as per the original source code. All search and training experiments are performed on two NVIDIA Tesla V100 GPUs within PyTorch framework.

In the following, we compared with 7 SOTA competitors, including DIDFuse [Zhao et al., 2020], SDNet [Zhang and Ma, 2021], UMFusion [Wang et al., 2022], DeFusion [Liang et al., 2022], ReCoNet [Huang et al., 2022], U2Fusion [Xu et al., 2020a], and TarDAL [Liu et al., 2022a] 2 .

4.2 Evaluation in Object Detection

Qualitative detection comparisons. We visualize some results of object detection in Figure 2. The first set of images are daytime road scenes, in which pedestrians are basically invisible in the visible images, while the infrared images have significant target information. The methods in the first row of red area fail to fully retain useful information from both modalities, and their low-contrast fusion results do not support high-precision detection. Benefiting from the detectionfavorable loss obtained through searching, our method significantly outperforms the SOTA method TarDAL, in the detection performance of this scene. The second set of images features small targets in overcast, bright night scenes. Similarly, our method achieves the highest detection accuracy. However, TarDAL misses the car in the lower right corner due to excessive contrast.

Quantitative detection comparisons. Table 1 presents the object detection results of various fusion methods on the M3FD and Multispectral [Takumi et al., 2017] datasets, which are measured by average precision (AP) and mean average precision (mAP). The former includes a variety of challenging road scenes, while the latter mostly comprises lowresolution campus scenes. The two source image modalities show particularly notable results in the categories of people and motorcycle, where other methods focusing on visual effects cannot maintain consistent superior detection performance. The SOTA method TarDAL, achieves second best in multiple categories, but its overall performance still has a considerable gap compared to our method on both datasets. That also verifies that our search framework can effectively provide a task-appropriate loss to boost performance.

4.3 Evaluation in Semantic Segmentation

Qualitative segmentation comparisons. Figure 3 displays the visual effects of segmentation, where the labels are embedded in the source images. In both daytime and nighttime scenarios, infrared and visible images exhibit complementary segmentation expressions characteristic of their own modalities. Notably, the proposed method effectively withstands the interference from infrared to segment thermally insensitive objects (e.g., the car in the night scene). On the other hand, other fusion-based methods cannot correctly predict thermally sensitive objects (e.g., the pedestrian in the daytime scene), demonstrating that our result is capable of making full use of infrared information.

2 The version guided by detection is selected for comparision

Quantitative segmentation comparisons. Table 2 presents the quantitative results of various fusion methods on the MFNet dataset, measured by Intersection-over-Union (IoU) and Accuracy (ACC). The proposed method achieves the highest numerical performance in multiple categories with the fusion loss obtained from search. Furthermore, fusion schemes with pleasant visual effects struggle to produce excellent segmentation performance, which also validates the effectiveness of the proposed framework that compactly integrates the two tasks.

Validation through a pretained basedline. Besides the fine-tuned model, Deeplab-V3+ [Chen et al., 2018] trained on the Cityscapes dataset [Cordts et al., 2016] is also leveraged to measure various approaches. Source and fused images fromMFNet are directly input into Deeplabv3+, respectively, and their visualized results are reported in Figure 4. In the complex lower region, our method nearly flawlessly segments the terrain, people, and the black fence. The results demonstrate that the loss function searched from the proposed framework are task-informative and accommodate various scenarios of the perception task.

4.4 Analysis of Image Fusion

Influence of different tasks. We first analyze the impact of different tasks on the search results when they are integrated into the framework. Figure 5 shows the final search results, with the visualization on the left and main parts of the final searched loss function on the right. It is quite apparent that different tasks generate different interests in the components within the search space: compared to the segmentation task, the detection loss is more abstract, so it pays more attention to the loss at semantic feature level. In contrast, the segmentation task corresponds to the pixel level and prefers to capture the most interesting parts of the image through constraints on the maximum pixel of the source image. Reflected in the visualization effect, the result guided by segmentation tends to be closer to the visible image (as shown in the green boxes), with lower contrast and limited preservation of scene details; whereas, the result guided by detection incorporates more infrared features, and the content within the salient pedestrians have richer contours (as shown in the red box).

Qualitative comparisons. Figure 6 presents visual comparison of various methods. In the smoggy scene, our approach perfectly retains the rich details from the visible image (leaves shown in the red box) and also preserves nonprominent information from the infrared image (the bushes on the right side of the person in green box). In the tunnel scene, we retain information from darker areas that is closest to the intensity of infrared (the area in the red box), a feat that most other methods struggle to achieve; moreover, we effectively preserve the visible content at the exit. All in all, our method achieves a fusion effect that surpasses the current state-of-the-art in terms of visual appeal.

Quantitative comparisons. Besides, we make comparisons on three datasets using four objective metrics, including Mutual Information (MI) [Qu et al., 2002], Standard Deviation (SD), Visual Information Fidelity (VIF) [Han et al., 2013], and Sum of Correlation Differences (SCD) [Aslantas and Bendes, 2015]. As shown in Figure 7, our results demonstrate consistent advantages in these statistical metrics. Specifically, higher MI and SCD indicate that we are able to significantly preserve the correlated information between source images. Excellent SD and VIF suggest that our results possess higher contrast and are in line with visual perception.

4.5 Analysis of the Proposed Framework

Model-irrelevant generality. In essence, CAF represents a generalized learning paradigm that can potentially be directly applied to existing works. It effectively introduce the responses/guidance from percetion tasks by the bi-level optimization, enabling the adaptive search for loss functions that are tailored to the current network. This approach enhances the understanding of model characteristics at the level of loss functions, facilitating better alignment with downstream perception tasks. In this regard, we explore the application of our CAF using representative works including U2Fusion, ReCoNet, and UMFusion as the fusion network F.

Table 3 presents the quantitative results of fusion visual metrics and task-specific metrics before and after (with P) employing our CAF for adaptive search. They all achieved consistent improvements, both in terms of fusion quality and performance on perception tasks, which highlight the flexibility and strong model-irrelevant generality of our framework.

Effectiveness of automatic loss function search. To verify the effectiveness of the proposed framework, we constructed several variants that discard the search process (H/R/A represent handcrafting/random/average initialization of β). The last three rows of Table 3 present the related results. Specifically, the manually designed variant introducing prior knowledge exhibited some effectiveness than policy-free initialization. It is evident that results of CAF are far superior to any other variants in both fusion and task performance.

5 Conclusion and Remark

In this paper, we established a compact, automatic, and flexible framework for infrared and visible image fusion and applications. We not only conducted an in-depth exploration to demonstrate the exceptional characteristics of CAF, but also performed comprehensive experiments to affirm our superior performance in both image fusion and its follow-up high-level tasks.

Broader impacts. From the task perspective, CAF provides a solution paradigm to better understand the relationship between fusion and subsequent tasks. In terms of method design, as a highly universal learning search framework, CAF is capable of being inspiringly applied to existing methods and providing reference and guidance for the manual design of loss functions.

Acknowledgments

This work is partially supported by the National Key R&D Program of China (No. 2022YFA1004101), the China Postdoctoral Science Foundation (2023M730741), the National Natural Science Foundation of China (Nos. U22B2052, 62302078, 61936002) and the Liaoning Revitalization Talents Program (No. 2022RG04).