Bi-level Dynamic Learning for Jointly Multi-modality Image Fusion and Beyond

1 Introduction

In real-world scenarios, multi-sensor vision systems play a fundamental role in intelligent applications, e.g., autonomous driving and robotics. With the rapid deployment of sensor hardware, how utilizing multi-modality images to provide comprehensive scene parsing become an urgent issue [Liu et al., 2020; Jiang et al., 2022; Liu et al., 2021c]. Among them, infrared and visible sensors are two widely-used devices, aiming to capture the complete scene information. In detail, visible images can effectively describe the texture details but are sensitive to illumination changes. Infrared images can effectively highlight thermal targets but lack texture details. Thus, combining the diverse modalities into one image to preserve complementary characteristics is a significant way for both observation [Xu et al., 2022; Zhao et al., 2023b] and understanding [Li et al., 2022; Zhao et al., 2022; Ma et al., 2022].

∗Corresponding author

Benefiting from the effective feature extraction of deep learning, various learning-based image fusion approaches have been developed, which can be divided into two mainstream categories, i.e., auto-encoders [Liu et al., 2021a; Li and Wu, 2018] and end-to-end learning approaches [Xu et al., 2022; Zhang and Ma, 2021; Liu et al., 2023]. As for auto-encoders, ℓ1 norm [ ], weighted average [Liu et al., 2021a] and maxing choose [Zhao et al., 2020] are widely used for fuse features from pretrained autoencoders. Introducing diverse statistic measurements (e.g., modal gradients [Zhang and Ma, 2021] and image quality assessment [Xu et al., 2022]), various end-to-end learning methods are proposed by designing effective architectures. However, we argue that most fusion methods focus on visual quality, ignoring the role to facilitate the downstream perception tasks. Constrained by the visual statistics, typical features, both benefit for visual quality and perception, are easy to be neglected, causing insufficient learning of both tasks.

Lately, few works [Liu et al., 2022a; Tang et al., 2022; Sun et al., 2022] attempt to jointly realize the pixel-wise image fusion and semantic perception tasks. These works mostly cascade related networks directly and utilize end-toend training with multi-task loss functions to realize taskdriven image fusion. Unfortunately, there are two shortcomings that limit their performance. (i) Lacking the investigation of underlying connection: Joint learning may place an obstacle to preserving distinct features of tasks and cannot formulate intrinsic mutual-promoted relationships. (ii) Inflexible trade-off of multi-task learning: Existing methods mostly utilize manual hyper-parameters to balance the diverse loss functions, which cannot guarantee optimal performances for both tasks. Thus, the primary goal of this paper is to realize comprehensive image fusion, in order to achieve joint promotion both for observation and semantic understanding.

1.1 Our Contribution

To mitigate these issues, we develop a generic bi-level dynamic learning paradigm for bridging the relationships jointly between multi-modality image fusion and semantic percep-

Algorithm 1 Dynamic Aggregated Solution.

Require: Multi-modality inputs x, y, loss function ϕV, ϕP, and Φ and other necessary hyper-parameters.

• 1: Preparing pairs {x, y} with perception labels z∗ .
• 2: %Warm start phase.
• 3: Pretrain the fusion network F for initializing ω.
• 4: while not converged do
• 5: θV ← θV -∇θV ϕV(TV(u; θV(ω))).
• 6: θP ← θP -∇θP ϕP(TP(u; θP(ω))).
• 7: Calculate gradient GV and GP with Eq. (4).
• 8: Generate λV and λP by using RLW.
• 9: ω ←ω - (λVGV +λPGP).
• 10: end while
• 11: return ω∗ , θ∗ P .

comprehensive perception. These networks can be directly decomposed as an image fusion module (denoted as F) and a task-oriented module (denoted as T ). These mainstream architectures can be formulated as N = F ◦ T , training by the combination of diverse loss functions. We argue that existing methods rely on handcrafted visual measurements, which are not adaptative and flexible for joint learning. Thus, we propose a hierarchical dual-tasks driven model. In detail, as for image fusion, we introduce two dense residual blocks [Yan et al., 2019] to composite the network F with parameters ω, to maintain the complementary characteristics from source images for generating fused images u. Supposing infrared and visible images as x and y with gray-scale, the fusion learning can be written as u = F(x, y; ω). In order to measure the adaptive intensity distribution, we introduce a discriminator as TV with parameters θV to measure the texture similarity with source images. Denoted the classified output as zV, the discrimination can be formulated as zV = TV(u; θV). This formulation can provide adaptive learnable responses, compared with handcrafted loss functions.

Moreover, as for the semantic understanding, two representative networks for object detection [Tian et al., 2019] and semantic segmentation [Xie et al., 2021]) are selected as the task module TP with parameters θP. Similarly, the task solution can be written as zP = TP(u; θP). Thus, the complete

tion tasks. Concretely, we first establish a hierarchical deep model, composited by an image fusion module, visual discriminator, and commonly used perception network. We introduce dual learnable modules for the measurement of visual quality and semantic perception respectively, to provide distinct task-specific responses for image fusion. More importantly, a bi-level learning paradigm is proposed to formulate the latent connection of hierarchical modules. We also derive a dynamic weighting aggregation with efficient approximation to realize the mutually reinforcing of visual results and perception jointly. Figure 1 demonstrates our proposed strategy achieves better visual-appealing fused images and precise semantic perception performance (detection and segmentation) against state-of-the-arts. Our contributions can be concluded as follows:

• Considering the visual quality and semantic information richness of image fusion as two correlative goals, we propose a hierarchical deep model to realize the mutually reinforced task-driven image fusion.
• For the training strategy, we devise a bi-level formulation to bridge the image fusion with two task-specific constraints, providing an efficient way to formulate their inner mutual-benefit relationship.
• For the solving procedure, we drive a dynamic aggregated solution, yielding the efficient gradient approximation and adaptive weighting schemes to balance the gradients from diverse modules automatically, for learning the optimal parameters for both tasks.
• Comprehensive evaluations on three multi-modality vision tasks (i.e., image fusion, object detection, and semantic segmentation) are conducted to illustrate the superiority against state-of-the-art methods. Sufficient analytical results also substantiate the effectiveness.

2 The ProposedMethod

2.1 Motivation

As aforementioned, the most straightforward way is to establish cascaded architectures [Liu et al., 2022a; Tang et al., 2022; Zhao et al., 2023a] (denoted as N) for realizing the cascaded architecture can be formulated as:

where k ∈ {V, P} and the workflow is shown in Figure 2 (a). Specifically, compared with multi-task learning, the fusion network F actually plays a role for robust feature extraction (can be viewed as "encoder"). TV and TP are as task-specific "decoders" introduced to learn the ability to discriminate fusion quality and measure the informative richness to support the downstream scene perception.

2.2 Bi-level Formulation

Recently, various training strategies are proposed to address the high-level task-driven image fusion, including unrolled end-to-end training [Liu et al., 2022a], separate stage-wise training [Wu et al., 2022] and adaptive loop-based training [Tang et al., 2022]. However, we emphasize that these optimization strategies cannot model the coupled mutual-benefit relationship between visual quality and semantic reinterpretation, which is untoward to balance the influences of distinct tasks. Therefore, designing the learning paradigm to realize the "Best of Both Worlds" simultaneously is the core goal of this paper. In this part, we provide a bi-level formulation to depict the overall optimization procedure, in order to illustrate the mutual collaboration and guidance between visual inspection and semantic perception. The bi-level learning [Liu et al., 2021b] can be formulated as:

V = argmin

θ∗

θV

P = argmin

θP

θ∗

ϕV(TV(u; θV(ω))),

ϕP(TP(u; θP(ω))), where u = F(x, y; ω∗ ). Φ and ϕ are the objectives on the validation and training datasets respectively. λk represents the dynamic multi-task trade-off parameters. To be more specific, the primary part is to optimize the fusion network F for extracting rich features, which can be a benefit for the visual quality and semantic perception, expressed by Eq. (2). Furthermore, the discrimination of visual effects and responses of semantic understanding are two vital constraints to provide diverse task-specific information, as shown in Eq. (3). On the other hand, the hierarchical formulation between Eq. (2) and Eq. (3) are nested with mutual promotion. Fused images u are the fundamental data dependency for following task learning. Based on the responses from vision tasks, task-driven feedback can assist in the optimization of fusion from downstream vision tasks.

2.3 Dynamic Aggregation Solution

This part details the solution to address the aforementioned bi-level formulation (Eq. (2) and Eq. (3)). In order to accelerate the training convergence, we first introduce a warm-start strategy to pretrain the fusion network. Then, we present a dynamic aggregation solution to jointly address fusion and perception. The concrete optimization procedure is shown in Figure 2 (b). It actually can be expressed with hierarchical optimization, i.e., task learning (Eq. (3)) and task-guided fusion learning (Eq. (2)). Following with existing practical strategies [Liu et al., 2021b], we first optimize the lower-level task constraints with several steps to estimate the optimal parameters θ∗ V and θ∗ P , in order to learn the measurement for visual quality and perception based on task-specific losses.

Considering the mutual influence between fusion and lower vision tasks, represented by θk(ω), k ∈ {V, P}, there actually exists a complicated connection between the hierarchical tasks, which can be used to measure the response of tasks facing with the changes of fused images. As for the op-

s.t.

(3)

timization of image fusion, dual gradients can be obtained, which can be written as:

In detail, the gradient GV and GP are computed by Φk(F(x, y; ω) ◦ Tk(θk)). The first term is a direct gradient in term of ω and the second term depicts the latent coupled connection with follow-up perception tasks.

First-order approximation. In literature, solving Eq. (4) is a challenging issue, where the bottleneck is to compute the second-order gradient (the second term). Inspired by the Gaussian-Newton approximation, which provides a first-order computation to address continuous learning [Zhou et al., 2021a] and generative adversarial learning [Liu et al., 2022b], we introduce this strategy to approximate the Hessian in gradients GV and GP. Based on the implicit function theory, we can drive that ∇ωθ(ω) =

-∇2ω,θ Φ(ω; θ)∇2θ,θΦ(ω; θ)-1 . Gaussian-Newton approximation can covert the complicated Hessian matrix with the production of first-order vectors, i.e.,

Dynamic gradient aggregation. Another challenging issue is how to adaptively balance the gradients GV and GP to jointly optimize the image fusion network F. Recently, Random Loss Weighting (RLW) [Lin et al., 2022] is advanced in Multi-Task Learning (MTL), which can avoid the local minima with higher generalization and comparable performance. We leverage normal distribution p(λ) to generate λV and λP, to avoid the focus on one strongly correlated task and ignoring another. The whole solution is summarized in Alg. 1.

2.4 Loss Functions

In this part, we will elaborate the concrete loss functions to define ϕ and Φ respectively, which can be divided into two parts for visual quality and semantic perception respectively.

As for the learning of discriminator TV [Isola et al., 2017], we introduce the generative adversarial mechanism to discriminate the visual quality of image fusion. In concrete, we first construct the pseudo fused images um to maintain the salient informations by saliency weight maps (m1 and m2) using VSM [Ma et al., 2017], i.e., um = m1x + m2y. We also gradient-penalty Wasserstein strategy [Gulrajani et al., 2017] to guarantee the learning stability, thus the concrete formulation to train Tv can be written as

where RPenalty is the penalty term, calculated by E˜s∼Pfake [(∥∇uTV(u)∥2 - 1)2 ] and η is a trade-off term.

Furthermore, image fusion network F can be considered as the generator. In order to balance the pixel intensity and avoid the texture artifact, we also leverage the pixel error loss for the fusion learning, i.e.,

As for the semantic perception optimization of TP, we adopt the common task-specific loss functions to training the perception-related objectives (i.e., ΦP and ϕP). For object detection, we utilize the hybrid loss functions from FCOS [Tian et al., 2019] to define the objective. As for the semantic segmentation, common to previous literature, we utilize the cross-entropy loss function.

3 Experiments

3.1 Implementation Details

We implemented the experiments on five representative datasets (TNO, RoadScene [Xu et al., 2022] and M3FD for visual evaluations, M3FD [Liu et al., 2022a] and MultiSpectral [Takumi et al., 2017] for detection, and MFNet [Ha et al., 2017] for segmentation). SGD optimizer is utilized to update the parameters of each module. As for the perception task, the initialized learning rate is 2e-4 and will be decreased to 2e-6 with a multi-step decay strategy. As for the optimization of fusion, we utilize the same learning rate. All experiments are implemented with the PyTorch framework and on an NVIDIA Tesla V100 GPU.

3.2 Evaluation inMulti-modality Image Fusion

We conducted qualitative and quantitative analyses with ten state-of-the-art competitors, including DDcGAN [Xu et al., 2019], AUIF [Zhao et al., 2021], MFEIF [Liu et al., 2021a], SDNet [Zhang and Ma, 2021], DIDFuse [Zhao et al., 2020], DenseFuse [Li and Wu, 2018], ReCoNet [Huang et al., 2022], UMFusion [Wang et al., 2022], TarDAL [Liu et al., 2022a] and U2Fusion [Xu et al., 2022]. 1

Qualitative comparisons. The qualitative results on diverse challenging scenarios are depicted in Figure 3. It can be clearly observed that our method outperforms other methods in two eye-catching aspects. First, our method sufficiently reserves the high-contrast information from infrared images. For instance, in all scenarios, the buildings and pedestrians are best highlighted. Furthermore, our method reserves visible structure on the premise of retaining significant infrared characteristics. Overall, our method highlights the vital information (e.g., pedestrians) and suppresses the disturbing information (e.g., large darkness).

1 "Fusion with detection" is selected for comparison.

Quantitative comparisons. We also report the numerical results with the other ten fusion competitors on TNO, RoadScene, and M3FD in Table 1. We adopt three objective metrics for analysis. We can clearly observe that our method demonstrates superiority in terms of these statistical metrics. Specifically, the highest MI and FMI indicate our method can effectively integrate source information, e.g., edges and contrast. Moreover, the significant improvement of VIF implies that our method is consistent with human vision.

3.3 Comparing with SOTA in Object Detection

Qualitative comparisons. We visualize the object detection results in Figure 4. At the first row, it can be clearly observed that DIDFuse, TarDAL, and UMFusion lead to the classification or dislocation of the bounding box (e.g., the truck to be detected completely). On the contrary, our method sufficiently preserves the information of categories and results in accurate detection. As for the detection of small targets, i.e., in the second row, our method detects almost all cars while other methods lead to the omission.

Quantitative comparisons. The quantitative detection results are presented in Table 2. We can see that our method substantially outperforms other methods in terms of most categories and realizes 10.8% and 9.6% higher detection mAP on the M3FD and Multi-Spectual datasets, respectively. Apart from detection results, we further conduct the efficiency analysis in Table 2. Obviously, benefiting from the dilated convolutions, the minimal time consumption indicates that our method achieves real-time image fusion.

3.4 Extension toMulti-modality Segmentation

Besides comparing with the advanced image fusion, we also provide the evaluations with three specifically-designed methods: EGFNet [Zhou et al., 2021b], ABMDRNet [Zhang et al., 2021] and LASNet [Li et al., 2022].

Qualitative comparisons. We select two challenging scenarios to illustrate the remarkable performance compared with eight methods in Figure 5. Two significant advantages can be concluded from these instances. Firstly, our scheme can effectively preserve the complete structure of thermalsensitive targets (e.g., pedestrian) with precise prediction. For example, the pedestrian in the distance cannot be accurately estimated by most methods. Secondly, our scheme also highlights abundant texture details for the normal categories (e.g., car and curve). Obviously, our method can precisely estimate the structure of cars on both image pairs.

Quantitative comparisons. Table 3 reports the concrete numerical results on eight categories of the MFNet dataset. Specifically, compared with SeaFusion, which is a segmentation-driven method, our scheme drastically improves 15.2 % of mIOU. Noting that, only utilizing one backbone network, our method can achieve comparable performance with specially designed segmentation methods (e.g, LASNet and EGFNet). Especially, we achieve better accuracy in the classification of persons and cars, which is vital for the employment of intelligent vision systems.

3.5 Ablation Studies

Effects of proposed training strategy. To demonstrate the effectiveness of the bi-level learning paradigm, we conduct the comparison with two mainstream training strategies (i.e., unrolled training [Liu et al., 2022a] and adaptive loop training [Tang et al., 2022]). Table4 reports the numerical results of image fusion and detection based on these strategies. Both existing strategies are concrete on one task and neglect another. We can observe that the proposed strategy achieves consistent performance for the two scene parsing tasks.

Evaluating of dynamic aggregation strategy. In order to evaluate the effectiveness of dynamic weights, we provide other four classical adjustment ways, i.e., manual-design, equal weighting, gradient normalization [Chen et al., 2018] and DWA [Liu et al., 2019]. The numerical results (trained with 10 epochs) are reported in Table 5. The random aggregation can effectively avoid the local optimum with higher performance compared with representative multi-task learning and manual adjustment.

Evaluating of generalization ability. The proposed bilevel dynamic learning is a generalized scheme, which is network-agnostic. Table 6 illustrates the significant improvement for advanced fusion networks (i.e., TarDAL and UMFusion) using our training strategy. Subscripts "J" and "P" denote the joint and proposed training strategies.

Influences of perception tasks for image fusion. We also analyze the influence of diverse perception guidance for image fusion in Table 7 and Figure 6 respectively. We can observe that both quantitative results are higher than current advanced methods. Detection-guided fusion (denoted as "DgF") realizes better results on TNO, which contain abundant targets with high contrast. Segmentation-guided fusion (denoted as "SgF") achieves a significant improvement on M3FD, which is with vivid texture details. From Figure 6, we can observe DgF trends to preserve the salient targets (e.g., person). The results of SgF prefer to maintain the scene background with obvious structural information for pixelwise classification.

4 Conclusion

In this paper, a hierarchical dual-task deep model was proposed to bridge multi-modality image fusion and semantic perception jointly. We presented a bi-level formulation to depict the coupled relationship, to realize mutual promotion. Then a dynamic aggregation strategy was proposed to combine the gradients from distinct tasks to achieve the comprehensive image fusion. Extensive experiments demonstrated our method is superior to address diverse vision tasks.

Acknowledgments

This work is partially supported by the National Key R&D Program of China (2020YFB1313503), the National Natural Science Foundation of China (U22B2052), the Fundamental Research Funds for the Central Universities and the Major Key Project of PCL (PCL2021A12).