DIDFuse: Deep Image Decomposition for Infrared and Visible Image Fusion

1 Introduction

Image fusion is an image processing technique for information enhancement. The principle is to preserve the complementary and redundant information from source images containing the same scene without artifacts [Meher et al., 2019]. In image fusion, the infrared and visible image fusion, a.k.a. IVIF, can be applied to many domains, such as surveillance [Bhatnagar and Liu, 2015], modern military and fire rescue tasks [Lahoud and Susstrunk, 2018; Hu et al., 2017], face recognition [Ma et al., 2016] and so on.

Proverbially, infrared images can avoid visually cognitive obstacles caused by illumination changes and artifacts, but they are with low spatial resolution and poor texture detail information. Conversely, visible images are with high spatial resolution and rich information of appearance and gradient, while they are easily affected by obstructions and light reflections. Therefore, making the fusion image retain both thermal radiation information of the infrared images and gradient in- In general, the IVIF algorithms can be divided into two groups: traditional methods and deep learning methods. Specifically, representative traditional methods include image multi-scale transformation [Li et al., 2011], sparse representation [Zong and Qiu, 2017], subspace learning [Patil and Mudengudi, 2011] and the saliency based method [Zhang et al., 2017].

∗Co-first Author

†Corresponding Author

Currently, deep learning (DL) has emerged as a prevalent tool in the field of IVIF. DL based methods can be categorized into three groups. The first group is based on Generative Adversarial Networks (GANs). In FusionGAN [Ma et al., 2019b], a generator creates fused images with infrared thermal radiation and visible gradient information and a discriminator forces the fused images to have more details from the visible images. In the light of Conditional GANs [Mirza and Osindero, 2014], detail preserving GAN [Ma et al., 2020] changes the loss function of FusionGAN for improving the quality of detail information and sharpening the target boundary. The second group [Li et al., 2018; Lahoud and S¨usstrunk, 2019] is an extension of image multiscale transformation. Generally speaking, they transform images from the spatial domain to background and detail domains by means of filters or optimization based methods. Background images are simply averaged. Since there are high-frequency textures in detail images, they fuse feature maps of detail images extracted from a pre-trained network (for example, VGG [Simonyan and Zisserman, 2014]). At last, a fusion image is recovered by merging the fused background and detail images. The third group consists of AE based methods [Li and Wu, 2018]. In the training phase, an AE network is trained. In the test phase, they fuse feature maps of source images, which then pass through the decoder to recover a fusion image. In summary, in DL based methods, deep neural networks (DNNs) are often employed to extract features of input images and then a certain fusion strategy is exploited to combine features to complete the image fusion task.

It is worth pointing out a shortcoming of the second group, i.e. DL is used only in the fusion stage and they employ filters or optimization based methods in the decomposition stage. To overcome this shortcoming, by combining principles of the second and third groups, we propose a novel IVIF network, called deep image decomposition based IVIF (DIDFuse). Our contributions are two-fold:

(1) To the best of our knowledge, this is the first deep image decomposition model for IVIF task, where both fusion and decomposition are accomplished via an AE network. The encoder and the decoder are responsible for image decomposition and reconstruction, respectively. In training phase, for decomposition stage, the loss function forces background and detail feature maps of two source images similar/dissimilar. Simultaneously, for reconstruction stage, the loss function maintains pixel intensities between source and reconstructed images, and gradient details of the visible image. In the test phase, background and detail feature maps of test pairs are separately fused according to a specific fusion strategy, and then the fused image can be acquired through the decoder.

(2) As far as we know, the performance of existing IVIF methods [Ma et al., 2016; Li and Wu, 2018; Zhang et al., 2017; Li et al., 2018] is only verified on a limited number of handpicked examples in TNO dataset. However, we test our model on three datasets, including TNO, FLIR and NIR. In total, there are 132 test images with indoor and outdoor scenes, and with daylight and nightlight illuminations. Compared with SOTA methods, our method can robustly create fusion images with brighter targets and richer details. It can be potentially utilized in target recognition and tracking.

The remaining article is arranged as follows. Related work is introduced in section 2. The mechanism of the proposed network is described in section 3. Then, experimental results are reported in section 4. At last, some conclusions are drawn in section 5.

2 RelatedWork

Since our network structure is closely related with U-Net, we introduce U-Net in section 2.1. Then, traditional two-scale image decomposition methods are briefly reviewed in section 2.2.

2.1 U-Net and Skip Connection

U-Net is applied to biomedical image segmentation[Ronneberger et al., 2015], similar to AE network, U-Net consists of a contracting path for feature extraction and an expanding path for precise localization. Compared with AE, there is a channel-wise concatenation of corresponding feature maps from contracting and expanding paths in U-Net. In this manner, it can extract "thicker" features that help preserve image texture details during downsampling. In literature [Mao et al., 2016], a U-Net-like symmetric network is used for image restoration. It employs skip connection technique, where feature maps of convolution layers are added to corresponding deconvolution layers to enhance the information extraction capability of the neural network and to accelerate convergence.

2.2 Two-Scale Decomposition

As a subset of multi-scale transformation, two-scale decomposition in IVIF decomposes an original image into background and detail images with background and target information, respectively. In [Li and Wu, 2018], given an image I, they obtained the background image I b by solving the following optimization problem,

where ∗ denotes a convolution operator, and gx = [-1, 1] and gy = [-1, 1]T are gradient kernels. Then, the detail image is acquired by I d = I - I b . Similarly, a box filter is used to get the background image in [Lahoud and S¨usstrunk, 2019], and the method of obtaining the detail image is the same as that of [Li and Wu, 2018]. After decomposition, background and detail images are separately fused with different criteria. At last, the fused image is reconstructed by combining fused background and detail images.

3 Method

In this section, we will introduce our DIDFuse algorithm and the proposed network structure. In addition, details of training and testing phases are also illustrated.

3.1 Motivation

As described in section 2.2, two-scale decomposition decomposes the input image into a background image containing lowfrequency information with large-scale pixel intensity changes and a detail image embodying high-frequency information with small-scale pixel intensity changes. Currently, most algorithms incorporate certain prior knowledge, and employ filters or optimization based methods to decompose images. Hence, they are manually designed decomposition algorithms. We highlight that image decomposition algorithms are intrinsically feature extractors. Formally, they transform source images from spatial domain into feature domain. It is well known that the DNN is a promising data-driven feature extractor and has great superiority over traditional manually-designed methods. Unfortunately, it lacks a DL based image decomposition algorithm for IVIF task.

Consequently, we present a novel deep image decomposition network in which an encoder is exploited to perform two-scale decomposition and extract different types of information, and a decoder is used to recover original images.

3.2 Network Architecture

Our neural network consists of an encoder and a decoder. As illustrated in Figure 1, the encoder is fed with an infrared or a visible image and generates background and detail feature maps. Then, the network concatenates two kinds of feature maps along channels. At last, concatenated feature maps pass through decoder to recover the original image. To prevent the detail information of the feature maps from being lost after multiple convolutions and to speed up the convergence, we add the feature maps from the first and second convolutions to the inputs of the last and penultimate convolutions, and the adding strategy is concatenating the corresponding feature maps along channels. As a consequence, the pixel intensity and gradient information of the source images can be better retained in the reconstructed image.

Table 1 lists the network configuration. Encoder and decoder contain four and three convolutional layers, respectively. Each layer consists of a padding, a 3 × 3 convolution, a batch normalization and an activation function. The first and the last layers utilize reflection padding to prevent artifacts at the edges of the fused image. Activation functions of conv3 and conv4 are set to the hyperbolic tangent function (tanh) since they output background and detail feature maps. As for conv7, it is activated by sigmoid function since it reconstructs original images. Other layers are followed by parametric rectified linear units (PReLU).

3.3 Loss Function

In the training phase, we aim to obtain an encoder that performs two-scale decomposition on the source images, and at the same time, acquire a decoder that can fuse the images and preserve the information of source images well. The training process is shown in Figure 1(a).

Image decomposition. Background feature maps are used to extract the common features of source images, while detail feature maps are used to capture the distinct characteristics from infrared and visible images. Therefore, we should make the gap of background feature maps small. In contrast, the gap of detail feature maps should be great. To this end, the loss function of image decomposition is defined as follow,

where BV , DV are the background and detail feature maps of the visible image V , and BI , DI are those of the infrared image I. Φ(·) is tanh function that is used to bound gap into interval (-1, 1).

Image Reconstruction

As for image reconstruction, to successfully retain the pixel intensity and detailed texture information of input images, the reconstruction loss function is given by

where I and ˆ I, V and ˆ V represent the input and reconstructed images of infrared and visible images, respectively. ∇ denotes the gradient operator, and

where X and ˆX represent the above input image and the reconstructed image, and λ is the hyperparameter. SSIM is the structural similarity index[Wang et al., 2004], which is a measure of the similarity between two pictures. Then LSSIM can be described as

Remark that L2-norm measures the pixel intensity agreement between original and reconstructed images, and that LSSIM computes image dissimilarity in terms of brightness, contrast and structure. Specially, since visible images are with enriched textures, the reconstruction of visible images is regularized by gradient sparsity penalty to guarantee texture agreement.

Combining Eqs. (1) and (2), the total loss Ltotal can be expressed as

where α1, α2, α3, α4 are the tuning parameters.

3.4 Fusion Strategy

In the above subsections, we have proposed network structure and loss function. After training, we will acquire a decomposer (or say, encoder) and a decoder. In the test phase, we aim to fuse infrared and visible images. The workflow is shown in Figure1(b). Different from training, a fusion layer is inserted in the test phase. It fuses background and detail feature maps separately. In formula, there is

where BF and DF denote the fused background and detail feature maps, respectively. In this paper, three fusion strategies are considered as follows:

• Summation method: BF = BI ⊕BV , DF = DI ⊕DV , where the symbol ⊕ means element-wise addition.
• Weighted average method: BF = γ1BI ⊕ γ2BV , DF = γ3DI ⊕ γ4DV , where γ1 + γ2 = γ3 + γ4 = 1 and the default settings for γi(i = 1, · · ·, 4) are all equal to 0.5.
• L1-norm method: Referring to [Li and Wu, 2018], we use the L1-norm as a measure of activity, combining with the softmax operator. In detail, we can obtain the activity level map of the fused background and detail feature maps by ‖Bi(x, y)‖1 and ‖Di(x, y)‖1 (i = 1, 2), where B1, B2, D1 and D2 represent BI, BV , DI and DV , and (x, y) represents the corresponding coordinates of the feature maps and the fused feature map. Then the adding weights can be calculated by:

where ψ(·) is a 3 × 3 box blur (also known as a mean filter operator). Consequently, we have

where ⊗ means element-wise multiplication.

4 Experiment

The aim of this section is to study the performance of our proposed model and compare it with other SOTA models, including FusionGAN [Ma et al., 2019b], Densefuse [Li and Wu, 2018], ImageFuse [Li et al., 2018], DeepFuse [Prabhakar et al., 2017], TSIFVS [Bavirisetti and Dhuli, 2016], TVADMM [Guo et al., 2017], CSR [Liu et al., 2016] and ADF [Bavirisetti and Dhuli, 2015]. All experiments were conducted with Pytorch on a computer with Intel Core i7-9750H CPU@2.60GHz and RTX2070 GPU.

We employ six metrics to evaluate the quality of a fused image, that is, entropy (EN), mutual information (MI), standard deviation (SD), spatial frequency (SF), visual information fidelity (VIF) and average gradient (AG). More details for these metrics can be seen in [Ma et al., 2019a].

Datasets and preprocessing. Our experiments are conducted on three datasets, including TNO [Toet and Hogervorst, 2012], NIR [Brown and S¨usstrunk, 2011] and FLIR (available at https://github.com/jiayi-ma/RoadScene). In our experiment, we divide them into training, validation, and test sets. Table 2 shows the numbers of image pairs, illumination and scene information of the datasets. We randomly selected 180 pairs of images in the FLIR dataset as training samples. Before training, all images are transformed into grayscale. At the same time, we center-crop them with 128 × 128 pixels.

Hyperparameters setting. The tuning parameters in loss function are empirically set as follows: α1 = 0.05, α2 = 2, α3 = 2, α4 = 10 and λ = 5. In training phase, the network is optimized by Adam over 120 epochs with a batch size of 24. As for learning rate, we set it to 10-3 and decrease it by 10 times every 40 epochs. Figure 2 displays loss curves versus epoch index. It is shown that all loss curves are very flat after 120 epochs. In other words, the network is able to converge with this configuration.

4.1 Experiments on Fusion Strategy

As described in section 3.4, fusion strategy plays an important role in our model. We investigate the performance of three strategies on validation set. Table 3 reports numerical results of six metrics on validation set. Obviously, it is shown that summation strategy achieves higher values, especially in terms of SD, SF, VIF and AG. Hence, the following experiments adopt summation strategy.

4.2 Experiments on Image Decomposition

One of our contributions is the deep image decomposition. It is interesting to study whether decomposed feature maps are able to meet our demands. In Figure 3, it displays the first channels of feature maps which are generated by conv3 and conv4. It is evident that our method can separate the backgrounds and details of infrared and visible images. For background feature maps, it is found that BI and BV are visually similar, and they reflect the background and environment of the same scene. Conversely, the gap betweenDI andDV is large, which illustrates the distinct characteristics contained in different source images. That is, the infrared images contain target highlight and thermal radiation information while gradient and texture information of targets are involved in the visible images. In conclusion, it to some degree verifies the rationality of our proposed network structure and image decomposition loss function.

4.3 Comparison with Other Models

In this subsection, we will compare our model with the other popular counterparts.

Qualitative comparison. Figure 4 exhibits several representative fusion images generated by different models. Visual inspection shows that, in the images containing people, other methods have problems such as weak high-lighted objects, poor contrast and less prominent contour of targets and backgrounds. Similarly, if the images are natural landscapes, others have blurred boundaries of mountains and trees, poor color contrast, and insufficient sharpness. Conversely, our method can obtain fused images with brighter targets, sharper edge contours and retaining richer detailed information.

Quantitative comparison. Subsequently, quantitative comparison results on test set are listed in Table 4. It is found that our model is the best performer on all datasets in terms of all metrics. As for competitors, they may perform well on a dataset in terms of part of metrics. This result demonstrates that images fused by our model are with enriched textures and satisfy human visual system.

4.4 Experiments on Reproducibility

As is known, deep learning methods are often criticized for instability. Therefore, we test the reproducibility of DIDFuse in the last experiment. We repeatedly train the network 25 times and quantitatively compare the 25 parallel results. As shown in Figure 5, the black solid curves report six metrics over 25 experiments. The red dashed line and blue dotted line represent the greatest and the second greatest values in the comparison methods, respectively. Similar to the above results, our method can basically keep the first place all the time, indicating that DIDFuse can generate high-quality fused images steadily.

5 Conclusion

To solve the IVIF issue, we construct a new AE network in which the encoder is used for two-scale image decomposition and the decoder is responsible for image reconstruction. In the training phase, the encoder is trained to output background and feature maps, then the decoder reconstructs original images. In the test phase, we set a fusion layer between the encoder and decoder to fuse background and detail feature maps through a specific fusion strategy. Finally, the fused image can be acquired through the decoder. We test our model on TNO, FLIR, and NIR datasets. Qualitative and quantitative results show that our model outperforms other SOTA methods, since our model can steadily obtain a fusion image of highlighted targets and rich details.

Acknowledgments

The research is supported by the National Key Research and Development Program of China under grant 2018AAA0102201 and 2018YFC0809001, the National Natural Science Foundation of China under grant 61976174, 11671317 and 61877049, the Fundamental Research Funds for the Central Universities under grant number xzy022019059.