MMPN: Multi-supervised Mask Protection Network for Pansharpening

1 Introduction

Remote sensing images with high spatial and spectral resolution are widely used in various fields, such as change detection, rescue, navigation, and mapping [Deng et al., 2022]. However, due to the limitations of the physical properties of satellite sensors, high-spatial-resolution multispectral (HRMS) images cannot be directly obtained. Therefore, researchers proposed to fuse the spatial information of the panchromatic (PAN) image and the spectral information of the low-spatial-resolution multispectral (LRMS) image through pansharpening methods to obtain HRMS images. Up to now, in the pansharpening task, some related issues, such as how to extract the spatial details of PAN images more accurately and keep the spatial and spectral consistency between the fused HRMS image and the source images, are still hot research topics.

At present, the existing pansharpening methods are mainly divided into four categories [Vivone et al., 2020], i.e., component substitution (CS) methods [Tu et al., 2001; Xu et al., 2014; Aiazzi et al., 2002], multi-resolution analysis (MRA) methods [Ghassemian, 2016], variational optimization (VO) methods [Lu et al., 2021], and deep learning (DL)-based methods [Deng et al., 2022; Peng et al., 2022]. Among them, CS, MRA, and VO methods, called traditional methods, obtain HRMS images through filter estimation or sparse representation. Traditional pansharpening methods are simple to implement and have physical interpretabilities, but they rely on defined feature extraction methods and fusion rules to ensure the accuracy of results [Ghassemian, 2016].

Due to the powerful feature extraction ability of convolutional neural networks (CNNs), numerous DL-based pansharpening methods have been developed. PNN [Meng et al., 2022] is the first CNN-based pansharpening method, which extracts and fuses the features from PAN and LRMS images. MSDCNN [Yuan et al., 2018] proposed a multi-scale and multi-depth CNN that adopts multi-scale feature extraction and residual connection for the pansharpening. PCDRN [Yang et al., 2020] proposed a cascaded progressive residual network by increasing the depth of the network to obtain more accurate spatial information. ColorGAN [Ozcelik et al., 2020] provided a solution for colorizing PAN images, which constructs a generative adversarial network (GAN) to realize the generation of spectral information. TFNet [Liu et al., 2020] proposed a two-stream fusion network, which extracts features of PAN and LRMS images by constructing two different branches. FusionNet [Deng et al., 2022] proposed a network to estimate the injection details between LRMS and HRMS images. DSCNN [Yang et al., 2021] proposed a dual-stream CNN for pansharpening by constructing an information complementation block to extract and enhance spatial details at different resolutions. ADKNet [Peng et al., 2022] was proposed for pansharpening by building source-adaptive discriminative kernels from PAN and LRMS images. TDNet [Zhang et al., 2022] proposed a fusion network with doublelevel, double-branch, and double-direction structures to fully exploit spectral information with MRA.

The DL-based pansharpening methods can achieve better objective performance indexes than the traditional methods. However, their results still have the problems of spectral distortion and edge blur compared to the ideal HRMS images [Deng et al., 2022; Meng et al., 2022]. The main reason is that the convolution operation in CNNs can capture abundant features by increasing the sizes of convolution kernels to expand the receptive field. However, the larger the convolution kernel, the greater the interference of adjacent pixels to the central pixel in the convolution area, so this operation will weaken the gradient information of image edges. In addition, most methods only consider the similarity of gradient information between PAN and HRMS images, but ignore the intensity change of adjacent areas at the edges between LRMS and HRMS images, which leads to spectral distortion in the fusion results. To solve the above problems, this paper first analyzes the change trend of intensity values between highresolution images and corresponding degraded images, and proposes a mask-protected strategy (MPS) that is used to protect edges and guide the network to separately learn the features of different regions on both sides of image edges. Then, based on MPS, a multi-supervised mask-protection network (MMPN) is constructed to fuse PAN and LRMS images. Finally, to obtain better fusion results and increase the generalization of the network, a joint multi-supervised loss function is defined. The contributions of this work are as follows:

1. An MPS is proposed by defining a mask protection matrix to protect the edge information from the interference of neighborhood information in feature extraction.
2. Based on MPS, an MMPN is constructed for pansharpening, which contains four branches with the same structure for multiple learning tasks. This network can obtain fusion results with dual fidelity of spectral and spatial information.
3. In each branch of MMPN, a dual-stream multi-scale fea-

ture fusion module (DMFFM) is designed to better extract and fuse the features of two input images by constructing two encoders and one decoder for feature extraction and fusion, respectively.

4) To better train the network, a multi-supervised loss function based on multiple tasks is defined, in which a down-scale loss term is designed to improve the generalization of the network.

2 Definition of MPS

The current pansharpening methods mainly use the spatial information of PAN images to improve the resolution of LRMS images. However, LRMS images exist low contrast between adjacent target areas, that is, the difference of pixel values in the adjacent areas on both sides of the edges is small. Therefore, if only considering supplement details of PAN images for LRMS images without considering the change trend of pixel values in the neighborhoods, which may cause edge blur and spectral distortion.

To better show the change of pixel values between a highresolution image and its corresponding low-resolution image, we take PAN image and its corresponding degraded image as an example. Figure 1 shows the change of pixel values of HP and LP images on a horizontal line, where HP and LP denote PAN image and its degraded version, respectively. It can be seen from Figure 1 that the intersection points of the two curves are about the positions where the pixel values change most steeply, or the positions where the gradient values are the largest, that is, the edge positions. If the pixel values of LP image are restored to those of HP image, on both sides of the edges, the pixel values of one side need to be raised, and those of another side need to be suppressed. Therefore, to realize the reconstruction from LP images to HP images, the pixel values of these two sides need different processing. Based on the above analysis, this paper proposes an MPS, which decomposes the areas on both sides of the intersection by constructing the mask protection matrices. In this way, the features of two sides need to be learned separately, and the image edges can be protected from being damaged.

In our network, an MPS is proposed to decompose an input image into two mask images, which contain adjacent regions with different pixel changes on both sides of the edges. The structure of MPS is shown in Figure 2. For pansharpening task, the proposed MPS is to protect the edges of the reconstructed HRMS image from being damaged. The calculation processes of MPS are as follows.

First, a PAN image Y HP is blurred through a modulation transfer function (MTF) [Vivone et al., 2020] to obtain a degraded image with low resolution, named Y LP .

For Y HP and Y LP , the pixels with same values are the intersection points. For other pixels, two mask protection matricesMP and MP INV are defined by Formulas 2 and 3. If the pixel values of Y HP are larger than those of Y LP , the element of MP is set to 1, otherwise set 0.

where (i, j) represent the pixel coordinates. Then, a data filling operation is performed on an input image (i.e., PAN, LRMS or HRMS) and two different mask protection matrices MP and MP INV to obtain two mask images named Mask data1 and Mask data2 , which contain complementary information of different areas. The data filling operation is defined as:

where ∗ represents an element-wise product.

Finally, the obtained mask data is used as the input of MMPN and to guide the network to separately learn the features of different regions on both sides of image edges.

3 ProposedMMPN

In this section, aiming at the problems of edge blur and spectral distortion existing in the current pansharpening methods, an MMPN is proposed, as shown in Figure 3. First, the input source images and the ground truths (GT) are decomposed by the proposed MPS to obtain the mask data and the mask supervised images (M GT1 and M GT2), respectively. Then, four DMFFMs with weight sharing are constructed to realize four network branches, which perform the fusion of the source images, the two mask data, and the low-resolution version of the source images, respectively. Finally, a joint loss function containing four loss terms is defined to train the network.

3.1 Dual-streamMulti-scale Feature Fusion Module (DMFFM)

DMFFM is constructed to realize the fusion of two images, and its structure is shown in Figure 3. The DMFFM is designed as a neural network with dual stream inputs and multiscale feature extraction layers to fit the distribution of supervised images. To reduce the confusion of two image features, two encoders with same structure are designed to respectively extract features from two input images, and a decoder is constructed to reconstruct the image features and output the fusion result.

In the encoder, a STEM layer is set at the first layer of DMFFM to extract the initial features. A feature extraction block (FEB) is constructed to extract the features of different scales in the encoders. The structure of FEB is shown in Figure 3, and it is designed as a residual structure, which consists of a depth-wise convolution, a LayerNorm, a pointwise convolution, and a ReLU function. Since the MPS used in the network can protect the edge information of the image, the impact of increasing the size of a convolution kernel on the edge can be reduced. Therefore, the convolution kernels with a large receptive field are used to extract features in the spatial dimension. In FEB, the depth-wise convolution with kernel size of 7×7 is adopted to extract features of spatial dimension, and the point convolution is used to integrate channel features and reduce the numbers of feature channels. In the decoder, multiple FEBs are also used to fuse features of different scales from the encoders and output the fused image.

3.2 Joint Loss Function

Since four DMFFMs generate four different fused images, they need to be trained by different supervised images according to different input images. The output of a DMFFM can be defined as follows.

where Wθ represents the network parameters, Yinput 1 and Yinput 2 represent the different network inputs, and Youtput represents the network output. Corresponding to different branches, Youtput represents different fused results including the fused HRMS image YF H , two fused mask data YF MH1 and YF MH2 , and the down-scaled fused image YF DH . To guarantee the consistency between the fused results and supervise images, the loss items corresponding to the four branches in MMPN are defined as follows.

where Lossrec, Lossmask1 , Lossmask2 , and Lossds represent the loss terms corresponding to four branches. Yhrms denotes the GT image, YM GT1 and YM GT2 represent mask images obtained by decomposing GT images using MPS, and YDS GT denotes the down-scaled GT image. Therefore, the joint loss function used to supervise the MMPN is defined as follows.

4 Experiments

To validate the effectiveness of the proposed MMPN 1 , subjective and objective experiments are conducted on the simulated and real satellite datasets, including IKONOS (4 bands), Pl´eiades (4 bands), and WorldView-3 (8 bands) In the simulated dataset, the LRMS images are generated according to the WALD protocol [Ghassemian, 2016] by using MTF and down-sampling operations on HRMS images. The sizes of LRMS and PAN images are 64×64 and 256×256, respectively. In real datasets, the sizes of LRMS and PAN images are 256×256 and 1024×1024, respectively.

In the experiments, the performance of the proposed MMPN is compared with those of state-of-the-art methods. The comparison methods include GSA [Aiazzi et al., 2007], DRPNN [Wei et al., 2017], MSDCNN [Yuan et al., 2018], PCDRN [Yang et al., 2020], ColorGAN [Ozcelik et al., 2020], TFNet [Liu et al., 2020], DSCNN [Yang et al., 2021], FusionNet [Deng et al., 2022], ADKNet [Peng et al., 2022], and TDNet [Zhang et al., 2022]. The benchmark images are obtained by the polynomial kernel method (EXP) [Aiazzi et al., 2002]. Significantly, all deep learning-based methods are retrained using the same datasets for fairness, and tested on the environment of NVIDIA GeForce RTX 3090 and INTEL 11700K.

1 The code is available at github.com/sharpeningNN/MMPN.

4.1 Experiments on Simulated Dataset

As shown in Figure 4, the subjective fusion images of all comparison methods on a pair of images from IKONOS dataset can be observed. From the figure, we can find that since the result of EXP is obtained by directly interpolating the LRMS image, it is the most blurred compared with those of other methods due to the lack of spatial details. The results obtained by other deep learning-based methods have the problems of edge blurring and spectral distortion. For example, the results of DSCNN, FusionNet, MSDCNN, DRPNN, PCDRN, TDNet, ADKNet, and TFNet have more blurry edges and lower brightness compared to GT. The result of GSA has serious spectral distortion, and our result is the closest to GT. To show the difference between fusion results and GT more clearly, we calculate the residual maps between them and GT, and show an enlarged local area and its corresponding residual map below each result. From the residual maps, it can be clearly seen that the results of comparison methods have obvious spectral distortion and lose some spatial details. The residual map of our result has the least residual information, which also indicates the effectiveness of the proposed method.

To further compare the performance of each method, the quantitative evaluation results are summarized in Table 1. The evaluation metrics [Deng et al., 2022], including the peak signal to noise ratio (PSNR), the root mean square error (RMSE), the relative average spectral error (RASE), the universal image quality index (UIQI), Qn2 , the spectral angle mapper (SAM), and the erreur relative globale adimensionnelle de synth`ese (ERGAS), are used to objectively evaluate the performance of different methods. For the objective evaluation experiments, the best results are marked with bold fonts, and the second-best results are underlined. In addition, the test time of all methods is given to compare the computational cost of each method. From Table 1, we can see that our method achieves the best results in all three datasets, which means that the proposed MMPN has a better fitting performance than other methods.

In terms of test time, since MMPN uses many convolution kernels with large size, the computational time is increased compared with other deep learning-based methods. However, due to the GPU support, MMPN still can achieve real-time performance.

4.2 Experiments on Real Dataset

Figure 5 shows the fusion results on a pair of images on the Pl´eiades dataset. To more clearly observe the difference of spectral and detail information between different fusion results, two small regions are selected and amplified. As can be seen from the figure, the result of EXP has the least spatial details. The result of GSA method has the problem of overinjection of details, but lacks of spectral information. Compared with other results, the result of ColorGAN displays a distinct color cast. It can be clearly observed from the enlarged areas that our result has more abundant spectral information than those of other comparison methods.

Due to the absence of GT, some non-reference quantitative metrics including Dλ , Ds , and QNR [Wald, 2000] are used to assess the similarity of spectral and spatial details between fusion images and source images. Dλ measures the spectral similarity between the fusion results and LRMS images, Ds measures the spatial similarity between the fusion results and PAN images, and QNR calculates the overall similarity through Dλ and Ds . As shown in Table 2, the results of EXP method have the lowest Dλ values, which indicates the EXP method maintains the best spectral information.

Model 3 � � 37.2155 1.9946

Table 4: Ablation experiment of different branches on the simulated data from Pl´eiades.

The results obtained by the traditional GSA method have lower QNR values that those of most comparison methods, which indicates that GSA method obtains worser pansharpening performance. Among the deep learning-based methods, the proposed MMPN has the highest QNR indexes on 4channels (Pl´eiades) and 8-channels (WorldView-3) datasets, which also indicates the advanced performance in dual fidelity of spectral and spatial information.

In summary, the proposed MMPN achieves advanced performance in both subjective visual effects and quantitative metrics compared with other methods on different datasets.

4.3 Application Experiment

To evaluate the application property of all comparison methods, image classification experiments are implemented for all fusion results. Referring to the reference [Lu et al., 2021], the ENVI tool is utilized for classification. The GT images in the simulated dataset are first fed to the classification model to obtain classification results, which are regarded as reference images for the classification results of other methods. The classification accuracy value is used to quantitatively evaluate the classification performance of different methods. Table 3 shows the classification results of Figure 4. As shown in Table 3, the proposed MMPN achieves the best classification accuracy than other comparison methods.

4.4 Ablation Experiment

In this section, some ablation experiments are conducted to demonstrate the effectiveness of MPS, downscale branch (DB, top branch in Figure 3), and DMFFM in MMPN. In the experiments of validating MPS and DB, three models containing different components are tested. Model 1 contains only one DMFFM, without MPS and DB. Model 2 contains three branches, using MPS but not DB. Model 3 denotes the proposed MMPN with MPS and DB. Table 4 shows the quantitative indexes obtained by the three models on the simulate dataset. The performance of model 2 is significantly improved compared with that of model 1, which indicates that MPS effectively protects the spatial features from being corrupted. The PSNR value obtained by model 3 is slightly lower than that of model 2, but the difference is not significant. This is because DB can be regarded as a data enhancement, which can increase the generalization of the network, but leads to a slight reduction in the fitting effect for a certain distribution. To verify that DB can improve the generalization of the network, the model 2 without DB is compared with model 3 on the real dataset , as shown in Table 5. It can be seen from Table 5 that the indicators obtained by model 3 with DB are all better than that of model 2. Therefore, the performance of DB is verified.

In DMFFM, FEB plays a key role. To prove the performance of FEB in DMFFM, the ablation experiments using two network structures to replace FEB are performed. The first structure uses convolutional block (ConvBlock) to replace FEB, and another structure uses a residual block (ResBlock) to replace FEB. As shown in Table 6, we can find that the DMFFM with FEB obtains the best performance than other network structures, which indicate the effectiveness of FEB.

5 Conclusion

In this paper, firstly, based on the analysis of the gray value changes between the high-resolution image and the corresponding degraded image, an MPS is proposed to protect the spatial information from being destroyed in the feature learning of the network. Then, based on this strategy, an MMPN containing four branches with the same structure is proposed to realize the learning of multiple fusion tasks, and each task is realized through the constructed DMFFM. Finally, different loss items are defined for the four task branches and combined into a joint loss function for network training. Experimental results show that our method achieves better performance than other advanced methods in terms of subjective vision and objective indicators.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No. 62072218, No. 61862030, and No. 62261025) and the Project of the Education Department of Jiangxi Province under Grant YC2021-B131.

Contribution Statement

Changjie Chen and Shuying Huang contributed equally to this work. Corresponding author: Yong Yang.