Source-Adaptive Discriminative Kernels based Network for Remote Sensing Pansharpening

1 Introduction

Because of hardware limitations, sensors of satellites cannot capture images of both spectral and high spatial resolution. Only low-resolution multispectral (LR-MS) images and high-resolution panchromatic (PAN) images are captured respectively by satellites like IKONOS, WorldView-2, and WorldView-3. Thus, pansharpening, which aims to fuse a PAN image and an LR-MS image to obtain a high-resolution multispectral image (HR-MS), becomes a fundamental technique in the field of remote sensing image processing. In addition, the task of pansharpening is proved to be popular by the contest in 2006 [Alparone et al., 2007], and the increasing number of review papers published recently.

∗Corresponding author

Most representative methods

The traditional methods of pansharpening can be classified into three categories [Meng et al., 2018], i.e., component substitution (CS) methods, multi-resolution analysis (MRA) approaches, and variational optimization-based (VO) techniques. With the rapid growth of deep learning, methods based on convolutional neural networks (CNNs) [Zhang et al., 2019; Fu et al., 2021; Deng et al., 2021] have been widely applied to the problem of pansharpening. Thanks to CNNs' excellent non-linear mapping and feature extraction capabilities, such methods yield satisfactory results.

In the field of pansharpening, since PAN images contain rich spatial details, while LR-MS images contain abundant spectral information, they are supposed to be processed discriminately. As shown in supplementary, most CNNbased methods simply concatenate PAN and LR-MS images, and throw them directly into meticulously designed networks, without considering the distinctive attributes of different sources. Thus, they may not extract features effectively, which may cause some network parameters to be wasted. Additionally, these methods apply standard convolution to achieve image fusion. In standard convolution, static kernels with the same weights are applied to different areas of given images, which is called content-agnostic [Su et al., 2019]. Previous studies indicate that standard convolution may not be optimal to capture features in some computer vision tasks [Wu et al., 2018]. Therefore, it is necessary to design adaptive kernels which dynamically change with the input.

To tackle the problems mentioned above, we propose a so-called ADKNet for pansharpening, which is composed of source-adaptive discriminative kernels generator (ADKG) modules in series. The spatial kernels and spectral kernels generated are combined through element-wise multiplication to form source-adaptive discriminative kernels, which are then applied to inject detailed information into the LR-MS image as shown in Fig. 1. Our contributions are as follows:

1. A novel ADKG for the task of pansharpening is designed to extract and process information from different sources discriminately and effectively, which guarantees the generalization ability and fewer parameters.
2. The source-adaptive discriminative kernels are generated in a pixel-by-pixel manner to characterize different information in distinct areas, which is proved to be optimal in capturing features for computer vision tasks.
3. Our network yields state-of-the-art (SOTA) outcomes on several datasets of pansharpening. Also, due to the effectiveness of ADKG, our ADKNet only costs about 60,000 network parameters for pansharpening.

2 RelatedWorks

2.1 CNN-basedMethods

With the rapid growth of deep learning, more and more CNNbased methods have emerged in the field of pansharpening, achieving competitive results. The initial work is the pansharpening neural network (PNN) by [Giuseppe et al., 2016], which fuses PAN and LR-MS images through three layers of standard convolution. After that, subsequent works, e.g., PanNet by [Yang et al., 2017], DMDNet by [Fu et al., 2021], and FusionNet by [Deng et al., 2021], further prove the potential of the CNNs by yielding remarkable results. However, most existing works do not fully consider the differences between spatial and spectral features as they simply concatenate PAN images and LR-MS images, and send them directly into the CNNs. Such a process cannot extract information discriminately and effectively, which may cause the waste of network parameters and the loss of generalization ability.

2.2 Adaptive Convolution

In standard convolution, static kernels with the same weights are shared across various areas of different images, leading to sub-optimal feature extraction and the loss of flexibility. Adaptive convolution replaces static kernels with adaptive ones generated from the input. Pioneering work is the dynamic filter networks (DFN) by [Jia et al., 2016], where kernels are generated directly from input contents by a separate network branch. Thus, kernel weights vary as data input to the network. With the development of attention mechanisms in deep learning, this strategy is introduced to generate adaptive kernels [Wu et al., 2018], which allows kernels to be learned from multiple neighboring areas of images. Later works, e.g., pixel-adaptive CNNs (PAC) by [Su et al., 2019], and decoupled dynamic filter networks (DDF) by [Zhou et al., 2021], further prove the superiority of adaptive convolution. It's notable that the above-mentioned adaptive convolutions are never used in the field of multi-source image fusion, e.g., pansharpening, hyperspectral image superresolution (HSISR), etc. Considering the advantages of adaptive convolution, we introduce it to the task of remote sensing pansharpening, by designing a generator that can extract features of distinct areas.

2.3 Motivation

For pansharpening, PAN images contain rich spatial details, while LR-MS images contain abundant spectral information.

However, most existing CNN-based methods overlook the differences between spatial and spectral features that they treat PAN and LR-MS images equally, which may cause weak capability of feature representation and the loss of generalization ability. Thus more network parameters are needed to reach considerable performance. Though adaptive convolution has been proved optimal in other computer vision tasks, regrettably, it has not been fully applied to the field of pansharpening. To alleviate the problems above, we propose ADKNet, which extracts features through specially designed ADKG and injects details into LR-MS images in a pixel-bypixel manner, aiming to obtain better generalization ability while reducing network parameters.

3 The ProposedMethod

3.1 Notations

The LR-MS image obtained directly from a remote sensing satellite is denoted as M ∈ RC×h×w , where C, h, w represents spectral band, height and weight. P ∈ RH×W denotes the PAN image, in which H = 4h, W = 4w because the scaling factor is generally 4 in pansharpening. The up-sampled LR-MS, the desired HR-MS and the ground-truth (GT) image are defined as MU , O, X ∈ RC×H×W , respectively.

3.2 ADKG

To extract and process information discriminately while being lightweight, we carefully design ADKG, shown in Fig. 2, for the task of pansharpening. ADKG is generally composed of two branches, i.e., spatial branch, and spectral branch. The former learns spatial details of various areas from PAN images, forming spatial kernels. And the latter extracts spectral information among different channels of LR-MS images to form spectral kernels.

For the spatial branch, we first apply a 1 × 1 standard convolution layer to alter the number of input channels. Then, we learn in-depth spatial features via 3 × 3 standard convolution layers. After that, we transpose the feature maps to the size of H ×W × k2 (k is the kernel size) and reshape them into spatial kernels in a pixel-by-pixel organized manner. Given the input spatial feature representation F spa ∈ RS×H×W (S

is the channel number of input feature maps), the formation of spatial kernels can be simply represented as:

where Bspa(·) denotes the operation of spatial branch in Fig. 2, and Kspa ∈ RH×W×k×k denotes the spatial kernels. The formed spatial kernels can be seen as a group of convolution kernels with the size of k×k, and each kernel corresponds to a pixel of a distinct position.

As for the spectral branch, global average pooling is first used to aggregate spectral information, while dislodging useless spatial details. Then, fully connected layers are applied to further extract spectral features of higher levels. After that, the feature maps are reshaped to the size of S × k × k and transposed to the spectral kernels. Given the input spectral feature representation F spe ∈ RS×H×W , the formation of spectral kernels can be simply presented as:

where Bspe(·) denotes the operation of spectral branch in Fig. 2, and Kspe ∈ RS×k×k denotes spectral kernels.

To obtain source-adaptive discriminative kernels, we first duplicate the produced spatial kernels and spectral kernels Kspa and Kspe , forming Kspa′ , Kspe′ ∈ RS×H×W×k×k , respectively. Then we operate element-wise product between Kspa′ and Kspe′ to combine them, producing kernels of rich spatial details and abundant spectral information, which can be represented as:Kad = Kspa′ ⊙ Kspe′ , (3)

where Kad ∈ RS×H×W×k×k denotes the desired sourceadaptive discriminative kernels, and ⊙ denotes element-wise product.

Since the generated kernels may contain extremely large or small values, the normalization method in [Zhou et al., 2021] is applied to enhance the stability of training.

3.3 Adaptive Convolution for Pansharpening

Standard Convolution. In standard convolution, given F = F(1) ∈ RS×N (N =H ×W) that denotes the mode-

1 unfolding 1 of input feature maps F ∈ RS×H×W , the ith pixel of the unfolded output feature maps can be written as a combination of input features:

in which F(·,j) ∈ RS denotes the input feature vector at the j th pixel, and F ′ (·,i) ∈ RS′ (S′ is the channel number of output feature maps) represents the ith pixel of the output feature vector. b ∈ RS′ defines the bias vector. Ω(i) is a k × k convolution window around the ith pixel, and W ∈ RS′×S×k×k is a bank of static kernels with the size of k × k. Since pi denotes 2D pixel coordinates, [pi - pj] represents indexing of the spatial dimensions of an array with 2D offsets, which makes W[pi - pj ] ∈ RS′×S the kernels at position offset between the ith and j th pixels. Thus, the same bank of kernels is shared across various areas of different images in standard convolution, leading to sub-optimal feature extraction.

Adaptive Convolution. To alleviate the limitation of standard convolution, we apply adaptive convolution on spectral feature maps with kernels generated by ADKG for pansharpening, which can be written as:

1 The mode-1 unfolding of a tensor X ∈ Rn1×n2×n3 can be defined as a matrix X ∈ Rn1×n2n3 , where the tensor's (i, j, k)-th element maps to the matrix's (i, l)-th element satisfying l = (j -

1)n2 + k.

where F spe (r,j) ∈ R denotes the value at the j th pixel of the rth channel of the unfolded input spectral feature maps, and F spe (r,i) ′ ∈ R denotes the output one. Kad (r,i) ∈ Rk×k defines a single kernel at the ith pixel-wise position of the rth channel of the unfolded Kad . Thus, the kernels applied in adaptive convolution are element-wise generated from the input, considering the spatial details and spectral information.

3.4 ADKNet

To prove the effectiveness of ADKG and adaptive convolution for pansharpening, the ADKNet, as shown in Fig. 3, is designed to be a simple series network architecture with a skip connection.

Firstly, we apply the basic forward propagation section to process P and M, obtaining F spa and F spe . Then, the acquired feature maps are sent into ADKG to form sourceadaptive discriminative kernels in a pixel-by-pixel manner. The generated kernels are utilized to apply adaptive convolution on F spe , delivering spatial and spectral information.

We called the procedure above an adaptive convolution layer. In our network, the layer number is set as 7, which preferably simulates a physical process of keeping injecting detailed information of different levels into maps with insufficient features. After these layers, we restore the shape of output feature maps and add them with MU derived through operating deconvolution on M, obtaining O which contains rich spatial details and abundant spectral information.

3.5 Loss Function

To further verify the effectiveness of our network, we choose the simple mean square error (MSE) as the loss function:

where M{m} , P {m} and X {m} denote the mth LR-MS and PAN training pair, and GT image, respectively. fΘ(·) represents our network and Θ defines the involved model parameters. M is the number of training examples, and ‖ · ‖2 indicates the ℓ2 norm.

4 Experiments

In this section, we measure the performance of the proposed method by comparing it with some recent state-ofthe-art (SOTA) pansharpening approaches belonging to the CS-based, MRA-based, and CNN-based methods through a series of experiments on various datasets acquired by WorldView-3 (WV3) and WorldView-2 (WV2) satellites.

4.1 Experiment Settings

Datasets. In this work, we mainly conduct our experiments on WV3 with a spatial resolution of about 0.3 m for the PAN and 1.2 m for the LR-MS images. The spatial resolution ratio is equal to 4 and the radiometric resolution is 11 bits. The MS bands are composed of four standard colors (RGB and nearinfrared 1) and four new bands (coastal, yellow, red edge, and near-infrared 2). The dataset is downloaded from the public website 2 , which contains 12580 samples. We process the dataset to PAN/LR-MS/GT image pairs (70%/20%/10% as training/validation/testing dataset) with the size of 64 × 64, 64 × 64 × 8 and 16 × 16 × 8 following Wald's protocol by [Wald et al., 1997], same as FusionNet by [Deng et al., 2021].

Benchmarks. We compare our method with several stateof-the-art (SOTA) approaches consist of four CS-based methods: GS by [Laben and Brower, 2000], BDSD by [Garzelli et al., 2008], BDSD-PC by [Vivone, 2019] and PRACS by [Choi et al., 2010]; three MRA-based methods: SFIM by [Liu, 2000], GLP-HPM by [Vivone et al., 2014], and GLPCBD by [Alparone et al., 2007]; and five CNN-based methods: PNN by [Giuseppe et al., 2016], PanNet by [Yang et al., 2017], DiCNN by [He et al., 2019], DMDNet by [Fu et al., 2021] and FusionNet by [Deng et al., 2021]. For a fair comparison, all CNN-based approaches are trained on the same Nvidia GPU-2080Ti and Pytorch environments.

Evaluation Metrics. According to the pansharpening research standard, we choose five quality indexes for the reduced resolution, including SAM, ERGAS [Wald, 2002], SCC [Zhou et al., 1998], QAVE [Wang and Bovik, 2002] and Q8 [Garzelli and Nencini, 2009]. And we apply QNR, Dλ and Ds indexes [Vivone et al., 2015] for the full resolution.

Parameters Tuning. In our ADKNet, we set the initial learning rate, epoch, and batch size as 0.003, 1000, and 32, respectively. Thus, the number of iterations is 2.5 × 105 . In addition, the learning rate is reduced by half every 100 epochs and Adam is used as the optimizer. In particular, for the setting of other compared CNN-based methods, we apply the default asset in related papers and codes.

4.2 Reduced Resolution Assessment

We train our network and other CNN-based methods on the training dataset acquired by the WV3 satellite. Then, we evaluate the most representative methods on 1258 reducedresolution images. The quantitative assessment results are presented in Tab. 1 which shows our ADKNet obtains the best average quantitative performance for all the quality indexes with the fewest network parameters.

2 https://resources.maxar.com/

We further implement the test on a new dataset acquired by WV3 named Rio with the size of 256 × 256 for a PAN image. The quantitative assessment results are presented in Tab. 2, which proves the priority of our network. Besides, we present the qualitative assessment results in Fig. 4 and Fig. 5. The darker the absolute error map, the better, indicating our ADKNet outperforms other methods.

4.3 Full Resolution Assessment

To demonstrate the application value of our proposed method, we perform experiments on 30 full-resolution samples on WV3 with the size of 256×256 for PAN images. The quantitative assessment results are shown in Tab. 3. Obviously, the ADKNet performs the best on all three indexes, which firmly indicates the superiority of our method.

4.4 Generalization

One significant problem of CNN-based methods for pansharpening is the generalization ability. Once the testing dataset varies a lot, some CNN-based approaches may not perform well. Owing to the distinctive features extraction process, our ADKNet possesses a stronger generalization ability over other CNN-based methods. We verify this by applying models trained on the dataset of WV3 to another dataset named Stockholm acquired by WV2, with the size of 256 × 256 for PAN image. The quantitative assessment results are shown in Tab. 4, from which we can observe that the ADKNet far exceeds the performance of other methods on all five indexes, which strongly proves the superior generalization ability of our network. More relevant visual results can be found in the supplementary material.

4.5 Ablation Study

To prove the effectiveness of our ADKNet, we design two networks to carry out the ablation study. The network architectures are presented in the supplementary material.

For the first network (called ConvNet), we concatenate PAN and LR-MS images and throw them into seven standard convolution layers with the kernel size of 3 × 3. As for the second network (called AKNet), the concatenation of the PAN and LR-MS images is thrown into both spatial and spectral branches of ADKG, ignoring the distinction between spatial and spectral features.

ConvNet is designed to verify the superior of adaptive convolution for pansharpening, while AKNet is designed to prove the correctness of processing different sources discriminately. We train ConvNet and AKNet on WV3 and test on WV2. The results are shown in Tab. 5. It is obvious that the ADKNet achieves far better performance than ConvNet and AKNet, which proves that our strategy can greatly improve the performance for pansharpening.

5 Conclusion

In this work, we proposed a novel scheme named ADKNet, which consists of ADKG modules in series. Through ADKG block, spatial details from PAN images and spectral information from LR-MS images can be extracted effectively, thereby forming source-adaptive discriminative kernels which can inject detailed information into LR-MS images in a pixel-bypixel manner. ADKNet yields state-of-the-art (SOTA) outcomes on various datasets of remote sensing pansharpening with the fewest network parameters, proving the strong feature learning ability of the proposed method. In addition, the excellent generalization capability of ADKNet indicates that it is more reliable and robust than other advanced methods.

Acknowledgments

This research is supported by NSFC (12171072, 61876203, 61702083), Key Projects of Applied Basic Research in Sichuan Province (Grant No. 2020YJ0216), and National Key Research and Development Program of China (Grant No. 2020YFA0714001).