Cross-Scale Domain Adaptation with Comprehensive Information for Pansharpening

1 Introduction

Low-resolution multi-spectral (MS) images and panchromatic (PAN) images are two distinct modalities of remote sensing images captured by satellites [Zhang et al., 2021; Gong et al., 2023]. The former possess low spatial resolution but high spectral resolution, while the latter exhibit the opposite characteristics. Since single MS images or PAN images limit their applications in downstream tasks such as land cover mapping [Garzelli et al., 2018] and environment monitoring [Zhang et al., 2014], high-resolution multi-spectral (HRMS) images are in high demand. Pansharpening aims to merge HRMS images from MS images and PAN images and has become a hot topic due to low technical cost [Ghassemian, 2016].

∗Corresponding author

Existing pansharpening methods can be classified into two main types: traditional methods and deep learning-based methods. Early research primarily focused on traditional methods. These methods can be further divided into detail injection (DI)-based methods and variation optimization (VO)-based methods. Generally, DI-based methods inject the spatial details from the PAN image to the MS image through image decomposition and component substitution [Tu et al., 2001; Aiazzi et al., 2002]. VO-based methods treat the pansharpening process as an optimization problem and leverage an objective function to find the suitable solution in the optimization scheme [Fu et al., 2019; Li et al., 2013]. In stark contrast to traditional methods, deep learning-based approaches exploit the powerful feature extraction of neural networks to extract features that are friendly for the pansharpening task. They have achieved significantly improved performance compared to traditional algorithms. According to the training ways, deep learning-based methods can be further classified into two categories: supervised methods and unsupervised methods. Most deep learningbased methods belong to the supervised category. For example, the first deep learning-based pansharpening method is PNN [Masi et al., 2016], which is inspired by SRCNN [Gao et al., 2017]. PNN constructs a simple three-layer convolutional neural network framework, achieving impressive performance and high efficiency. Recently, Gong et al. [Gong et al., 2022] introduced a method that utilizes the advantages of ConvLSTM in transmitting long-term information to implement effective information transfer across multiple layers and scales, achieving promising results in both spatial and spectral information. All these supervised methods follow the Wald's protocol [Wald et al., 1997] to generate simulated data for training, ignoring the domain gap between different scales. Compared to supervised methods, unsupervised methods do not rely on simulated data while constructing observation models to constrain the network. However, in many cases, observation models are unknown, and their inverse operation has countless solutions. Both of them result in inaccurate results obtained by unsupervised methods. One typical example of unsupervised pansharpening methods is PanGAN [Ma et al., 2020], which employs dual discriminators to continually reduce the distribution distance between the HRMS results and the two source images. To fit the distribution of PAN images, PanGAN exploits a linear weighted strategy. However, the real spectral observation model is not simply based on linear weighted [Jiang et al., 2015].

Although existing pansharpening methods have achieved promising performance, there are still critical issues that require urgent attention. Due to the lack of the ground truth, researchers generally downsample the FMS images (i.e., MS images at the full-resolution scale) and FPAN images (i.e., PAN images at the full-resolution scale) to obtain images at the reduced-resolution scale. The downsampled images are used as training data and FMS images are treated as the ground truth. This strategy has two main limitations. On the one hand, FPAN images are disregarded during training, while they contain the richest spatial detail information. On the other hand, existing methods directly generalize models trained at the reduced-resolution scale to the full-resolution scale, while the domain gap between the two scales can significantly affect their performance at the full-resolution scale. Therefore, addressing these issues is crucial for further advancing the performance of pansharpening methods at the full-resolution scale.

To tackle the above challenges, in this work we propose the cross-Scale Domain adaptation with comprehensive Information for PanSharpening (SDIPS), which addresses the problem from two aspects: cross-scale information utilization and model generalization. Firstly, regarding the information utilization of both reduced-resolution scale and full-resolution scale, we introduce a cross-scale comprehensive information capture (CSCIC) module. Unlike traditional feature extractors limited to be effective at the reduced-resolution scale, the feature extractor of the CSCIC module combines the pansharpening process at the reducedresolution scale and the reconstruction process at both two scales. Thus, we obtain a robust feature extractor with crossscale comprehensive information capture, where the "robustness" attributes to the ability to simultaneously utilize information from both scales. Meanwhile, it address the issue of under-utilizing FPAN image information during training. Secondly, to address the model generalization issue at the full-resolution scale, we adopt the cross-scale domain adaptation (CSDA) strategy from the reduced-resolution scale to the full-resolution scale. Specifically, we employ two loss functions to bridge the domain gap between the two scales: the maximum mean discrepancy (MMD) [Gretton et al., 2012] loss and the loss related to pixel-level correlations between the two scales, which captures the inherent prior that the reduced-resolution images are obtained through downsampling the full-resolution images. The MMD loss is treated as a regularization term based on the pixel-level prior. It enables the model to adapt to different scales, thus improving its performance at the full-resolution scale. It is worth noting that, it is the first time to address the generalization issue of pansharpening models across different scales from the perspective of domain adaptation.

The contributions of our work can be summarized as the following two aspects:

• We design a cross-scale comprehensive information capture module to incorporate the FPAN images into the network training process for pansharpening, which improves the utilization of accessible information.
• We adopt a novel domain adaptation perspective to address the model generalization issue across different scales. By leveraging the MMD loss and the inherent pixel-level relationships between the two scales, we reduce the domain gap and improve the performance at the full-resolution scale.

2 Method

As mentioned above, existing methods commonly generalize the model trained at the reduced-resolution scale to the full-resolution scale. The neglected information of the FPAN image and the inherent scale domain lead to undesirable performance at the full-resolution scale. To meet the above challenges, we propose the SDIPS method, which introduces a CSCIC module to incorporate information from both scales, and adopts a CSDA strategy to facilitate the model generalization. In what follows, details about the above modules will be described.

2.1 Cross-Scale Comprehensive Information Capture

As we know, Wald's protocol is a common approach in pansharpening to train the networks in a supervised manner. Specifically, FMS images (IFM) and FPAN images (IFP ) are first downsampled to obtain MS images (IMS) and PAN images (IPAN). Consequently, pairs of IMS and IPAN are treated as training data, and the corresponding IFM is considered as the ground truth to realize supervised learning. Apparently, the information of IFP is partly discarded. Some works attempt to compensate for this deficiency by incorporating the pansharpening process at the full-resolution scale during training like PS(PS(IMS, IPAN), IFP ), where PS(·) represents the pansharpening model. However, the lack of ground truth at the original full-resolution scale limits the exploitation of the FPAN information.

To cope with this challenge, we introduce a CSCIC module to enhance the utilization of information from different scales. As shown in Figure 1, in the pansharpening process at the reduced-resolution scale, the MS encoder block (EMB) and PAN encoder block (EPB) are employed to extract features from IMS and IPAN, respectively, written as:

Subsequently, the extracted features fMS and fPAN are concatenated to generate HRMS results IH, and this process is formalized as:

where DPSB(·) represents the decoder block of the pansharpening network.

To address the limitation of existing methods that do not utilize FPAN information, we design a reconstruction network at the full-resolution scale to better utilize comprehensive information at this scale. This process is defined as:

where DMB and DPB are decoder blocks for MS and PAN of the reconstruction network, respectively. Further, to ensure that the features extracted by the reconstruction network and that of the pansharpening network are mutually beneficial, we also add the reconstruction process of the reduced-resolution images, formalized as:

By sharing parameters between the encoder blocks of the reconstruction network and the pansharpening network, the encoder ensures the perceptual capability of effective features from images on both scales.

Regarding network architectures, the Unet-like structure is employed for better learning of details. Additionally, considering small objects in remote sensing images, we incorporate dilated convolution structure into the network to enlarge the receptive field without altering the resolution. The decoder block of the pansharpening network (i.e., DPSB) and that of the reconstruction network (i.e., DMB and DPB) enjoy the same network architecture (nuances only in input channels) but different parameters due to discrepant tasks.

Obviously, both the pansharpening task and the reconstruction task are fully-supervised. The loss function of the pansharpening network can be formalized as:

where SSIM(·) means the structure similarity.

Similarly, the loss functions of the reconstruction network are defined as:

where Lfs and Lrs represent the reconstruction loss of the full-resolution scale and the reduced-resolution scale, respectively. Since the pansharpening network and the reconstruction network are jointly trained, the loss of this joint training process can be expressed concisely as:

where λ1 and λ2 are hyper-parameters to control the balance between the pansharpening task on the reduced-resolution scale and the reconstruction task on two scales.

2.2 Cross-Scale Domain Adaptation

In the previous section, we introduce robust encoder blocks that have the perceptual capability to capture effective information at both two scales. However, the domain gap between different scales has not been addressed yet. As is well known, existing deep learning-based methods often perform worse at the full-resolution scale compared to the reducedresolution scale. This performance degradation is attributed to the domain differences between the reduced-resolution scale and the full-resolution scale. As such, directly generalizing the model trained at the reduced-resolution scale to the full-resolution scale leads to unsatisfactory results.

To address this issue, we pioneer an approach by considering model degradation from the perspective of domain adaptation. If the features extracted from input images at different scales are domain-invariant, the DPSB trained on one scale can be effectively generalized to the other scale. We refer to this process as "cross-scale domain adaptation".

Specifically, as shown in Figure 2, we employ maximum mean discrepancy (MMD) and the pixel-level correlation prior to constrain distributions Prs(f) and Pfs(F) of two features, where f and F denote the features from the reduced scale and the full scale, respectively, and Prs(·) and Pfs(·) represent the corresponding distributions.

Firstly, we introduce the details of computing the MMD loss. The MMD loss is commonly used in the classification area to narrow the domain discrepancy between the real image domain and the synthetic image domain, thereby achieving model generalization across different domains [Van Opbroek et al., 2018; Zhang et al., 2023; Zhu et al., 2021]. Essentially, the MMD loss is an approach based on kernel techniques. It measures the distribution differences by computing the average distribution discrepancy between samples from two domains in the Hilbert feature space. Specifically, it can be mathematically formalized as:

where Nf and NF are the number of samples from the reduced-resolution scale domain and the full-resolution scale domain. ‖ · ‖H means the norm of the Reproducing Kernel Hilbert Space (RKHS), and φ represents the mapping from the original feature space to the RKHS.

In existing implementations of the MMD loss, the inputs typically consist of images of the same size [Bonev et al., 2008; Zhu et al., 2019]. However, in this study, we use the MMD loss to measure the similarity of features at different scales with a size ratio of 4. Specifically, to maintain the characteristics of the specific scale domain, we employ a 3 × 3 window to simultaneously slide over the features of both scales. The features obtained from the sliding window are then averaged to generate samples for computing the MMD loss. The process of generating samples is shown in Figure 3.

Secondly, we describe the prior pixel-level correlation between features from two scales. In the pansharpening task, features from the full-resolution scale can be downsampled to establish a certain pixel-level correspondence with that from the reduced-resolution scale. As such, we additionally introduce a downsampling block (DSB) to utilize this prior relationship, and the MMD loss serves as a regularization term for the domain adaptation purpose. The overall formulation of the CSDA part is defined as:

Algorithm 1: The training process of SDIPS

where DSB(·) is the downsampling block, the details of which are shown in Figure 2, and η is the regularization parameters to balance the two terms.

In general, the loss function of the whole part can be written as follows:

where σ is the hyper-parameter to ensure the balance.

3 Experiments

3.1 Experimental Settings

Datasets. We conduct experiments on three satellites: QuickBird (QB), GaoFen-2 (GF2) and WorldView-II (WV2). QB and GF2 capture 4-band MS and 1-band PAN images, while WV2 captures 8-band MS and 1-band PAN images. To expand the training dataset, we crop the FMS images to sizes of 80 × 80 × 4/80 × 80 × 8, and correspondingly crop the FPAN images to a size of 320×320. Subsequently, all images are downsampled by 4 times following the Wald's protocol to obtain images at the reduced-resolution scale, with sizes of 20 × 20 × 4/20 × 20 × 8 for MS images and 80 × 80 for PAN images. We obtain a total of 10000 pairs of images for the training set by applying rotations, adding noise, and other means. During testing, for better assessment, MS images at the reduced-resolution scale are of size 66×66×4/66×66×8, and PAN images are of size 264×264. At the full-resolution scale, FMS images are of size 264×264×4/264×264×8, and FPAN images are of size 1056×1056. The testing dataset consists of 200 pairs of images in total.

Training Details. The overall training process is reported in Algorithm 1. We first train the pansharpening network with the learning rate 5e-4 for 40 epochs. The network is optimized by minimizing Lps using the Adam optimizer with a decay rate of 0.95 every five epochs. After that, the pansharpening network, the reconstruction network and DSB are jointly optimized by minimizing Lall using the Adam optimizer with a decay rate of 0.95 every epoch, while the learning rate is set as 1e-4 and this part continues for 10 epochs. As the pansharpening network and the reconstruction network are equally important to train the EN block, λ1 and λ2 are set as 1. η and σ are set as 0.5 and 0.1 to minimize the potential performance degradation of the pansharpening network. α in the PRELU activation function is set as 0.25. Experiments are implemented on the PyTorch platform using a 3.5-GHz Intel Core i9-9920X CPU and NVIDIA Titan RTX GPU.

Comparison Methods. We select nine comparison methods, including four traditional ones (CAIS [Restaino et al., 2017], GLP-Reg FS [Vivone et al., 2018], BDSDPC [Vivone, 2019] and CDIF [Xiao et al., 2022]) and five deep learning-based ones (PanGAN [Ma et al., 2020], D2TNet [Gong et al., 2022], LPPN [Jin et al., 2022], STPSOM [Zhang and Ma, 2023] and MMNet [Zhou et al., 2023]). To ensure fairness, all deep learning-based methods are trained using the publicly available code with the same training dataset.

Evaluation Metrics. Since the ground truth exists at the reduced-resolution scale, we use two comprehensive metrics, relative dimensionless global error in synthesis (ERGAS) and structural similarity index measure (SSIM), for quantitative evaluation. At the full-resolution scale, where the ground truth is absent, we employ commonly used quality with no reference (QNR) [Vivone et al., 2014], spectral distortion index (Dλ) and spatial distortion index (Ds) for assessment.

3.2 Experimental Results

QB Dataset. We present quantitative results on the QB dataset in Table 1. At the reduced-resolution scale, our method achieves significantly superior performance on the metrics ERGAS and SSIM, indicating that the proposed SDIPS can produce results closer to the ground truth. At the full-resolution scale, our method presents outstanding performance among deep learning-based methods, although slightly inferior to CDIF, which employs some priors friendly to the metrics. It proves the generalization ability of our method across different scales. In contrast, LPPN performs better than D2TNet at the reduced-resolution scale but is inferior to D2TNet at the other scale.

Further, we select the top 5 methods based on metrics at the reduced-resolution scale and the top 6 methods based on metrics at the full-resolution scale, providing qualitative experimental results shown in Figure 4. At the reduced-resolution scale, the enlarged parts in the red box demonstrate that the proposed method exhibits higher consistency in both spectral and spatial aspects with the ground truth. The residual maps further validate this observation. At the full-resolution scale, the enlarged parts demonstrate that our method possesses clearer contours and more normal spectral information. For example, the gap between two lines within the red box is significantly more distinct in our method. The details of the buildings within the green box appear enhanced details and real spectral information, while some methods show blurred details like D2TNet, LPPN, MMNet, and other methods appear spectral distortion like CDIF, PanGAN and STP-SOM.

GF2 Dataset. Quantitative results on the GF2 dataset are also presented in Table 1. At the reduced-resolution scale, our method achieves consistent results with QB, indicating the superior results of the proposed SDIPS. At the full-resolution scale, our method presents significant outstanding performance among all comparable methods, showing performance gains in terms of both spectral distortion and spatial retaining.

Also, qualitative experimental results of the top few methods are provided in Figure 5. At the reduced-resolution scale, the enlarged parts and the residual maps demonstrate that the proposed method exhibits higher consistency in both spectral and spatial aspects with the ground truth. At the full-resolution scale, the enlarged parts demonstrate that our method possesses clearer details and higher contrast. More specifically, in the green box, the proposed SDIPS shows the clearest details, while other methods show blurred details and obvious artifacts. In the red box, the proposed SDIPS shows the most normal spectral information.

WV2 Dataset. Quantitative results on the WV2 dataset are presented in Table 2. Our method exhibits significant advantages both at the reduced-resolution scale and the fullresolution scale. It indicates that despite the simplicity of our network architecture, the strong generalization ability across different scales enables consistent results.

From a visual perspective, Figure 6 presents qualitative results on WV2 dataset. Results at the reduced resolution scale demonstrate the higher consistency of our method with the ground truth. Meanwhile, results at the full resolution scale show that the proposed SDIPS possesses clearer details. In contrast, other methods exhibit apparent artifacts like CDIF, D2TNet, LPPN, and MMNet, or spectral distortion like BDSD-PC and PanGAN.

3.3 Ablation Studies

In this section, we conduct ablation experiments to study the impact of the specific designs including the CSCIC module and the CSDA strategy on the final results. All ablation experiments are conducted under the same training settings and environment. The same metrics as above are employed to evaluate the performance at two scales. Table 3 reports the corresponding quantitative results. In the table, the column PS, +RS, +RS+MSE and +RS+DA represent results with the CSCIC module, results with the CSCIC module and the downsampling block, results with the CSCIC module and the CSDA strategy, and our proposed results, respectively.

Effectiveness of The CSCICModule. As shown in the table, it is evident that by incorporating the CSCICmodule (the column +RS), the model significantly improves the performance at the full-resolution scale while only incurring a small loss at the reduced-resolution scale. Figure 7 shows qualitative results on the QB dataset and the WV2 dataset. Taking the results on the QB dataset as an example, at the reducedresolution scale, the final result is similar to that of the ablation results. However, at the full-resolution scale, the result of the original PS displays abnormal artifacts, while the result of the +RS shows normal spectral information. The advantages can be attributed to the powerful ability of information capture at both two scales.

Effectiveness of The CSDA Strategy. Similarly to the above situation, the addition of the CSDA strategy also achieves certain performance gains at the full-resolution scale as expected, while generating minimal performance loss at the reduced-resolution scale. From Figure 7, further incorporating domain adaptation strategies can not only remove artifacts but also enhance the clarity of the resulting details. The advantages can be attributed to the ability to reduce the domain gap between the two scales.

4 Conclusion

In this paper, we propose an effective pansharpening method SDIPS based on comprehensive information utilization with strong model generalization. On the one hand, through employing a cross-scale comprehensive information capture module, we obtain robust extractors to extract effective features at both scales and achieve comprehensive exploitation of available information. On the other hand, we introduce the cross-scale domain adaptation strategy, employing the inherent pixel-wise relationships between features at two scales as a prior and the MMD loss as a regulation term to reduce the scale domain gap. This strategy improves the ability of model generalization. Overall, our method effectively addresses the shortcomings of under-utilization of the original PAN information and low ability of model generalization across different scales. The experiments conducted on various datasets demonstrate the superiority of our method and validate the effectiveness of the proposed designs.

Ethical Statement

There are no ethical issues.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (62276192).