PanComplex: Leveraging Complex-Valued Neural Networks for Enhanced Pansharpening

1 Introduction

Multispectral (MS) images contain richer spectral information and have been widely applied in various fields, such as environmental monitoring, agriculture, and mapping services. However, the physical limitations of satellites hinder the direct acquisition of high-resolution multispectral (HRMS) images through sensors. As an alternative, only high-resolution panchromatic (PAN) images and corresponding low-resolution multispectral (LRMS) images can be obtained. Pansharpening involves merging LRMS and PAN images to generate high-resolution multispectral images. In recent years, with the rapid development of deep learning, neural network-based methods have made significant progress in pansharpening. Many methods achieve promising results by designing adaptive network architectures to capture spatial and spectral correlations between panchromatic and multispectral images [Zhou et al., 2022d; Zhou et al., 2022a; Yang et al., 2017; Masi et al., 2016]. However, most existing works are limited to operations in the real-valued domain. While some studies have explored the relationship between pansharpening and the frequency domain [Zhou et al., 2022c; Tan et al., 2024], exploration of the complex domain remains limited, though it naturally aligns with frequency analysis.

∗Corresponding author.

Complex-valued neural networks (CNNs) have gained widespread attention in recent years due to their performance in signal processing and computer vision tasks [Trabelsi et al., 2017; Quan et al., 2021a; Nguyen et al., 2022]. Compared to traditional real-valued neural networks, complexvalued networks can effectively leverage the structural information inherent in the complex domain, providing richer representation capabilities that are applicable in both the frequency and spatial domains. In the frequency domain, Fourier transforms are widely used in pansharpening [Zhou et al., 2022b; He et al., 2023], converting images into the frequency domain and representing them in complex form. Previous methods, limited by real-valued networks, cannot directly process the Fourier spectra and instead decompose them into real and imaginary parts for separate processing. In contrast, complex operators, such as complex convolutions, can operate directly on the Fourier spectra, yielding more continuous and richer frequency representations, as shown in Figure 1. In the spatial domain, complex representations allow PAN and MS images to interact as a whole, facilitating more enriched fusion representations. As shown in Figure 1, complex-based spatial fusion preserves more texture details. Furthermore, Complex-valued neural networks offer compact representations and strong generalization capabilities [Quan et al., 2021b], rendering them well-suited for the lightweight design of pansharpening methods and for enhancing their generalization ability. Based on this, we aim to reconstruct the pansharpening task using complex-valued neural networks to explore the potential of this powerful representation method in pansharpening.

In this work, we establish complex representations for the pansharpening task and introduce basic complex operators. Based on this, we design a spatial-frequency dual-domain pansharpening framework, PanComplex. Given that the frequency domain provides powerful tools for analyzing pansharpening degradation and that the Fourier spectra in the frequency domain align naturally with the complex domain, PanComplex models complex signals in both the spatial and frequency branches to extract complementary features from PAN and MS. The frequency-domain branch directly operates on the Fourier spectra of PAN and MS images in the complex space, yielding more continuous and richer representations. The spatial-domain branch, based on the complex representation of MS and PAN, uses complex convolutions to further facilitate their fusion. In addition, we introduce a complex-based complementary learning mechanism to promote spatial-frequency interaction between PAN and MS. Through these methods, our approach leverages the representational power of the complex domain to enhance the interaction between PAN and MS, improving the complementary representation of both spatial and frequency information, thus significantly improving pansharpening performance. Our contributions are as follows:

• We propose the PanComplex framework based on complexvalued neural networks, the first work to explore complexvalued pansharpening.
• We design a complex-based spatial-frequency dual-domain structure that efficiently integrates spatial and spectral information from PAN and MS images.
• We validate the superiority of the proposed method on multiple datasets, demonstrating strong generalization ability and low computational cost.

Our experimental results show that PanComplex outperforms existing methods on several benchmark datasets, achieving optimal performance with the fewest parameters, and generalizes well to other fusion tasks, such as infraredRGB fusion.

2 RelatedWork

2.1 Pansharpening

The pansharpening task aims to fuse low-resolution multispectral images with panchromatic images to obtain highresolution multispectral images. Traditional pansharpening methods are generally classified into three categories: Component Substitution (CS), Multi-resolution Analysis (MRA), and Variational Optimization (VO). CS methods are simple and feasible but prone to spectral distortion. MRA methods are generally simple and effective, providing good spectral information fusion; however, they lack modeling of the beneficial prior knowledge embedded in the data's spatial dimensions, often leading to spatial distortion. VO methods heavily rely on manually designed prior structural knowledge and often fail to sufficiently model the complex structural priors inherent in the data.

With the success of deep learning methods in various computer vision tasks, deep learning has also been applied to pansharpening, leading to significant improvements. PNN [Masi et al., 2016] first uses a three-layer convolutional neural network to learn the relationship between panchromatic images, low-resolution multispectral images, and highresolution multispectral images. Subsequently, PANNet [Yang et al., 2017] adopts the residual learning mechanism from ResNet. MSDCNN [Yuan et al., 2018] adds a multiscale module on top of the residual connections. GPPNN [Xu et al., 2021] improves interpretability using a deep unfolding approach. INNformer [Zhou et al., 2022a] introduces the Transformer architecture to the field, effectively modeling long-range dependencies and feature fusion. FAME-Net [He et al., 2024] combines MOE and frequency domain information, enabling the network to dynamically learn highfrequency information in remote sensing images. HFIN [Tan et al., 2024] combines local Fourier and global Fourier information for image fusion.

2.2 Complex-Valued Learning

Complex-valued neural networks are a type of neural network that processes information using complex-valued parameters and variables [Hirose, 1994]. Biological researchers have discovered that, in addition to firing rates, the relative timing of neuronal spikes may also carry information about sensory inputs and dynamic networks [Geman, 2006]. The amplitude and phase of the complex-valued units in complexvalued neural networks, which are similar to biological neurons, have sparked interest among researchers. There have been efforts to extend real-valued network architectures to their complex-valued counterparts. For example, Trabelsi et al. [Trabelsi et al., 2017] designed a complex-valued convolutional neural network, Sun et al. developed a complex-valued generative adversarial network [Sun et al., 2019]. Based on these network structures, researchers have investigated the applications of complex-valued neural networks across various tasks. Alan et al. [Oppenheim and Lim, 1981] demonstrated the importance of phase information in speech signals and images. Nguyen et al. [Nguyen et al., 2022] applied complex-valued convolutional neural networks to iris recognition. Yadav et al. [Yadav and Jerripothula, 2023] designed a novel projection method from real-valued to complex-valued space for RGB images. Quan et al. [Quan et al., 2021a] achieved state-of-the-art performance in image denoising using complex-valued convolutional neural networks. In recent years, remote sensing science and technology have seen significant advances [Tian et al., 2024; Deng et al., 2024]. Concurrently, complex-valued neural networks have also been applied to the research of Polarimetric Synthetic Aperture Radar images [Zhang et al., 2017] and Interferometric Synthetic Aperture Radar images [Sunaga et al., 2019]. In image classification, complex-valued convolutional neural networks have shown performance comparable to, or even exceeding, real-valued convolutional neural networks [Yadav and Jerripothula, 2023; Zhang et al., 2017]. In the theoretical domain, studies have also confirmed the viability and potential benefits of complex-valued neural networks [Wu et al., 2023; Zhang et al., 2022]. Our research leverages complex-valued convolutional neural networks for the pansharpening task of remote sensing hyperspectral images.

3 Methodology

3.1 Motivation

In the pansharpening task, the benefit of spatial-frequency interaction has been demonstrated [Zhou et al., 2022c]. Specifically, convolutions are used in the spatial domain to extract local information from both PAN and MS, and in the frequency domain, the Fourier spectrum is decomposed into real and imaginary parts (or amplitude and phase spectra), which are processed separately to extract global frequency information. After performing a 2D Fourier transform, the signal is complex-valued, and existing methods are limited by realvalued neural networks, which can only process the real and imaginary parts separately. While this approach has achieved some success, the internal correlations within the Fourier spectrum may be disrupted, potentially leading to a loss in the generalization ability of the features. In contrast, complex networks, such as those using complex convolution, can directly process signals in the frequency domain, resulting in a more continuous and richer representation space [Nguyen et al., 2022]. As shown in Figure 1, by using complex convolution to process the Fourier spectrum of the image, we obtain better feature interaction effects compared to processing the real and imaginary parts (or amplitude and phase spectra) separately with two convolutions.

Recent studies on pansharping have focused on extracting local features in the spatial domain through convolution blocks. The challenge lies in extracting richer features from the two-modal images. In our work, we replace the convolution blocks in a relatively simple GPPNN model that interact with features from different modalities with complex convolutions, and study the feature maps obtained from both approaches. As shown in Figure 1, we find that the feature maps extracted using complex convolution capture more diverse features from the different modalities. At the same time, we conducted a quantitative analysis of the experiment and present the results in the supplementary materials. Furthermore, complex convolution offers significant advantages in weight compactness. As shown in Figure 2, The response of complex convolution includes two feature maps, which are treated as a single channel. Therefore, using complex convolution networks can halve the model parameters, which is crucial for lightweight panchromatic sharpening. Additionally, biological neurons, apart from their firing rate, encode important information in the relative timing of neural pulses [Geman, 2006]. The complex values in complex neural networks better simulate the output of biological neurons, offering interpretability in the context of biological models [Reichert and Serre, 2013]. Complex neural networks also exhibit strong generalization abilities [Hirose and Yoshida, 2012], which is why our network performs exceptionally well in generalizing to two other image fusion tasks, achieving optimal results in each case.

3.2 Preliminary

Complex-valued convolution. For complex-valued images, each pixel value is represented as a complex number. A complex number z , defined as follows: z = a + ib ∈ C, a ∈ R, b ∈ R, where√-1 = i ,a is a real component and b is an imaginary component. The Complex number can also represented by a magnitude ,r ∈ R, and a phase, θ ∈ R, as z = re iθ . In real-valued convolution, the convolution operation between a feature map Z and a kernel K is expressed as Z ∗ K, where ∗ denotes the convolution operation. For complex-valued feature maps Z = A+ iB and complex-valued kernels K = X + iY , the convolution operation is denoted as

which is expanded as shown in Figure 2. The complex convolution operation is represented as C - Conv (.)

Fourier transformation of Images. The Discrete Fourier Transform (DFT) has long been used in image processing for analyzing the frequency content of images. The DFT can be represented in the following form: x ∈ R H×W×C is the given image, and F (X ) is the resulting complex-valued component, which is expressed as:

The DFT is performed separately on each channel of the image. The magnitude and phase can be expressed through the following formulas:

where R (x)and I (x) represent the real and imaginary parts of F (x) , respectively.The DFT operation is represented as F (.), and the IDFT operation is represented as F-1 (.).

3.3 Overview

We propose a novel pansharpening network PanComplex based on complex convolution, as shown in Figure 2. Given the panchromatic image (PAN) and the low-resolution multispectral image (LRMS), the LRMS is first upsampled using bicubic interpolation. Then, convolutional layers are applied to project both the panchromatic image and the upsampled multispectral image into feature maps of the same dimensionality. The PAN image is then passed through a series of cascaded convolutions to extract a sequence of informative features. Next, the obtained multimodal perceptual feature maps of both the multispectral and panchromatic images are jointly processed through N cascaded key modules, denoted as Complex Spatial and Frequency Interaction Block (CSFIB), for feature extraction and information integration. Finally, the complex feature maps obtained from the N modules are converted back to the image space through a 1×1 convolution, and then the residual of the upsampled LRMS is added to obtain the output image.In our training process, we employ the L1 loss to optimize the network.

3.4 Complex Feature Extractor

As shown in Figure 3, the complex information extraction module consists of both a Complex Frequency Feature Extractor (CFFE) and a Complex Spatial Feature Extractor

(CSFE). The two branches extract effective information representations from the image's frequency domain and spatial domain through complex convolution, enabling dual-domain information fusion.

Complex Frequency Feature Extractor. As shown in Figure 3, we first apply the discrete Fourier transform (DFT) to convert the panchromatic and multispectral images into the complex domain. Let the input features of the multispectral and panchromatic images be Ip and Ims, respectively, with the corresponding Fourier transforms denoted as

where R(.) and I (.) represent the real and imaginary parts. Then, we pass these through a two-layer complex-valued 1x1 convolutional neural network with complex ReLU activations. The resulting complex feature maps from the two images are concatenated and reduced in dimensionality through a complex-valued 1x1 convolution. Finally, we apply the inverse discrete Fourier transform (IDFT) to convert the obtained complex feature map back to the spatial domain to produce the frequency-domain feature map Ff . The entire process is formulated as follows:

Complex Spatial Feature Extractor. As shown in Figure 3, we combine the Ip and Ims into a complex-valued representation, which results in better performance compared to working in the real domain. Based on this, we design a spatialdomain information interaction branch, Complex Spatial Feature Extractor (CSFE), which conducts deep feature interaction through complex convolution in the spatial domain between the PAN and LRMS features. Specifically, we treat the multispectral feature Ims as the real part and the panchromatic feature Ip as the imaginary part, forming a complex feature map Z,which is formulated as: Z = Ims + iIp . We then pass the complex feature map through several residual blocks with a 3×3 complex convolution layer for information interaction between the multispectral and panchromatic features, eventually obtaining the spatial-domain information Fs.

3.5 Complex Spatial-Frequency Interaction

Spatial-frequency interaction is critical for pansharpening [Zhou et al., 2022c]. Since the powerful capabilities of complex convolutions have been demonstrated in Section 3.1, we have designed a Complex Spatial-Frequency Interaction (CSFI) that leverages complex convolutions to perform complementary learning of spatial and frequency domain information, facilitating the deep fusion of panchromatic and multispectral image data. Specifically, we first combine the frequency-domain and spatial-domain features Ff and Fs into a complex-valued feature Fc. Then, we use complex convolution to fully exploit the complementary information from both domains, outputting a set of complex-valued attention maps Amap, which are weighted by the corresponding real and imaginary parts. To facilitate further interaction of complementary information, we add the weighted real part of the feature map to the imaginary part of the input, and the weighted imaginary part of the feature map to the real part of the input, resulting in a complex-valued feature map Fca with stronger cross-domain interaction, which can be expressed as

Since the target image to be reconstructed is real-valued, we finally convert the complex-valued feature map back to the real domain and obtain the real-valued feature map Fa. As the feature map from the spatial domain branch is closer to the HRMS that needs to be restored, we perform residual connection with Fa and Fs and pass the resulting feature map through a 1x1 convolution block to obtain the final output feature map Ffuse.

4 Experiments

4.1 Dataset and Benchmarks

To evaluate the effectiveness of our network, we conduct experiments on three satellite datasets: WorldView-II (WV2), Gaofen2 (GF2) and WorldView-III (WV3). Each dataset contains a large number of paired low-resolution multispectral images and panchromatic images, which are divided into training, validation, and test sets. The dataset construction follows the methodology of previous studies, using the Wald protocol [Wald et al., 1997] tool to generate training and testing data. To evaluate the effectiveness of our approach, we compare it against several state-of-the-art pansharpening methods, including PNN [Masi et al., 2016], GPPNN [Xu et al., 2021], MutNet [Zhou et al., 2022d], INNformer [Zhou et al., 2022a], HFIN [Tan et al., 2024], and FAME-Net [He et al., 2024], as well as traditional pansharpening techniques such as GS [Laben and Brower, 2000], and IHS [Haydn, 1982].

4.2 Implementation Details

In our experiments, all deep learning models are implemented using PyTorch and trained on an NVIDIA GeForce GTX 3090 GPU. For each dataset, the multispectral (MS) images are cropped into patches of size 32×32, while the corresponding panchromatic (PAN) images are resized to 128×128. During the training phase, the networks are optimized using the Adam optimizer with an initial learning rate of 1×10-4 . After 200 epochs, the learning rate is reduced by half. We adopt standard evaluation metrics, including PSNR, SSIM, SAM, and ERGAS. Furthermore, three widely-used no-reference image quality assessment metrics, namely Dλ, DS and QNR [Vivone et al., 2020], are employed to evaluate real-world full-resolution scenes.

4.3 ComparisonWith State-of-the-Art Methods

Evaluation on reduced-resolution scenes. The comparison results across the three datasets are presented in Table 1, where the best results for each metric are highlighted in red, and the second-best results are indicated in blue. The results demonstrate that our method outperforms state-of-the-art approaches on the majority of metrics, with the only two exceptions being metrics where our method ranks second. Overall, our approach achieves the best performance across all three datasets. In particular, with respect to the Peak Signal-toNoise Ratio (PSNR), our method surpasses all other methods, indicating the high consistency between the images processed by our approach and the real high-resolution images. We also conducted qualitative experiments, and Figure 4 shows samples from the WV2 dataset. The first two rows compare the experimental results, while the last row displays the mean squared error (MSE) between the results of each network and the ground truth. It is evident that our results are closest to the ground truth, with minimal differences.

Evaluation on full-resolution scenes. We also conduct a full-resolution analysis in real-world scenarios to further validate the generalization capability of our method. Experiments are carried out on an additional 200 sets from the GF2 dataset. Since high-resolution multispectral images are unavailable for real-world scenarios, we utilize three commonly employed no-reference metrics, namely Dλ, DS, and QNR, for evaluation. The experimental results, presented in Table 2, show that our method achieves superior performance across all three metrics.

Evaluation on other fusion tasks. To further assess the generalization capability of our proposed method, we apply it to two additional tasks: depth image super-resolution using the NYU v2 dataset and infrared-RGB fusion on the RoadScene dataset. As for depth image super-resolution, we compare our proposed method with nine state-of-the-art deep image super-resolution methods: GF [He et al., 2012], DMSG [Hui et al., 2016], DJFR [Li et al., 2019], DSRNet [Guo et al., 2018], PacNet [Su et al., 2019] and HFIN [Tan et al., 2024]. As for infrared-RGB fusion, we compared our proposed method with ten state-of-the-art visible and infrared image fusion methods: DDcGAN [Ma et al., 2020], DIDFuse

[Zhao et al., 2020], ReCoNet [Huang et al., 2022], SDNet [Zhang and Ma, 2021], U2Fusion [Xu et al., 2020], UMFusion [Wang et al., 2022] and HFIN [Tan et al., 2024]. For the infrared-RGB fusion task, we evaluate the quality of the generated images using three metrics: MI, VIF, and FMI. For the depth image super-resolution task, we use RMSE as the metric to assess the quality of the generated images. The experimental results, shown in Table 3, highlight the best results in red and the second-best in blue. These results demonstrate that our method outperforms competing approaches in the depth image super-resolution task. This further supports the robustness and versatility of our approach, suggesting that our network can be effectively extended to other fusion tasks as well. We also conduct a qualitative analysis on the infraredvisible image fusion task. Figure 5 presents samples from the RoadScene dataset, showing the results of other state-of-theart methods alongside those of our approach. It is evident that the images generated by our method exhibit no distortion relative to the visible light image, while also capturing the texture details of the infrared image.

Table 3: Quantitative comparisons with other fusion methods

4.4 Ablation Experiments

We conducted an ablation study using the WorldViewII dataset to further demonstrate the effectiveness of our method. CFFE and CSFE are the core components of the proposed method, and thus, ablation studies were performed for each of them. Additionally, we also conducted an ablation study on the CSFI of the spatial-frequency fusion.

To validate the effectiveness of the complex-based spatialfrequency branches, we replaced the complex representations and operations with their real-valued counterparts, as shown in Table 4. The results in Table 4 indicate that the removal of either of the complex-based components leads to a performance degradation, thereby confirming the powerful representational ability of complex numbers in the spatialfrequency dual domain for pansharpening.

We further conducted an ablation study on the complex operations used in the spatial-frequency interaction module to evaluate the impact of complex operations on spatialfrequency interaction. Specifically, we replaced the CSFI with two alternatives: a convolution block with residual connections, and a convolution block with residual connections following the spatial attention mechanism (SA), and compared the results with our method. The results, as shown in Table 5, indicate that switching to complex operations enhances the performance of spatial-frequency collaboration.

5 Conclusion

In conclusion, we present PanComplex, a novel complexvalued neural network framework for pansharpening, marking the first attempt to leverage the rich potential of the complex domain for this task. By introducing a dual-domain spatial-frequency structure, we effectively capture and fuse both spatial and spectral features from panchromatic and multispectral images. This framework not only exploits the natural alignment between the complex domain and the frequency domain but also introduces complex convolutions and complementary learning mechanisms to enhance the fusion process. Our experimental results demonstrate that PanComplex achieves superior performance with fewer parameters compared to existing methods, while also exhibiting strong generalization capabilities for other fusion tasks.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (NSFC) under Grants 62422609 and 62476260.