Spatial-Spectral Similarity-Guided Fusion Network for Pansharpening

1 Introduction

Modern imaging sensors often face a trade-off between capturing high-resolution images with limited spectral information and low-resolution images rich in spectral data due to hardware limitations. It is challenging to acquire images with both high spatial resolution and abundant spectral information from a single sensor. To address this problem, satellites are typically equipped with two distinct sensors: one for capturing high-resolution panchromatic (PAN) images and another for low-resolution multispectral (LRMS) images [Zhou et al., 2025]. These complementary images are then fused through pansharpening techniques to generate high-resolution multispectral (HRMS) images for various tasks, such as land-cover recognition [Zhang et al., 2024] and anomaly detection [Shikhar and Sobti, 2024].

∗Corresponding author.

Pansharpening is a promising research topic in the remote sensing community, as it enhances the spatial resolution of multispectral (MS) images while preserving their spectral information through the integration of high-resolution singleband PAN images. There are numerous effective traditional and deep pansharpening methods [Deng et al., 2022]. Traditional pansharpening methods are mainly divided into three categories: component substitution (CS) methods [Choi et al., 2010; Vivone, 2019], multiresolution analysis (MRA) methods [Vivone et al., 2013; Vivone et al., 2018], and variational optimization (VO) methods [Fu et al., 2019; Tian et al., 2021]. These methods are well-suited for a range of satellite images. However, their performance is limited by a heavy reliance on hand-crafted features.

Due to the remarkable feature extraction capabilities of neural networks, numerous deep pansharpening methods have been proposed. Based on their network architecture, these methods are primarily categorized into single-branch and dual-branch structures. As shown in Figure 1 (a), singlebranch networks concatenate both PAN and LRMS images through a single pathway to obtain fusion images [Masi et al., 2016; Deng et al., 2021]. They neglect the unique characteristics of features from different modalities, leading to inadequate feature extraction. In contrast, as shown in Figure 1 (b), dual-branch networks process PAN and MS images through two separated branches, and then fuse them via concatenation before passing them to the subsequent decoder [Zhang et al., 2022; Zhou et al., 2022]. They enable complementary feature extraction and facilitate high-quality image fusion.

Although deep pansharpening methods yield impressive fusion results, two potential issues hinder their performance. First, most existing methods are ineffective for cross-modal feature extraction and fusion [Zhang et al., 2022; Xing et al., 2024]. The differences in spatial and spectral information between PAN and LRMS images make them challenging to extract and combine features effectively in single-branch or dual-branch structures with simple concatenation, limiting the full utilization of complementary modalities. To solve this issue, as shown in Figure 1 (c), a multi-branch structure with an additional fusion branch should be considered to better extract and integrate features from both input images, enabling more effective fusion and enhancing overall performance. Second, most previous methods neglect the similarities between the fused image and both PAN and LRMS images [Peng et al., 2023; Duan et al., 2024]. As shown in Figure 2, the correlation analysis reveals that an LRMS image exhibits a strong spectral correlation with the ground-truth (GT) image, while a PAN image displays a strong spatial correlation with the GT image. The absence of similarity constraints between the fused image and input images in existing methods may result in the loss of important spatial and spectral details, thereby compromising the pansharpening quality. To solve this issue, similarity guidance between the fused features and those extracted from input images should be introduced during network training.

Motivated by the above observations, in this paper, we propose a spatial-spectral similarity-guided fusion network (S3FNet) for pansharpening. As shown in Figure 3, there are three key components in this architecture. Specifically, a shallow feature extraction layer processes PAN and LRMS images through Transformer blocks and convolution to learn initial spatial, spectral and fused features. To enable highlevel feature learning at different scales, a multi-branch asymmetric encoder is presented by incorporating distinct spatial, spectral and fusion branches. Based on a well-designed crossfeature multi-head attention fusion block, a multi-scale reconstruction decoder is able to generate high-quality HRMS images by fusing diverse intermediate feature maps. To preserve as much spatial and spectral information as possible in HRMS images, a similarity-constrained loss is formulated to facilitate the network training and parameter update. Our main contributions are as follows:

• We propose a similarity-guided fusion network designed with a multi-branch structure for PAN and LRMS image fusion. This framework facilitates the effective exploitation of complementary information while capturing both spatial and spectral correlations.
• We present spatial, spectral and fusion branches with distinct designs in the multi-branch asymmetric encoder to hierarchically learn high-level features with discriminative information.
• We design a cross-feature multi-head attention fusion block with spatial and spectral cross-attention mechanisms in the multi-scale reconstruction decoder to facilitate effective cross-modal feature fusion.
• We formulate a similarity-constrained loss for network training that integrates spatial and spectral correlation measures to ensure the preservation of maximum details in the fused images. Experimental results demonstrate the effectiveness of our proposed method.

2 RelatedWorks

In recent years, numerous deep pansharpening methods have achieved remarkable results, owing to their exceptional feature extration capabilities. Pansharpening neural network (PNN) [Masi et al., 2016] is a pioneer to perform image fusion using a three-layer convolutional network. Further, detail injection based convolutional neural network (DiCNN) [He et al., 2019] injects the details of PAN images into LRMS images to provide explicit physical interpretations. Considering the complementarity of input images, bidirectional pyramid network (BDPN) [Zhang et al., 2019] processes them in two separated branches to reconstruct fusion images. To explore local and global context, bidomain modeling pansharpening (BiMPan) method [Hou et al., 2023]

presents local adaptive and global detail branches for better image fusion. Additionally, double U-shape network (U2Net) [Peng et al., 2023] processes PAN and LRMS images through spatial and spectral U-Nets, and performs fusion via the S2Block. To solve edge blur and spectral distortion problems, multi-supervised mask protection network (MMPN) [Chen et al., 2023] utilizes a dual-stream multiscale feature fusion module to fuse input images and their masked data. To improve the interpretability, deep intrinsic supervision pansharpening network (DISPNet) [Wang et al., 2024] incorporates spatial consistency and spectral projection priors into the formulated variational minimization model. In addition, there are many other effective methods by introducing diverse techniques [Duan et al., 2024; Xing et al., 2024; He et al., 2025].

Although great success has been achieved by previous single-branch or dual-branch networks, cross-modal feature fusion and similarity guidance between fusion image and input images are not well taken into consideration. To handle these issues, we will introduce our proposed method in detail.

3 Methodology

Figure 3 illustrates the overall structure of S3FNet. The proposed network mainly consists of three parts, including a shallow feature extraction (SFE) layer, a multi-branch asymmetric encoder and a multi-scale reconstruction decoder with the well-designed cross-feature multi-head attention fusion block (CMAFB). The proposed network is trained using a similarity-constrained loss to enhance image fusion.

3.1 Shallow Feature Extraction Layer

To ensure the consistent feature map size, we first replicate the PAN image P ∈ RH×W C times along the spectral dimension, resulting in Prepeat ∈ RH×W×C . Then, we perform bicubic interpolation on the MS image MS ∈ R H4 × W4 ×C to obtain MS×4 ∈ RH×W×C . Both preprocessed images are input to the shallow feature extraction layer consisting of four cascaded Transformer blocks [Zamir et al., 2022], yielding spatial features X1 ∈ RH×W×S from Prepeat and spectral features Y1 ∈ RH×W×S from MS×4. Besides, we concatenate Prepeat and MS×4 to produce the initial fused features F1 via a convolutional layer. The shallow feature extraction layer is designed to capture initial spatial, spectral and fusion features from PAN and LRMS images. They are served as input for the subsequent encoder-decoder network to enable further feature extraction and fusion.

3.2 Multi-Branch Asymmetric Encoder

Unlike existing single-branch or dual-branch structures, the multi-branch asymmetric encoder adopts a three-branch design, consisting of a spatial branch, a spectral branch and a fusion branch. Each branch is specifically tailored with distinct designs to learn high-level spatial, spectral and fused features at different scales for high-quality image fusion.

Spatial Branch. The spatial branch takes X1 as input and enhances spatial features X2 and X3 through a cascaded process using a combination of ResNet block (ResBlock) [Szegedy et al., 2017], spatial attention block (SpaAttBlock), and downsampling operation. The ResBlock focuses on local feature extraction, the SpaAttBlock emphasizes important local features across regions, and the downsampling operation reduces spatial resolution. The process for learning the enhanced spatial features is as follows:

where n represents the stage index (n ∈ {2, 3}). ResBlock(·) denotes the ResBlock, comprising two 3 × 3 convolutional layers followed by a leaky ReLU activation function. SA(·) represents the SpaAttBlock, composing two parallel pooling operations to aggregate channel information, followed by a 3 × 3 convolutional layer and a sigmoid activation function. Input features of the SpaAttBlock are processed through element-wise multiplication with attention features, followed by element-wise addition. Down(·) represents the downsampling operation, involving a 2 × 2 kernel with stride 2 and a depth-wise convolutional layer to increase the feature map channels.

Spectral Branch. The spectral branch takes Y1 as input and learns advanced spectral features Y2 and Y3 through a cascaded process using a combination of ResBlock, spectral attention block (SpeAttBlock) [Woo et al., 2018] and downsampling operation. The ResBlock and downsampling operation are the same as those in the spatial branch, while the SpeAttBlock highlights key features by learning the importance of each spectral channel. The process for learning advanced spectral features is as follows:

where CA(·) represents the SpeAttBlock, comprising two parallel pooling operations to aggregate spatial information and generate spatial context descriptors, followed by two multi-layer perceptrons (MLPs) with 1 × 1 convolution and ReLU activation function, and a sigmoid activation function to obtain attention features. The input features of the SpeAttBlock are combined with attention features through elementwise multiplication and addition.

Fusion Branch. The fusion branch takes F1 as input and learns complex fusion features F2 and F3 in a cascaded manner using a combination of two Transformer blocks and a downsampling operation. The Transformer block is the same as in the shallow feature extraction layer, while the downsampling operation mirrors those in the spatial and spectral branches. The process for learning fusion features is as follows:

where TB(·) represents the Transformer block. The use of the Transformer block is motivated by its self-attention mechanism, enabling global feature extraction. This mechanism allows the model to capture long-range dependencies and relationships between different parts of input data, enhancing its ability to understand complex patterns.

3.3 Multi-Scale Reconstruction Decoder

To facilitate cross-modal image fusion, the outputs from the three branches of the encoder are fed into a multi-scale reconstruction decoder. This decoder consists of three stages. Each stage contains one upsampling operation (except for the first stage) and a multi-head attention fusion block (CMAFB). In each stage, the spatial and spectral features extracted from the corresponding branches, along with the fused features from the previous stage, are input to the CMAFB. The resulting features are then added to the skip-connected fusion features from the encoder (F3, F2, and F1) to generate the fused features for the next stage (F4, F5, and F6). Finally, F4 and F5 are upsampled to the spatial resolution of F6, and all three are concatenated and processed by a convolutional layer. The output is added to the MS×4 to produce the final fused result ̂ MS, as follows:

where Up(·) denotes the upsampling operation used to align the spatial dimensions of feature maps at different scales.

3.4 Cross-Feature Multi-Head Attention Fusion

Unlike traditional fusion strategies that directly concatenate features, as illustrated in Figure 4, the CMAFB incorporates spatial cross-attention (SpaCA) and spectral cross-attention (SpeCA) mechanisms to enhance the cross-modal fusion of shallow and deep features by highlighting their distinctions. Specifically, the SpaCA focuses on the differences between spatial and fusion features along the spatial dimension, while the SpeCA emphasizes the differences between spectral and fusion features along the spectral dimension. For each CMAFB, spatial, spectral and fusion feature maps of corresponding scale are taken as inputs. For simplicity, we ignore the subscripts of these feature maps and reshape them to matrix representations. Then, they are linearly mapped to the query Q and key K1 in the speCA, and to the key K2 and value V in the SpaCA as follows:

where WQ, WV , WK1 and WK2 are learnable parameters in the linear mapping layer. Subsequently, K1 , K2 , Q and V are divided into m heads based on S channels of the features processed in the current stage. The dimension of each head is d = S/m. For simplicity, the operations of splitting and merging the multiple heads are omitted in the figure. Thus, the calculation of the spatial and spectral attention matrices of the j-th head, Aspa j and Aspe j , is as follows:

where (·)T denotes the matrix transpose operation.

After obtaining Aspe j and Aspa j , we perform matrix multiplication between them and Vj from spatial features and Qj from spectral features, respectively. This results in the fusion output of the j-th head, headj, combining both spatial and spectral information. The specific calculation is as follows:

Subsequently, headj from different heads are concatenated along the channel dimension to form the final fusion output Head via the cross-attention mechanism. This fusion features integrate spatial and spectral information while employing attention mechanisms to capture long-range dependencies.

Followed by the cross-attention mechanisms, a locallyenhanced feed-forward (LeFF) network layer [Wang et al., 2022] is introduced. It consists of a 1 × 1 convolution to increase the number of channels, a 3×3 convolution to capture local information, and another 1 × 1 convolution to reduce the number of channels. This design strengthens the representation of local information. The final fusion features Fn are derived as follows:

where LN(·) represents layer normalization, and LeFF(·) denotes the locally-enhanced feed-forward layer. This process integrates spatial and spectral details through cross-attention mechanisms and local feature enhancements, resulting in generating the final fusion features.

3.5 Loss Function

Overall Loss. To achieve better fusion, we incorporate the similarity constraint into our loss function, i.e.,

where L1 = ∥GT - ̂ MS∥1 measures the ℓ1 loss between the GT and reconstructed fusion images, and Lsim represents the multi-scale spatial-spectral similarity-constrained loss by imposing constraints on the correlations between the fused features and both the spatial and spectral features at different scales. α is a hyperparameter that balances the two loss terms.

Multi-Scale Spatial-Spectral Similarity-constrained Loss.

To ensure the fusion image is closer to the GT image, similarity constraints from both spatial and spectral perspectives should be incorporated into the pansharpening process. Inspired by [Zhao et al., 2023], we design a multi-scale spatialspectral similarity-constrained loss based on spatial and spectral correlations between intermediate features from the spatial, spectral and fusion branches at different scales. Measured by the Spearman correlation coefficient, this loss ensures the fusion image maintains high spectral and spatial correlations with the GT image. It is formulated as follows:

where β is a parameter to control the ratio of spatial and spectral similarities. CC spa i and CC spe i denote the spatial and spectral correlation coefficients at the i-th scale, respectively.

Spatial and Spectral Correlation Coefficients. Spatial correlation is calculated to measure the correlation between the fused and spatial feature maps in the spatial dimension. It is formulated as:

where Spearman(·) calculates the Spearman correlation coefficient, and Si denotes the number of spectral channels at the i-th scale. Xs i and F s i represent the s-th spectral channel of features Xi and Fi, respectively. Before calculating the Spearman correlation coefficient, they are flattened along the spatial positions of the pixels. Similarly, spectral correlation is computed to quantify the correlation between the fused and spectral feature maps in the spectral dimension. It is represented as:

where Hi and Wi represent the height and width of features Yi and Fi at the i-th scale, respectively. Y j i and F j i denote the j-th pixel of features Yi and Fi, respectively.

4 Experiments

4.1 Experimental Setup

Datasets. Three datasets used in this study were obtained from the PanCollection repository [Deng et al., 2022], including data from the WorldView-3 (WV3), GaoFen-2 (GF2), WorldView-2 (WV2) satellites. Both reduced-resolution and full-resolution datasets are adopted for experiments. The reduced-resolution data is synthesized using Wald's protocol, where LRMS and PAN images are downsampled by a factor of 4, with the original LRMS images serving as the ground truth.

Compared Methods. To evaluate the proposed method, several state-of-the-art pansharpening methods are used for comparison, including three traditional methods (BT-H [Aiazzi et al., 2006], MTF-GLP-FS [Vivone et al., 2018], and BDSD-PC [Vivone, 2019]) and eight deep learning methods (PNN [Masi et al., 2016], MSDCNN [Wei et al., 2017], DiCNN [He et al., 2019], BDPN [Zhang et al., 2019], FusionNet [Deng et al., 2020], LAGNet [Jin et al., 2022] , BiMPan [Hou et al., 2023] and Pan-Mamba [He et al., 2025]).

Evaluation Metrics. To evaluate the quality of the fused images, the spectral angle mapper (SAM), relative global dimensional synthesis error (ERGAS), Q4/Q8 metrics, and structural content correlation (SCC) are employed for the reduced-resolution datasets. For the full-resolution datasets, Dλ, Ds, and quality with no reference (QNR) are adopted. Details of these metrics can be found in [Lu et al., 2023].

Implementation Details. Our model was implemented using PyTorch on a mahine with Nvidia 4090 GPU. The Adam optimizer is used for network training over 300 epochs with a batch size of 16. The initial learning rate is set to 0.001 and halved every 100 epochs. For the loss function, α = 0.001 and β = 0.5, with α decaying during training. For the shallow feature extraction layer, the output channels are set to S = 32 for the three datasets. Additionally, C = 8 for the WV3 and WV2 datasets, and C = 4 for the GF2 dataset. For BiMPan and Pan-Mamba, we use the default settings from their released code. For other deep learning methods, we adopt the settings specified in [Deng et al., 2022].

4.2 Experimental Results

Comparison on Reduced-Resolution Data. The quantitative results of different methods on reduced-resolution datasets are presented in Table 1. The three traditional methods exhibit limited performance because they heavily rely on hand-crafted features. In contrast, the deep learning-based methods significantly outperform traditional methods owing to their superior feature extraction capabilities. Compared to other methods, our S3FNet shows the best performance on reduced-resolution datasets, verifying its effectiveness in pansharpening tasks. Additionally, the visual results on a sample from the WV3 dataset are displayed in Figure 5. It is obvious that our S3FNet obtains the fusion image most similar to the GT image. The absolute error maps show that S3FNet has notably fewer residuals than other methods.

Comparison on Full-Resolution Data. To further verify the generalization ability of our S3FNet, experiments on fullresolution datasets are conducted, and the quantitative results are presented in Table 2. Experimental reults demonstrate that our S3FNet also achieves the best performance among all compared methods on full-resolution datasets to produce fusion images with low spectral and spatial distortion. This indicates its practicality and adaptability for remote sensing image fusion.

4.3 Ablation Study

To validate the effectiveness of key components in S3FNet, ablation study is conducted on the WV3 dataset with three degraded variants. Specifically, the multi-branch (MB) encoder can be replaced with a dual-branch structure, the CMAFB in the decoder can be replaced with a standard convolution, and the similarity-constrained loss can be included or excluded. The corresponding results are shown in Table 3. Setting (I) shows the worst results because no key components are included. By introducing the multi-branch encoder, setting (II) outperforms setting (I) due to more effective feature processing across different branches. By retaining the CMAFB, setting (III) is superior to setting (II) because of more effective cross-modal feature fusion. Finally, our S3FNet, with the multi-branch encoder, CMAFB and similarity-constrained loss, outperforms other degraded settings. These results confirm the effectiveness of the design in S3FNet.

4.4 Parameter Study

In S3FNet, α controls the impact of the similarity-constrained loss, while β adjusts the balance between spatial and spectral similarities. To investigate the effect of different parameter settings, α is varied over {0.0001, 0.001, 0.01, 0.1, 1, 10} with β fixed at its default value of 0.5. Conversely, β is varied from 0.1 to 0.9 in increments of 0.1 with α fixed at its default value of 0.001. The quantitative results for different parameter settings are reported in Figure 6. When α is set too large, the two loss terms become unbalanced in magnitude. With α = 0.001 and β = 0.5 as default values, S3FNet achieves superior performance.

5 Conclusion

In this work, we proposed a spatial-spectral similarity-guided fusion network named S3FNet for pansharpening. The S3FNet is designed with a multi-branch architecture. Specifically, the multi-branch structure is implemented in the encoder to generate high-level spatial, spectral and fusion features from the initial features of input images obtained by the shallow feature extraction layer. Driven by the well-designed cross-feature multi-head attention fusion block, the decoder reconstructs the fused images by incorporating spatial and spectral details at different scales. The proposed network is trained using a similarity-constrained loss to ensure highquality image fusion. Extensive experiments on three satellite datasets verify the effectiveness of the proposed method against state-of-the-art methods.

Acknowledgments

This work was funded by the China National Key R&D Program (Grant 2023YFF0807000).