Dual-Perspective United Transformer for Object Segmentation in Optical Remote Sensing Images

1 Introduction

Object segmentation in ORSIs has gained considerable attention [Liu et al., 2023b] due to their wide-ranging practical applications, such as environmental monitoring, urban planning, military reconnaissance, etc. In contrast to common object segmentation in natural scene images (NSIs) [Sun et al., 2022; Duan et al., 2025], remote sensing objects are more challenging [Li et al., 2019] due to arbitrary orientations, drastic scale changes, and complex backgrounds of targets. Moreover, ORSIs are typically collected by sensors on aircraft or satellites and presented as bird's-eye views. Compared to NSIs, they suffer from unstable imaging conditions and greater susceptibility to interference, which increases the difficulty of the object segmentation task. This means that the designed architecture should pay more attention to how to effectively leverage a variety of significant information.

∗Corresponding author.

Existing ORSIs object segmentation methods are mainly based on data-driven deep learning approaches. As shown in panels (a)-(c) of Fig. 1, based on differences in encoding features, we classify them into three types, that is, CNN-based, Transformer-based, and hybrid structure-based features. Early, ORSI-based object segmentation methods extract initial features using convolutional encoders (e.g., VGG16 [Simonyan and Zisserman, 2014] or ResNet50 [He et al., 2016]). These initial features are then refined through various optimization strategies, including multi-scale learning [Wang et al., 2022c; Liu et al., 2023b], boundary guidance [Zhou et al., 2022b; Zeng et al., 2023], attention mechanisms [Zhang et al., 2021; Gu et al., 2023; Xia et al., 2024], and among others. Despite their success, CNN-based methods struggle to capture the relationships between all pixels, limiting the network's ability to achieve global perception due to the inherent constraints of the receptive field. As shown in SeaNet [Li et al., 2023b] and SFANet [Quan et al., 2024] in Fig. 1, relying solely on local receptive fields can lead to incomplete segmentation results, particularly for remote sensing objects that span a relatively large area in the input ORSIs.

To tackle this limitation, Transformer-based methods are proposed, which utilize self-attention to model long-range dependencies that benefit global understanding. Specifically, GeleNet [Li et al., 2023a], and TLCKDNet [Dong et al., 2024] adopt a Transformer encoder (e.g., ViT [Yuan et al., 2021] or PVTv2 [Wang et al., 2022b]) to extract initial features with global information for segmenting remote sensing objects. However, as we know, Transformer-based models lack local inductive bias when processing images [Chen et al., 2023; Sun et al., 2024b], resulting in initial features that are deficient in spatial detail information, which leads to the inferior prediction of clear object boundaries (As depicted in TLCKDNet [Dong et al., 2024] in Fig. 1). For ORSIs object segmentation, effectively utilizing global relationships and local details is crucial for accurate results. Recently, hybrid structure-based methods [Wang et al., 2022a; Zhao et al., 2024; Yan et al., 2024] have emerged, which directly combine parallel CNN and Transformer features (as illustrated in Fig. 1 (c)) to obtain features that incorporate both global semantic relationships and local spatial details.

In fact, directly aggregating two types of features in the spatial domain is challenging. As presented in Fig. 1(e), in our experiments, we observe that when features from different architectures are directly aggregated, their performance, although significantly higher than CNN-based methods, is slightly lower than that of Transformer-based methods. This is unexpected, as the two feature types should theoretically be complementary, yet the experiment shows a discrepancy. The reasons can be attributed to two factors: on the one hand, the internal structure and output paradigm of the Transformer differ significantly from those of the CNN [Zhu et al., 2023; Huang et al., 2024], leading to distinct feature representations and information distributions. On the other hand, these hybrid models use independently trained public encoders, meaning the features are not influenced or adapted to each other during the pre-training process. As a result, there is a lack of correlation between the features. Directly integrating these features may introduce unintended noise, as the two feature types do not fully align in the semantic space, potentially bringing in irrelevant or conflicting information, which can lead to unsatisfactory outcomes (as demonstrated in ADSTNet [Zhao et al., 2024] in Fig. 1). Besides, hybrid structure-based methods often involve many parameters due to the use of two parallel encoders, which increases the complexity of models (as depicted in Fig. 1(f) ). Consequently, both training and inference become time-consuming and resource-intensive.

To address the above challenges, we propose a novel framework named the Dual-Perspective United Transformer (DPU-Former) for object segmentation in ORSIs. As indicated in Fig. 1(d), unlike existing ORSIs object segmentation frameworks that rely on the public encoder, DPU-Former utilizes a unified architecture (i.e., Global-local mixed attention, called GLMA) to model local details and global relationships of input images simultaneously, which introduces the Fourierspace merging strategy that projects heterogeneous CNN and Transformer features from the spatial domain to the Fourier space, unifying them into the same frequency band for efficient aggregation. Moreover, the DPU-Former encoder incorporates a gated linear feed-forward network (GLFFN) for enhancing the model's expressive capability. Different from the previous FFN [Wang et al., 2022b], our GLFFN further increases diversity of features and selectively filters redundancy information. For the DPU-Former encoder, the dualperspective simultaneous modeling strategy in GLMA enables the two feature types to enhance their complementarity through iterative adaptation during the pretraining stage. Furthermore, the DPU-Former decoder constructs the adaptive cross-fusion attention and the structural enhancement module, inheriting the benefits of our GLMA, to strengthen and integrate multi-source information from different stages and enhance the structural consistency of objects, thereby improving the accuracy of ORSIs object segmentation.

The proposed DPU-Former model does not rely on public encoders (e.g., ResNet50 [He et al., 2016], VGG16 [Simonyan and Zisserman, 2014], or PVTv2 [Wang et al., 2022b]) that model data from a single perspective. It extracts complementary information from global and local perspectives, effectively balances global dependencies and local details, and addresses the problem of heterogeneity between CNN and Transformer features. Extensive experimental results on widely-used ORSIs datasets demonstrate the superiority of our DPU-Former method against 16 state-of-the-art (SOTA) models. Additionally, we extend the DPU-Former model to multiple visual scenarios (i.e., natural, camouflage, and medical images), achieving outstanding performance.

2 RelatedWork

Object Segmentation in ORSIs. Compared to natural scene images, ORSIs possess characteristics such as irregular topology structures, extreme-scale variations, and imaging interference, making object segmentation tasks for ORSIs particularly challenging. To address these issues, numerous CNN-based methods [Zhou et al., 2022b; Liu et al., 2023b; Quan et al., 2024] have been proposed, which consistently employ various strategies to optimize the initial features from the CNN encoder for detecting remote sensing objects within scenes. Specifically, LVNet [Li et al., 2019] first constructed an ORSIs dataset and proposed the L-shaped module to perceive the diverse scales and local details of objects. DAFNet [Zhang et al., 2021] designed the dense attention fluid structure to integrate multi-level attention cues. Next, ERPNet [Zhou et al., 2022b] utilized a shared CNN encoder and two parallel decoders (i.e., edge extraction and feature fusion) to segment remote sensing objects. Similarly, MJRBM [Tu et al., 2022], AESINet [Zeng et al., 2023], and SeaNet [Li et al., 2023b] used edge cues to improve segmentation accuracy. Furthermore, SDNet [Liu et al., 2023b], and SFANet [Quan et al., 2024] introduced a series of convolutional modules to increase the diversity. Although the above CNN-based methods attain superior performance, they are limited by the receptive field of convolutional structures, which can lead to only local attribute features in the proposed models.

Accordingly, researchers attempt to introduce Transformer frameworks [Gao et al., 2023; Wu et al., 2024; Sun et al., 2024c; Xie et al., 2025] with global perception. For example, GeleNet [Li et al., 2023a] exploited PVT blocks [Wang et al., 2022b] to capture global contexts. TLCKDNet [Dong et al., 2024] adopted the ResT [Zhang and Yang, 2021] as the extractor to provide effective representations. However, Transformer-based models generally suffer from poor handling of local bias, which is not conducive to edge details. Recently, HFANet [Wang et al., 2022a] and ADSTNet [Zhao et al., 2024] incorporated CNN and Transformer encoders to obtain initial features with abundant local and global information. However, they overlook the heterogeneity in the CNN and Transformer features and adopt two separate public encoders, increasing the model's burden. Differently, we propose a novel DPU-Former model, which no longer utilizes the public encoders to extract initial features. It is based on our unique architectures to capture local details and global relationships simultaneously and projects two types of features into Fourier space to eliminate their heterogeneity.

3 Methodology

The overview framework of our DPU-Former is illustrated in Fig. 2, consisting of an encoder-decoder architecture. The following DPU-Former encoder, which is composed of a global-local mixed attention (GLMA) and a gated linear feed-forward Network (GLFFN) at each stage, is applied to capture initial encoding features with sufficient local details and global semantics. Additionally, we introduce an adaptive cross-fusion attention (ACFA) and a structural enhancement module (SEM) into the designed DPU-Former decoder, aimed at integrating diverse information at different layers and enhancing the consistency of object structures.

3.1 DPU-Former Encoder

For ORSIs object segmentation tasks, a strong encoder is more advantageous for achieving accurate results. However, existing methods rarely consider this important factor, focusing instead on optimizing the initial features provided by public encoders. This leads to an awkward problem where initial features from poor-quality encoders make it difficult to achieve great performance, despite various optimization strategies. Based on this, we design a DPU-Former encoder that simultaneously encodes input images from local and global perspectives to obtain powerful initial features.

Global-local mixed attention enhances the encoder's ability to capture both fine-grained details and broader contexts, which consists of three components (i.e., global perspective, local perspective, and Fourier-space merging strategy). Concretely, given an input image I ∈ RH×W×3 , we first utilize the patch and position embeddings to divide the image into patches with positional information and then use a layer normalization to generate the feature ˇ F1 with a stable input distribution. In the global perspective, we adopt efficient self-attention to model long-range dependencies between all pixels. Technically, we exploit linear layers to generate the Query (Q1), Key (K1), and V alue (V1) required for selfattention, which can be defined as:

where Θ(·) denotes a linear layer, SR(·) is the spatial reduction [Wang et al., 2022b] to reduce the complexity. Subsequently, we establish the relationships between all pixels to generate the global feature ˇ F g 1 to focus on the object body information, i.e., ˇ F g 1 = δ( Q1K⊤ 1 √dk )V1, where √dk presents a scaling factor to prevent values from becoming too large during the dot-product operation, δ(·) is the Softmax function. Meanwhile, in the local perspective, we adopt a series of well-designed convolutional operations to extract local details, which contain abundant object boundaries. Different from the classic plugin [Liu et al., 2018], we select the lightweight depthwise separable convolution, which features lower parameters and complexity, and introduce short connections to enhance the correlations between different local features. Specifically, the input feature ˇ F1 is first resized to H4 × W4 ×C1, and then multi-scale local features are extracted by combining pointwise convolutions and depthwise separable convolutions, which can be formulated as:

where C1 represents a 1 × 1 pointwise convolution, DCn denotes a depthwise separable convolution of n×n kernels, and we set n to 2k+1. + is an element-wise addition. Afterward, we aggregate these features to obtain a local feature ˇ F l1 with abundant spatial details, i.e., ˇ F l1 = C1 ∑3k=1 ˇ F lk 1 .

Although features ˇ F l1 and ˇ F g 1 are theoretically complementary, the different architectural paradigms of CNN and Transformer lead to the heterogeneity [Zhu et al., 2023; Huang et al., 2024] of local and global features in the spatial domain, resulting in direct aggregation not maximizing the advantages of both. Based on this, we propose the Fourierspace merging strategy, which transforms different spatial features into spectral features with the same frequency range, placing them on the same baseline to eliminate the differences in spatial features [Sun et al., 2024a]. Technically, as depicted in Fig. 3, we employ the Fast Fourier Transform to map the local and global features into the Fourier space, i.e.,

where FFT[·] is the Fast Fourier Transform, f(x, y) denotes a spatial feature with pixel coordinates (x, y), i is an imaginary unit, u and v present horizontal and vertical indexes in the Fourier space. Furthermore, inspired by the enhancement of spectral features through exploiting a set of weights [Huang et al., 2023], we learn a set of composite weights to strengthen the significant information in spectral features. Different from the weight for a single source, our composite weight Λc1 is derived from the features of both ˜ F l1 and ˜ F g 1 , effectively integrating local and global contextual information. Λc1 can be represented as follows:

where ϱ(·) is the sigmoid function, BR(·) denotes a batch normalization and a ReLU function. Afterward, two spectral features are adaptively corrected using the obtained composite weight, which integrates local details and global relationships, enhancing the frequency bands within the spectral features to different degrees. Subsequently, we aggregate the optimized two spectral features and convert them back to the original domain through the Inverse Fast Fourier Transform to obtain the powerful feature ˇ F gl 1 , as followed:

where IFFT[·] denotes the Inverse Fast Fourier Transform.

Gated linear feed-forward network strengthens the model's capacity to learn complex patterns. Unlike the typical FFN [Liu et al., 2021b; Wang et al., 2022b], our GLFFN introduces a gating mechanism to control the flow of meaningful features and weaken noise. Additionally, it incorporates multiple depthwise separable convolutions to further enhance feature diversity. Technically, for the input feature ̂ F gl 1 , we first utilize a layer normalization (LN(·)) to generate the feature ˇ F gl 1 ( ˇ F gl 1 = LN( ̂ F gl 1 )) with a stable distribution. Subsequently, we apply a linear layer and several depthwise separable convolutions to perform linear transformations, generating features ˇ G 1 1 and ˇ G 2 1 . Regarding multiple depthwise separable convolutions, we adopt a hierarchical manner to increase the diversity of features and provide a broad context. The process can be described as follows:

where [·] is the concatenation operation, SP{·} represents splitting the feature into four parts based on the number of channels. Note that z = 1, 2, 3, 4 and z - 1 ≥ 1. Afterward, we utilize the GELU function to perform non-linear activation on the feature ˇ G 2 1 , and aggregate the activated feature with the feature ˇ G 1 1 to control the significant information flow for obtaining high-quality feature F e 1 , as shown in:

where Φ(·) denotes the GELU non-linear activation function, ⊗ denotes the element-wise multiplication operation.

In the subsequent stages, F e 1 serves as input and continues to perform multi-stage feature extraction through the proposed GLMA and GLFFM, ultimately generating three initial features F e 2 , F e 3 , and F e 4 in our encoder, that is,

where GLMA(·) and GLFFN(·) denote the components of GLMA and GLFFM in our DPU-Former encoder.

3.2 DPU-Former Decoder

The proposed DPU-Former decoder acts on the initial features {F e i }4i=1 from the DPU-Former encoder, with a resolution of ( H 2i+1 , W 2i+1 ) and the number of channels {64,128,320,512}, which contains two key components (i.e., adaptive cross-fusion attention and structural enhancement module). Specifically, we first reduce the dimensionality of input initial features to 128 channels using a set of convolution operations, and then adopt a joint attention [Woo et al., 2018] and a layer normalization to activate positive clues to obtain the feature ̂ F e i , which is defined as:

where Ψ(·) represents a set of convolution operations that contains two 1×1 convolutions and a 3×3 convolution, JA(·) denotes the joint attention, F e 5 is obtained by DASPP [Yang et al., 2018] on F e 4 to provide semantic guidance.

Adaptive cross-fusion attention not only inherits the advantages of our GLMA by optimizing the input features from two perspectives, but also integrates complementary information (i.e., high-level semantics and low-level structures) from different layers of input features. It is worth noting that our ACFA utilizes interactive cross-attention to model long-range dependencies and exchange internal information. Technically, we first employ depthwise separable convolutions and pointwise convolutions to generate the required sets of Query, Key, and V alue, and then perform information exchange to obtain the fused global feature ̂ Wg 4 , as followed:

Moreover, we use the same local perspective in our GLMA to capture local information, i.e., ̂ Wl4 = LP( ̂ F e 5 ) + LP( ̂ F e 4 ), where LP(·) denotes the operation of local perspective in the proposed GLMA. Furthermore, the optimized features ̂ Wg 4 and ̂ Wl4 are aggregated in Fourier space through the FSM strategy to produce the feature ̂ Wac 4 , ̂ Wac 4 = FMS(̂ Wg 4 , ̂ Wl4 ). Structural enhancement module focuses on important region of objects that might have been overlooked and strengthens structural consistency. Specifically, for the input ̂ Wac 4 , we exploit several atrous convolutions with different filling rates and reverse operations to obtain a hybrid reverse attention map Atth4 , that is, Atth4 = ∑4k=1RA(AC 2k-1 3 C1 ̂ Wac 4 ), where RA(·) represents reverse attention, i.e., RA(·) = -1 ⊗ ϱ(f(x, y)) + 1. AC 2k-1 3 denotes the 3×3 atrous convolution with a filling rate of 2k -1. Furthermore, we embed the hybrid reverse map to enhance structural information to generate the feature ̂ Wse 4 , that is, ̂ Wse 4 = C1[Atth4 ⊗C1 ̂ Wac 4 , C1 ̂ Wac 4 ] + ̂ Wac 4 .

Finally, we integrate the optimized features ̂ Wac 4 and ̂ Wse 4 to obtain a high-quality representation Wd 4 to accurately detect remote sensing targets. It can be formulated as:

Note that in the subsequent DPU-Former decoder, the feature Wd i+1 generated in the previous layer and the same layer feature F e i from the encoder are used as inputs to gradually aggregate to produce the discriminative feature Wd i through the proposed ACFA and SEM operation.

3.3 Loss Function

In our method, we combine the weighted binary crossentropy (BCE) and the weighted intersection over union (IoU) functions to supervise the training model, as followed:

where LWIoU and LWBCE are the weighted IoU and BCE loss functions. Pi represents the feature Wd i after being reduced to one channel, and Gt denotes the ground truth. Note that Wd 5 is the feature F e 5 from the DASPP [Yang et al., 2018].

4 Experiments

4.1 Experimental Settings

Datasets and Evaluation Metrics. We conduct experiments on three ORSIs datsets, including ORSSD [Li et al., 2019], EORSSD [Zhang et al., 2021], and ORSIs-4199 [Tu et al., 2022]. ORSSD comprises 600 images for training and 200 images for testing. EORSSD contains 1400 training images and 600 testing images. ORSIs-4199 is the most challenging dataset and consists of 2000 images for training and 2199 images for testing. Furthermore, we adopt several evaluation metrics to evaluate our model, including the Weight Fmeasure (F w m ), Maximum F-measure (F m m ), S-measure (Sm), and E-measure (Em), and Mean Absolute Error (M).

Implementation details. We leverage the PyTorch framework to implement our DPU-Former, training it on a setup of four NVIDIA GTX 4090 GPUs. Similarly to the typical ResNet [He et al., 2016] encoder, the DPU-Former encoder is pre-trained on ImageNet-1k [Deng et al., 2009]. In the ORSIs object segmentation task, we employ the Adam optimizer with an initial learning rate set to 1e-4 and apply weight decay at a rate of 0.1 every 40 epochs. The batch size is set to 48. During training, the input images are uniformly cropped to 352× 352, and the whole process takes 80 epochs. Following [Quan et al., 2024], we apply data augmentation techniques (e.g., horizontal flipping and rotation) to improve the model's adaptability to remote sensing targets.

4.2 Comparison with the State-of-the-Arts

We compare the DPU-Former with 16 ORSIs segmentation methods, including PA-KRN[Xu et al., 2021], VST [Liu et al., 2021a], DAFNet [Zhang et al., 2021], ERPNet [Zhou et al., 2022b], EMFINet [Zhou et al., 2022a], CorrNet [Li et al., 2022], HFANet [Wang et al., 2022a], MJRBM [Tu et al., 2022], SeaNet [Li et al., 2023b], AESINet [Zeng et al., 2023], SRAL [Liu et al., 2023a], SDNet [Liu et al., 2023b], ICON

[Zhuge et al., 2023], ADSTNet [Zhao et al., 2024], TLCKDNet [Dong et al., 2024], and SFANet [Quan et al., 2024]. To ensure a fair comparison, all results are provided either directly from the authors or from available open-source code.

Quantitative results. Table 1 lists the performance of our DPU-Former against the 16 comparison methods. Specifically, our F w m scores surpass the second-best method by a clear margin of 1.03% and 1.10% in the EORSSD [Zhang et al., 2021] and ORSSD [Li et al., 2019] datasets, respectively. In the extremely challenging ORSIs-4199 [Tu et al., 2022] dataset, our DPU-Former is better than the second-best method in terms of F w m (0.870 vs. 0.852), F m m (0.896 vs. 0.884), Sm (0.877 vs. 0.873), Em (0.953 vs. 0.944), andM (0.026 vs. 0.028). Furthermore, we provide the parameters and FLOPs of the model in Table 2. To ensure fairness, we set the input image size for all methods to 352 × 352. From Table 2, it is evident that our model is highly competitive in terms of both the parameters and FLOPs. These quantitative results demonstrate that our DPU-Former method achieves superior accuracy and efficiency.

Qualitative results. Fig. 4 gives the visual prediction results of the proposed DPU-Former and existing ORSIs segmentation methods under different remote sensing scenes, including building (1st and 5th rows), aircraft (2nd row), motorcar (3rd row), river (4th row), and court (6th row). As depicted in Fig. 4, our DPU-Former model outperforms the previous SOTA methods (e.g., SFANet [Quan et al., 2024] and ADSTNet [Zhao et al., 2024]) in the processing of remote sensing objects with various types and sizes.

4.3 Ablation Study

We conduct extensive ablation experiments to analyze the effectiveness of all proposed components and strategies on two public ORSIs object segmentation datasets.

Effectiveness of the DPU-Former encoder. In the DPUFormer encoder, the key components are "GLMA" and "GLFFN". We begin by assessing the influence of different perspectives in our "GLMA". From Table 3, embedding the local perspective (Tab. 3(c)) into the global perspective (Tab. 3(b)) significantly improves performance, which is attributed to the increase of local details in input features. Moreover, the Fourier-space merging strategy (Tab. 3(d)) outperforms element-wise addition (Tab. 3(c)) from the spatial domain because the representation of unified frequencies in the Fourier domain reduces the impact of differences in features. The visual heatmaps for each stage is shown in Fig. 5. Furthermore, we study the importance of our GLFFN by removing them in experiments. As shown in Table 3 (a) and (e), we can see that "GLFFN" is meaningful for the designed DPUFormer encoder due to its powerful linear enhancement and redundancy information filtering. Furthermore, we compare our DPU-former encoder with existing classic encoders (i.e., VGG16 [Simonyan and Zisserman, 2014], ResNet50 [He et al., 2016], Res2Net [Gao et al., 2019], Swin-small [Liu et al., 2021b], PVTv2-b3 [Wang et al., 2022b]) in Table 4. Additionally, as depicted in Fig. 6, we perform plug-and-play studies on the DPU-Former encoder using the four existing methods. Our DPU-Former encoder is highly competitive in the performance and efficiency of segmenting objects.

Effectiveness of the DPU-Former decoder. The DPUformer decoder inherits the advantages of retaining local details and capturing long-range dependencies and introduces cross-fusion and structural reinforcement. We conduct experimental analysis on the proposed "ACFA" and "SEM" of the proposed DPU-Former decoder. As shown in Table 3, the individual ACFA (Tab. 3(f)) and SEM (Tab. 3(g)) surpass the FPN [Lin et al., 2017] decoder (Tab. 3(d)), with a notable performance improvement of 2.16% and 2.39% respectively in the Sm evaluation metric on EORSSD dataset. Furthermore, the best result (Tab. 3(h)) can be obtained by combining our "ACFA" and "SEM", demonstrating the effectiveness and compatibility of these components.

4.4 Extended Applications

Beyond optical remote sensing images, we also evaluate the generalizability of our DPU-Former model on natural scene images, camouflage images, and medical images. Specifically, we apply the proposed DPU-Former model to three object segmentation tasks, including salient object detection (SOD), camouflage object detection (COD), and polyp segmentation (PS). From Fig. 7, the proposed DPUFormer method achieves outstanding segmentation accuracy, attributed to the cooperation of both local and global perspectives. More details in supplementary materials.

5 Conclusions

In this paper, we propose the DPU-Former model, a novel framework for the ORSIs object segmentation task. It obtains mutually beneficial features from both local and global perspectives and introduces the FMS to eliminate heterogeneity. Moreover, we present the GLFFN to enhance expression ability. Additionally, within the decoder, we construct the ACFA and SEM to integrate and strengthen multilevel features. As a result, the proposed DPU-Former method achieves outstanding performance on multiple object segmentation datasets.

Acknowledgments

This work was supported in part by the National Science Fund of China (No. 62276135, 62176124, and 62361166670).