PPTFormer: Pseudo Multi-Perspective Transformer for UAV Segmentation

1 Introduction

Semantic segmentation is a fundamental component of many practical applications in computer vision [Chen et al., 2023; Yang et al., 2024; Yang et al., 2022a; Yang et al., 2022b; Yang et al., 2023; Chen et al., 2024a; Yu et al., 2022a; Yu et al., 2024], such as autonomous driving, video surveillance, and precision agriculture, where the goal is to predict a category label for each pixel in an image. In recent years, a plethora of algorithms and networks [Zhu et al., 2021; Zhu et al., 2024a; Ji et al., 2023c; Ji et al., 2022; Zhu et al., 2023c; Zhu et al.,

∗Corresponding author.

2023b] have been developed for segmentation in conventional scenes, showcasing significant advancements in the field.

With the burgeoning growth of Unmanned Aerial Vehicles (UAVs), UAV segmentation has emerged as a pivotal area of research, playing a crucial role in applications ranging from environmental monitoring to disaster response. Unlike data captured from fixed perspectives, UAVs operate at varying altitudes and angles, offering a wealth of dynamic viewpoints. The images captured by UAVs reflect the rich and varied visual information from different angles and altitudes, which is of great importance for monitoring rapidly changing environments.

Addressing the segmentation challenges posed by such rich variability in perspectives necessitates a novel approach. The most intuitive solution would be to collect actual multiperspective data for training segmentation networks. However, as analyzed in [Ji et al., 2024b; Zheng et al., 2020; Yang and Ma, 2022; Ji et al., 2024a], due to the prohibitive costs of collection and fine-grained annotation, existing datasets lack multi-perspective images with detailed labeling. Conventional segmentation methods typically rely on explicit perspective transformation-based data augmentation techniques, such as scaling, rotating, or flipping images in 2D or 3D dimensions. These rudimentary methods of augmentation produce stiff and unnatural perspectives that fail to represent the true changes in perspective experienced during UAV flight, resulting in limited model performance in real-world UAV scenarios.

In light of these challenges and the rapid development of Vision Transformers in semantic segmentation, a subset of methodologies [Xie et al., 2021; Zheng et al., 2021] has been applied to UAV scene segmentation. However, these Transformer networks have not deeply analyzed or been designed with UAV perspective dynamics in mind. The DLPL [Ji et al., 2024b] introduces a comprehensive approach for latent perspective learning. However, DLPL employs a more intricate method involving discrete perspective decomposition and statistical representation, and relies on Homography transformations for converting latent perspectives. Consequently, this framework requires a slightly larger number of parameters and incurs higher latency. In this paper, we adopt the theoretical foundation of DLPL to propose a more lightweight and innovative network, termed PPTFormer (Pseudo MultiPerspective Transformer), specifically tailored for UAV segmentation tasks. The core of this network is composed of advanced PPTFormer blocks designed for pseudo multiperspective learning. These blocks utilize perspective prototypes that remain consistent throughout the network, thereby enhancing perspective-aware learning. Drawing inspiration from [Ji et al., 2024b], a critical component of the PPTFormer Blocks is the perspective representation and generation module. This module modifies visual features to emulate various UAV viewpoints while maintaining the integrity of scene semantics. Distinct from DLPL, PPTFormer utilizes a more lightweight contourlet transformation for texture extraction, followed by keypoint-based perspective description. To further reduce network complexity, we employ a non-parametric method for perspective prototype construction based on linear moving averages, coupled with a simplified Gaussian mixture distribution for generating new pseudo perspectives efficiently. We also utilize a more efficient all-MLP visual decoder instead of the complex architecture in DLPL. Moreover, to address domain shift in the multi-perspective learning process, a challenge which DLPL has overlooked, we improve the sequential attention mechanism (PIA in DLPL) with the iterative Pseudo Multi-Perspective (PMP) Attention, as well as utilize a lightweight Perspective Calibration process following the PPTFormer Blocks.

Our contributions can be summarized as follows:

• We propose a lightweight PPTFormer that enables implicit multi-perspective learning in the absence of multiperspective datasets. By generating pseudo multiperspective characterization about the scene and engaging in joint learning across them, PPTFormer can effectively simulate the varying viewpoints encountered during actual UAV flight, thereby improving the segmentation accuracy.
• Particularly, we employ contourlet transformation and keypoint-based perspective description, coupled with non-parametric perspective prototype construction. Moreover, a perspective calibration process is utilized to address domain shift in multi-perspective learning.
• PPTFormer achieves competitive performance across five UAV segmentation datasets while maintaining a lightweight network, demonstrating its effectiveness in capturing the intricate dynamics of UAV-captured scenes through pseudo multi-perspective Learning.

2 RelatedWork

2.1 Semantic Segmentation

Semantic Segmentation has been a classical and fundamental task in computer vision area [Chen et al., 2024b; Ji et al., 2024a; Ji et al., 2020; Zhu et al., 2024b; Wang et al., 2021b; Wang et al., 2021a; Feng et al., 2018; Ji et al., 2019; Ji et al., 2023a; Zhu et al., 2023e; Zhu et al., 2023f; Zhu et al., 2023d; Yu et al., 2023]. In recent years, the majority of segmentation advancements are grounded in the use of fully convolutional networks (FCN) [Zhu et al., 2023c; Zhu et al., 2023b; Long et al., 2015; Zheng et al., 2023a; Zheng et al., 2023b; Yu et al., 2022b; Zhou et al., 2023; Yu et al., 2023]. Subsequent research has concentrated on capturing contextual relationships within images, employing sophisticated network architectures to enhance the understanding of scene composition [Hu et al., 2020; Ji et al., 2022; Zhu et al., 2023a].

2.2 UAV Scene Segmentation

Despite these advancements, there is a noticeable gap in the literature pertaining to semantic segmentation tailored for UAV imagery. Existing UAV segmentation methods mainly focus on solving the class imbalanced problems, such as SCO [Yang and Ma, 2022], FarSeg [Zheng et al., 2020] and PointFlow [Li et al., 2021]. The unique challenges posed by UAV scenes, characterized by diverse and dynamic changes in perspective, are seldom addressed. The development of segmentation models that can effectively handle the variability inherent in UAV-captured images also remains an area in need of further exploration.

2.3 Ultra-High Resolution Segmentation

In comparison to natural scene imagery, images captured from Unmanned Aerial Vehicles (UAVs) generally exhibit higher resolution characteristics. Existing research has also attempted to introduce segmentation algorithms that can concurrently balance accuracy and efficiency, such as WSDNet [Ji et al., 2023c], GPWFormer [Ji et al., 2023b], among others. In this paper, we endeavor to enhance segmentation precision from the perspective of UAV flight viewpoints. This approach is universally applicable and can be integrated with the aforementioned high-resolution image segmentation algorithms.

3 PPTFormer

3.1 Overall Structure

The overall architecture of our proposed Pseudo MultiPerspective Transformer (PPTFormer) is inspired by [Ji et al., 2024b] and depicted in Figure 1. Following the classic paradigms [Zheng et al., 2021; Xie et al., 2021], PPTFormer consists of a meticulously designed encoder for perspective learning and a generic decoder. The encoder comprises four Transformer Blocks: one Plain Transformer Block followed by three PPTFormer Blocks. The former is responsible for extracting basic low-level information, which serves as the foundation for pseudo multi-perspective learning in the subsequent blocks. Unlike [Ji et al., 2024b] which applies DLPL at every Transformer Block, we restrict perspective learning to only the last three Blocks in PPTFormer to enhance its efficiency. Specifically, within the PPTFormer Blocks, we extract implicit perspective representations from the visual features based on the structural texture. In conjunction with training across the entire dataset, we create perspective prototypes of the images present throughout the dataset. These prototypes are shared across the three PPTFormer Blocks to ensure a consistent learning of perspectives during the training process. Interleaved between the PPTFormer Blocks are Perspective Calibration modules, which are instrumental in aligning the visually fused features from pseudo perspectives with the original perspective of the image. This alignment prevents potential scene domain shifts, which are overlooked in DLPL [Ji et al., 2024b]. Finally, we concatenate features of varying scales produced by each block and feed them into the decoder network for further processing.

3.2 PPTFormer Block

As shown in Figure 1, the PPTFormer Block comprises an Efficient Perspective Generation module and M layers of Pseudo Multi-Perspective Attention (PMP Attention). Given the input of low-level visual features F from block 1, the Efficient Perspective Generation module implicitly represents the image's perspective p, generating a pseudo perspective p′ that simulates the movement and shift of viewpoints during an actual UAV flight, all while preserving the semantic information of the scene. During this process, the acquired perspective representation p contributes to the construction of perspective prototypes P for the entire dataset and also bases the pseudo perspective generation on these prototypes. The output visual feature F ′ with the pseudo perspective p′ , along with F, are both fed into the M layers of PMP Attention for multi-perspective correlation. This allows the model to understand the scene's semantic information from both the original perspective and the new pseudo perspective simultaneously. Specifically, in the first layer of PMP Attention, the inputs are F and F ′ , and the output is the first level of perspective fusion. Subsequently, in the followingM -1 layers of PMP Attention, the fused feature and F ′ work together to achieve the subsequent M - 1 levels of perspective fusion. Below, we will introduce the specific structures of the Efficient Perspective Generation and PMP Attention in detail.

3.3 Efficient Perspective Generation

As illustrated in Figure 2, the input is the visual feature F form Block 1, which first passes through a Perspective Representation encoder Ep to obtain an original Perspective p. Subsequently, on one hand, p contributes to the construction of the entire dataset's perspective prototypes P using the online sequential clustering updating technique. The length of P, which corresponds to the number of perspective prototypes in the dataset, is N. On the other hand, p is also used for Pseudo Perspective Generation based on P , to generate a new pseudo perspective p′ . Thereafter, a Visual Reconstruction decoder Dp uses p′ to ultimately reconstruct the visual feature F ′ . In order to reduce the complexity, Dp is all-MLP architecture which demonstrates promising performance here. During training, to ensure its reconstructive capability, p is also directly fed into Dp with the aim of restoring the original visual feature F, that is,

where Lrec is the reconstruction loss.

Perspective Representation

As depicted in Figure 3, distinct from DLPL, PPTFormer utilizes a more lightweight contourlet transformation for texture extraction, followed by keypoint-based perspective description. The Perspective Representation encoder Ep primarily encompasses two processes: extracting low-level structural texture from the image using contourlet decomposition [Do and Vetterli, 2005] and, based on this texture, extracting interest super points that are related to the image's perspective. The former ensures that the model captures the global structural texture information, which can represent the image's contours, edges, and other structural features, including perspective information. To further distill perspective-related features, the latter identifies key support points representing perspective as super points, whose spatial distribution and feature intensity can construct a structured high-dimensional description of the image's perspective.

Texture Extraction. Specifically, as traditional filters, contourlet decompositions inherently excel at texture representation across various geometric scales and directions in the spectral domain. Rather than describing texture features in the spatial domain, they analyze the energy distribution in the spectral domain to extract the inherent geometric structures of the texture, which naturally includes the image's perspective.

⋮ 𝑃𝑃𝑁𝑁 The contourlet decomposition comprises a cascaded Laplacian Pyramid (LP) [Burt and Adelson, 1983] and a directional filter bank (DFB) [Bamberger and Smith, 1992]. The LP decomposes input features into low-pass and high-pass subbands using pyramidal filters. The high-pass subband is processed through the DFB, which is employed to reconstruct the original signal with minimal sample representation, produced by t-level binary tree decomposition in the two-dimensional frequency domain, resulting in 2z directional subbands. For instance, when z = 3, the frequency domain is divided into 8 directional subbands, with subbands 0-3 and 4-7 corresponding to vertical and horizontal details, respectively [Ji et al., 2022]. Following [Ji et al., 2022], for a richer expression, we stack multiple contourlet decomposition layers iteratively, and concatenate the output of each level to form the final extracted structural texture.

Specifically, the output of level t ∈ [1, T] is denoted as

Fbds,t ,

where l and h represent the low-pass and high-pass subbands respectively, bds denotes the bandpass directional subbands. Then structural texture Ftexture is denoted as:

where Cat is the concatenation operation. Ftexture is rich in texture information including the image's perspective.

Perspective Support Description. Based on Ftexture, we use a SuperPoint network to extract key support points and corresponding support descriptors that are capable of characterizing the perspective from the texture features. Specifically, this network comprises two parallel heads, Ssp and Ssd, which respectively output the "point-ness" probability map and the corresponding point feature descriptor. The final output, the perspective feature p, is the concatenation of output features from the two heads, along the channel dimension:

Perspective Prototypes Construction

The construction of Perspective Prototypes is aimed to obtain and manage the scene perspective types of the whole dataset, by performing an online sequential clustering process on the coming ps. We utilize a lightweight memory bank and its length N is equal to the number of prototypes. Specifically, different from DLPL [Ji et al., 2024b], we employ a non-parametric mechanism to reduce the network complexity. Firstly, the N prototypes P = {P1, P2, ..., Pn, ..., PN} are initialized with the first input N ps, and we set the counts {c1, c2, ..., cn, ..., cN} to record the number of perspective features belonging to the corresponding prototype. Then, for each new coming p, we find its closest prototype Pn by L2 distances, and update the the prototype with:

So the final resulting prototype Pn is the moving average of ps that are closest to Pn.

Then we formulate overall perspective distribution of the whole dataset in form of Gaussian Mixed Model (GMM), following DLPL,

where N(·) indicates the Gaussian Distribution, the nth component of GMM has the center of Pn with the variance of Σn, and πn is the mixture coefficient of meets:

Pseudo Perspective Generation

Based on the dynamically updated perspective distribution G(P), we can generate a new semantic-related pseudo perspective p′ of the given probe p, by leveraging all the prototypes. Note that we avoid using Homography transformations in DLPL to accelerate the generating process.

where p′ is generated based on the overall perspective distribution over all perspective prototypes.

Finally, the corresponding visual feature F ′ for p′ can be reconstructed with Dp.

3.4 PseudoMulti-Perspective Attention

By the Efficient Perspective Generation, we obtain the visual feature F ′ with a new pseudo perspective for the input image. As seen that F and F ′ contain closely identical scene context and structured information but only differ in perspective. Next, they are fed into the M layers of Pseudo MultiPerspective (PMP) Attention to leverage the relationship between the F (with original perspective p) and F ′ (with generated pseudo perspective p′ ). Different from DLPL which use sequential attention mechanism, the PMP Attention employs

where C is the feature channel. By the iterative attention calculation, the multiple perspectives can be fully correlated.

3.5 Perspective Calibration

Within each PPTFormer Block, after undergoing N layers of PMP Attention, the original perspective and the pseudo perspective are thoroughly fused multiple times, ultimately enabling the model to capture scene information as observed from various perspectives. However, in practice, we observed that as the perspective fusion progresses, there could be some domain shift within the visual features' depiction of the scene. To prevent such occurrences, we further incorporate a straightforward Perspective Calibration process after PPTFormer Blocks. Specifically, this entails passing the visual feature with the original perspective, which is the input to the current PPTFormer Block, through a skip connection to calibrate the fused feature output by the current PPTFormer Block, by several layers of PMP Attention. In practice, we found this elegant approach to be effective in mitigating issues of domain shift.

3.6 Optimization

The overall loss function L is the combination of the main segmentation loss Lseg and the reconstruction loss Lrec:

where λ is the weight for Lrec, and set to 0.4.

4 Experiments

4.1 Datasets and EvaluationMetrics

In our experiments, we validate the effectiveness of PPTFormer on five datasets, including UDD6, iSAID, UAVid , Aeroscapes, and DroneSeg.

UDD6

Urban Drone Dataset (UDD) dataset is collected by a DJIPhantom 4 UAV at altitudes between 60 and 100 meters, and is extracted from 10 video sequences. The resolution is either 4k (4096×2160) or 12M (4000×3000). It contains a variety of urban scenes.

iSAID

iSAID totally consists of 2,806 images, where 1411, 458, and 937 images are for training, validation, and testing sets, respectively.

UAVid

UAVid dataset has 300 images of size of 3840×2160, where the training, validation, and testing set contains 200, 70, and 30 images respectively.

Aeroscapes

The Aeroscapes dataset provides 3,269 720p images and ground-truth masks for 11 categories, where the training and validation sets include 2,621 and 648 images respectively.

DroneSeg

The DroneSeg dataset [Ji et al., 2024b] extends the segmentation annotations from VisDrone dataset. The dataset consists of 10,209 images with fine-grained pixel-level annotations of 14 categories.

4.2 Implementation Details

In our experiments, we adopt the MMSegmentation toolbox as codebase and follow the default augments without bells and whistles. SuperPoint network is used for the perspective support description. To ensure training stability, during the initial 30% of epochs, we replace the PMP Attention with plain Self-Attention. This substitution aims to guarantee the reliability of perspective representation and reconstruction within Ep and Dp, as well as the stability of learning perspective prototypes. Subsequently, we revert to the PMP Attention mechanism to perform global joint optimization in the remaining epochs. In the training, SGD optimizer with momentum 0.98 for all parameters is used, the initial learning rate is configured as 5 ×10-3 and the maximum iteration number is set to 160K for all datasets. In Eq. 2, T is set to 2. The length of P is set to N = 64.

4.3 Comparison with State-of-the-Arts

We compare PPTFormer with both representative CNN-based (FarSeg [Zheng et al., 2020], FarSeg++ [Zheng et al., 2020], PFNet [Li et al., 2021] and SCO [Yang and Ma, 2022]) and ViT-based (SETR [Zheng et al., 2021], UperNet [Liu et al., 2021], PoolFormer [Yu et al., 2022c], SegFormer [Xie et al., 2021]) segmentation methods on five benchmark datasets.

UDD6, UAVid

Both the two datasets contain relative fewer images and lower scene complexity and we compare their results here. For fair comparisons with ViT-based methods, we adopt large backbones for CNN-based methods including ResNet101 and HRNet-W48. As shown in Table 1, the ordinary transformer (SETR) show even lower performance than CNN-based methods, and the advanced ones including PoolFormer, SegFormer shows better results. The proposed PPTFormer achieves further performance improvements on both the datasets.

iSAID, Aeroscapes, DroneSeg

These three datasets consist of more images than UDD6 and UAVid, and have higher scene perspective variances. So they would be more convincing to prove the superiority. PPTFormer outperforms other methods by a larger margin, which demonstrates the effectiveness of the proposed method on the description of perspective information.

4.4 Ablation Study

All ablation studies are performed on DroneSeg testing set, SegFormer is used as baseline network.

Comparison with Perspective Learning Methods

Given that PPTFormer is a perspective-oriented learning approach, we begin by comparing it with various perspectivebased augmentations. We observe that in UAV scenarios, perspective shifts are almost invariably linked to changes in altitude and angular positioning, which manifest as alterations in scale and rotation. Therefore, we employ a combination of these two data augmentation techniques to benchmark against PPTFormer. As illustrated in Table 2, we discover that utilizing either augmentation method in isolation yields only modest enhancements over the baseline approach. In contrast, PPTFormer secures a substantial increase in performance. We further demonstrate that PPTFormer remains compatible with standard data augmentations for additional gains.

The Impact of Contourlet Decomposition

The contourlet decomposition is capable of extracting structural texture information from images, which encompasses a wealth of perspective details. By initially employing it within the Perspective Representation, the network can swiftly focus on shallow image textures, thereby facilitating further extraction of Perspective and enhancing learning efficiency. Table 3 demonstrates its efficacy, as the number of contourlet decomposition layers increases, the mIoU correspondingly improves. Here, a layer count of zero indicates no application of contourlet decomposition.

Effectiveness of Perspective Calibration

The purpose of Perspective Calibration is to prevent the occurrence of scene domain shift that may arise as a consequence of deep perspective fusion. Figure 4 illustrates the impact of the number of PMP Attention layers in it on model performance. "Layer=0" implies the absence of Calibration, and the results indicate a low mIoU under this condition. As the number of layers increases, there is a significant improvement in mIoU, which underscores the effectiveness and necessity of Perspective Calibration.

The Quantity of Perspective Prototypes

The quantity of perspective prototypes represents the entirety of perspective variations found within the dataset, with a higher count enabling the retention of a more extensive set of prototypes. Figure 5 reveals that with a smaller allocation of prototypes (16, 32), the process fails to exhaustively capture all perspectives, resulting in an underfitting of the model. Conversely, as the number of prototypes increases (128, 256), we generate an overly dense array of prototypes. This surplus can introduce redundancy and give rise to numerous discrete perspectives, potentially hindering the learning process.

Efficiency Analysis

PPTFormer, inspired by the theoretical framework of DLPL, exhibits a more lightweight architecture and enhanced efficiency. In Table 4, we present a comparative analysis of the parameter counts between PPTFormer and DLPL (based on SegFormer). The comparison indicates that PPTFormer introduces only a slight increase in parameters over the baseline SegFormer, yet maintains a lower parameter count than DLPL. This demonstrates that PPTFormer not only achieves high accuracy but also sustains a high level of efficiency.

0

1

2

3

4

5

6

7

Layer Number in Perspective Calibration

Figure 4: Layer number in perspective calibration.

5 Conclusion

This paper presents the novel PPTFormer, a Pseudo MultiPerspective Transformer network for UAV scene segmentation. It addresses the challenges of capturing the dynamic perspectives inherent in UAV-captured imagery. By integrating systematic Pseudo Multi-Perspective Learning within the Transformer framework, PPTFormer adeptly performs efficient pseudo perspective generation and achieves multiperspective correlation, leading to a more nuanced understanding of UAV scenes. The experiments on several datasets validate the superior performance and efficiency of PPTFormer. The significant advancements made by PPTFormer underscore the importance of perspective-oriented learning in semantic segmentation and pave the way for further innovation in the processing of UAV-captured visual data.

Acknowledgments

This work was supported by the Anhui Provincial Natural Science Foundation under Grant 2108085UD12, the National Key R&D Program of China under Grant 2020AAA0103902, NSFC (No. 62176155), Shanghai Municipal Science and Technology Major Project, China (2021SHZDZX0102). We acknowledge the support of GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC.

Contribution Statement

The first two authors contribute equally to this work.

mIoU

mIoU