SGAT4PASS: Spherical Geometry-Aware Transformer for PAnoramic Semantic Segmentation

1 Introduction

There has been a growing trend of practical applications based on 360◦ cameras in recent years, including holistic sensing in autonomous vehicles [de La Garanderie et al., 2018; Ma et al., 2021; Gao et al., 2022], immersive viewing in augmented reality and virtual reality devices [Xu et al., 2018; Xu et al., 2021; Ai et al., 2022], etc. Panoramic images with an ultra-wide angle of view deliver complete scene perception in many real-world scenarios, thus drawing increasing attention in the research community in computer vision. Panoramic semantic segmentation (PASS) is essential for omnidirectional scene understanding, as it gives pixel-wise analysis for panoramic images and offers a dense prediction technical route acquiring 360◦ perception of surrounding scenes [Yang et al., 2021a].

∗Corresponding author.

(f) Roll angle & Mean mIoU

Figure 1: The results with 3D disturbance input. (a) is the original image, and (b) / (c) is the images rotated 5◦ in pitch / roll axis. Our baseline is Trans4PASS+. Compared with the minor change in images, the huge variance / performance change in SGA validation is shown in (d) / (e) and (f). "Mean" and "Variance" are defined in detail in Section 4.1.

Most existing PASS approaches use equirectangular projection (ERP) [Sun et al., 2021; Yang et al., 2021b] to convert original 360◦ data to 2D panoramic images. However, these methods often suffer from two main problems: large image distortions and lack of Spherical Geometry-Aware (SGA) robustness that resists 3D disturbance. These problems lead neural networks to only learn suboptimal solutions for panoramic segmentation [Yang et al., 2019; Wang et al., 2021a]. Although some recent works [Zhang et al., 2022a; Zhang et al., 2022b] take serious distortions into account in their models and become the current state-of-the-art (SOTA), they still do not pay enough attention to the SGA properties of the original 360◦ data, resulting in performance degradation even with small projection disturbance. As shown in Figure 1b and Figure 1c, applying 5◦ rotation on the pitch or roll axis of original 360◦ data carries only minor changes in 2D panoramic images. However, as shown in Figure 1e, Figure 1f, and Figure 1d, the performances of Trans4PASS+ [Zhang et al., 2022b] (the blue lines) drop a lot (about 4%), and the variance increases by almost 2 orders of magnitude, because the axis rotations lead to different spherical geometry relations between pixels in the projected panoramic images, which the existing methods fail to adapt. Besides disturbance, the ERP also introduces boundaries to panoramic images that the original 360◦ data do not have. Some adjacent pixels are disconnected and some objects are separated, which is a severe issue, especially for semantic segmentation. Furthermore, there also exists a difference in pixel sampling density between the original 360◦ data and its corresponding projection image, e.g., pixels are over sampled in the antarctic and arctic areas of 2D panoramic images. All these issues make panoramic semantic segmentation a challenging task, and the above characteristics should be well studied to design a robust model that adapts to disturbance, disconnection, uneven density, and other SGA properties.

Improving robustness and taking SGA properties into account, we propose a novel model, i.e., Spherical GeometryAware Transformer for PAnoramic Semantic Segmentation (SGAT4PASS), equipped with the SGA framework and SGA validation. The proposed SGA framework includes SGA image projection in the training process, Spherical Deformable Patch Embedding (SDPE), and a panorama-aware loss. SGA image projection provides images with 3D disturbance to improve the 3D robustness of the model. SDPE improves the patch embedding and makes it consider not only the image distortions with deformable operation but also spherical geometry with SGA intra- and inter-offset constraints. The panorama-aware loss deals with the difference in pixel density between the original 360◦ data and its corresponding 2D panoramic images. Moreover, we propose a new validation method, i.e., SGA validation, to evaluate the 3D robustness of various models comprehensively, which considers different 3D disturbances for input images, and measures the average performance and the variance for comparisons. Extensive experimental results on popular Stanford2D3D panoramic datasets [Armeni et al., 2017] demonstrate that our proposed approach achieves about 2% and 6% improvements on traditional metrics and SGA metrics, respectively.

The contributions of this paper are summarized as follows:

• We propose SGAT4PASS, a robustness model for the PASS task, which utilizes SGA image projection to deal with the 3D disturbance issue caused by ERP.
• We introduce SDPE to combine spherical geometry with deformable operation to better deal with panoramic image distortion. And we also propose panorama-aware loss to ease the oversampling problem.
• We evaluate SGAT4PASS on the popular benchmark and perform extensive experiments with both traditional metrics and proposed SGA metrics, which demonstrate the effectiveness of each part of the framework.

2 RelatedWork

The two most related fields are panoramic semantic segmentation and dynamic and deformable vision transformers.

2.1 Panoramic Semantic Segmentation

Semantic segmentation of panoramic images has many applications in real-world scenarios, such as autonomous driving [Ye et al., 2021], panoramic lenses safety and monitoring applications [Poulin-Girard and Thibault, 2012], etc. With the development of deep learning, many neural networks have been developed for panoramic semantic segmentation. Deng et al. [Deng et al., 2017] first proposed a semantic segmentation framework for wide-angle (fish-eye) images and transformed an existing pinhole urban scene segmentation dataset into synthetic datasets. Yang et al. [Yang et al., 2019] designed a semantic segmentation framework for panoramic annular images using a panoramic annular camera with an entire Field of View (FoV) for panoramic surrounding perception based on a single camera. Furthermore, Yang et al. [Yang et al., 2020] proposed DS-PASS to improve it with a more efficient segmentation model with attention connections. PASS solutions can be divided into two main fields: distortionaware strategies and 2D-geometry-aware ones.

For distortion-aware strategies, Tateno et al. [Tateno et al., 2018] proposed using specially designed distortion-aware convolutions in a fixed manner to address image distortions. Furthermore, ACDNet [Zhuang et al., 2022] combined convolution kernels with different dilation rates adaptively and used fusion-driven methods to take advantage of several projections. Jiang et al. [Jiang et al., 2019] designed a spherical convolution operation. Lee et al. [Lee et al., 2018] used spherical polyhedrons to represent panoramic views to minimize the difference in spatial resolution of the surface of the sphere and proposed new convolution and grouping methods for the representation of spherical polyhedrons. Hu et al. [Hu et al., 2022] designed and proposed a distortion convolutional module based on the image principle to solve the distortion problem caused by the distortion of the panoramic image. Zhang et al. [Zhang et al., 2022a] [Zhang et al., 2022b] designed their Trans4PASS and Trans4PASS+ that perceived spherical distortion and solved the distortion problem of spherical images better through their Deformable Patch Embedding (DPE) and Deformable Multi-Layer Perception (DMLP) modules. Also, Trans4PASS+ is the current SOTA panoramic semantic segmentation model and is our baseline. For 2D geometry-aware strategies, horizontal features are mainly used based on the ERP inherent property. Sun et al. [Sun et al., 2021] proposed HoHoNet and Pintore et al. [Pintore et al., 2021] proposed SliceNet to use the extracted feature maps in a 1D horizontal representation.

For our SGAT4PASS based on the distortion-aware SOTA model, Trans4PASS+, we add SGA information from the original 360◦ data instead of the 2D geometry prior to panoramic images to improve not only its performance but also its robustness when meeting 3D disturbance.

2.2 Dynamic and Deformable Vision Transformers

Regarding the field of vision transformers, some works have developed architectures with dynamic properties. Chen et al. [Chen et al., 2021] and Xia et al. [Xia et al., 2022] used deformable designs in later stages of the encoder. Yue et al. [Yue et al., 2021] used a progressive sampling strategy to locate discriminatory regions. Deformable DETR [Zhu et al., 2020] used deformable attention to deal with feature maps. Some other works used adaptive optimization of the number of informative tokens to improve efficiency [Wang et al., 2021b] [Rao et al., 2021] [Yin et al., 2022] [Xu et al., 2022]. Zhang et al. [Zhang et al., 2022a] [Zhang et al., 2022b] designed their Trans4PASS and Trans4PASS+ based on DPE and Deformable Multi-Layer Perception (DMLP) modules, and we use Trans4PASS+ as our baseline.

3 Method

We present Spherical Geometry-Aware Transformer for PAnoramic Semantic Segmentation (SGAT4PASS) in this section. First, we introduce the background of panoramic semantic segmentation in Section 3.1. Second, we describe our main idea to apply different SGA properties in panoramic semantic segmentation in Section 3.2. To improve the 3D

robustness of SGAT4PASS, we propose SGA Image Projection, Spherical Deformable Patch Embedding (SDPE), and panorama-aware loss. Specifically, SGA Image Projection adds rotated samples in training; SDPE adds SGA constraints on the deformable patch embedding; and the panorama-aware loss fuses sphere pixel density to training process.

3.1 Background

We first describe a general formulation of PASS and then introduce the spherical geometry property that we focus mainly on. Panoramic images are based on original 360◦ data formulated in the spherical coordinate system (based on longitude and latitude). To convert it to a rectangular image in a Cartesian coordinate system, ERP is a widely used projection in this field: x = (θ - θ0)cosϕ1, y = (ϕ - ϕ1), where θ0 = 0 is the central latitude and ϕ1 = 0 is the central longitude. The ERP-processed rectangular images are used as the input sample in datasets and fed to the neural network, and the rectangular semantic segmentation results are obtained to compare with the ground truth and calculate the metrics. Although traditional methods can treat PASS as the conventional 2D semantic segmentation task and deal with panoramic images easily, the spherical geometry property is partly ignored.

3.2 Spherical Geometry-Aware (SGA) Framework

We propose the SGA framework for PASS with SGA image projection, SDPE, and panorama-aware loss. To deal with the inevitable 3D disturbance during the acquisition of the input image, our SGA image projection aims to encode the original 360◦ data spherical geometry by generating input images with different rotations. We design SDPE to model spatial dependencies on a sphere, making patch embedding consider both spherical geometry and image distortions. Furthermore, a panorama-aware loss is proposed to model the pixel density of a sphere, making the loss weight distribution more similar to the original 360◦ data. With these three modules, the spherical geometry is well employed in the PASS task.

Spherical Geometry-Aware (SGA) Image Projection

The original 360◦ data follow a spherical distribution and are spherically symmetric. After rotating any angle along the yaw / pitch / roll axis, the transformed data are still equivalent to the original data. Traditional strategies assume that the images are taken with the yaw / pitch / roll angle equal to zero degrees, which is too ideal in real-world scenarios and ignores the camera disturbance and random noise. When the rotation angle is disturbed, traditional strategies usually have a large degradation in the PASS task. SGA image projection fuses this property between the inevitable equirectangular projection and regular image augmentation to make models robust to 3D disturbance.

We use T to represent the forward process of ERP transformation, which is the process of converting spherical coordinates to plane coordinates, and use T -1 to represent the inverse process of ERP that transforms the plane back onto the sphere. Given an ERP-processed input panoramic image, we first transform the image I originally in plane coordinates to spherical coordinates through the inverse ERP process. After that, we use the rotation matrix in the three-dimensional (3D) space to perform a 3D rotation in the spherical coordinate system. For a general rotation in a 3D space, the angles of yaw, pitch, and roll are αuse, βuse, and γuse, respectively. The corresponding rotation matrix is R(αuse, βuse, γuse). We multiply R by the data in the spherical coordinate system to obtain the rotated data in the spherical coordinate system. Finally, we use the ERP forward process to convert the rotated spherical coordinate system image into a panoramic image, thus obtaining a certain rotated image of the real input of the network. The corresponding point in input image of a pixel in rotated image may not have integer coordinates, and we select the nearest pixel as its corresponding pixel to be generic to the ground truth transformation. Based on these operations, we build our SGA image projection, O3D(I, αuse, βuse, γuse) = T (R(αuse, βuse, γuse) · T -1 (I)). (See Section C "Details for SGA Image Projection" in the supplementary material for details.) At the beginning of the training process, we set the maximum rotation angle of the yaw / pitch / roll axis at (αtrain, βtrain, γtrain).

SDPE: Spherical Deformable Patch Embedding

We first introduce DPE, and then fuse spherical geometry into DPE by SGA constraints to earn SDPE.

Faced with image distortions in panoramic images, DPE, considering different distortions in different regions of an input panoramic image, is a popular solution [Zhang et al., 2022a] [Zhang et al., 2022b]. In detail, given a 2D input panoramic image, the standard patch embedding handles it into flattened patches H×W , and the resolution of each patch is (s, s). A learnable projection layer transforms each patch into out-dimensional embeddings. For each patch, the offsets ∆DP E (i,j) of the ith row j th column pixel are defined as:

where g(·) is the offset prediction function. Hyperparameter kD puts an upper bound on the learnable offsets ∆DPE (i,j) . For implementation, the deformable convolution operation [Dai et al., 2017] is popularly employed to realize DPE.

When fusing spherical geometry into DPE, human photographic and ERP priors are taken into consideration, in which the plane formed by pitch and roll axes is always parallel to the ground plane and the projection cylinder is perpendicular to the ground plane. As a result, we add SGA constraints mainly on the yaw axis. In detail, we give intra-offset and inter-offset constraints on ∆DPE (i,j) . For convenience, we use ∆m,n (i,j) to represent the ith row j th column pixel of the learnable offset for the mth row nth column patch.

Intra-offset constraint. Based on the phenomenon that the original 360◦ data are symmetric on any longitude and the projection cylinder in ERP is symmetric in any line perpendicular to the base of the cylinder, the offset of any pixel in 2D input panoramic image I should be symmetric on its perpendicular. To be generic to the learnable offsets ∆m,n (i,j) dealing with the image distortions, we use a constraint Lintra:

where ∆S m,n (i,j) is the single patch offset that is formed symmetrically along the yaw axis with ∆m,n (i,j) as the template. Lintra 2 (·, ·) represents the element-wise L2 loss.

Inter-offset constraint. Based on the phenomenon that the projection cylinder in ERP can be slit and expanded from any line perpendicular to the base of the cylinder, the offset of any pixel in 2D input panoramic image I corresponding to the same latitude of the original 360◦ data should be similar. To be generic to the learnable ∆DPE (i,j) dealing with the image distortions, we use a constraint, Linter, to model this property. For a certain pixel, we use the average offset in the whole horizontal line as its constraint:

where ∆m,AVG (i,j) is the average of each component in {∆m,n (i,j) , n ∈ W}, and Linter 2 (·, ·) represents the L2 loss for each component length of the two vectors. Then the total SDPE loss is: LSDPE = Linter + Lintra.

Panorama-Aware Loss

Because the panoramic images are rectangular in shape, the region of the antarctic and arctic areas in the original 360◦ data is over sampled than the one near the equator. However, due to human photographic priors, the semantics of the antarctic (ground, floor, etc.) and arctic areas (sky, ceiling, etc.) are relatively simple, as seen in the sample images of Figure 1 and Figure 2. When using traditional segmentation loss for supervised training, we treat each pixel equally, which leads to models paying relatively less attention to semantic rich regions near the equator. To deal with this phenomenon, we design our panorama-aware loss. For an ERP-processed panoramic image, the number of pixels in each horizontal line is the same, but the corresponding resolution density on the original sphere of each horizontal line is very different. For this reason, we design a loss to reweight the loss proportion of different horizontal lines depending on its height. For a pixel (m, n)|m ∈ [1,HI], n ∈ [1,WI] (WI and HI are the width and height of the input image), we give Table 4: Effect of each SGAT4PASS module. We validate them on Stanford2D3D Panoramic datasets official fold 1 with traditional metrics. "SGAIP" / "SDPE" / "PA" means our SGA image projection / spherical deformable patch embedding / panorama-aware loss. Using anyone, an average improvement of 1.058% mIoU / 0.365% pixel accuracy is gained when using three gains 2.757% / 1.652%.

a weight w(m,n) pan when calculating its per pixel loss:

We use Wpan to represent the set that includes all w(m,n) pan .

When faced with a panoramic semantic segmentation problem, we first estimate the usage scenario to determine β and γ used in SGA image projection when α is often set as 360◦ in common condition. We set our total loss as:

where LSEG is the common per pixel loss for semantic segmentation, ⊙ is the element-wise matrix multiplication, λw and λs are hyperparameters.

4 Experiments

In this section, we evaluate our SGAT4PASS against the popular benchmark, Stanford2D3D, for both traditional metrics and our SGA validation.

4.1 Datasets and Protocols

We validate SGAT4PASS on Stanford2D3D Panoramic datasets [Armeni et al., 2017]. It has 1,413 panoramas, and

13 semantic classes are labeled, and has 3 official folds, fold 1 / 2 / 3. We follow the report style of previous work [Zhang et al., 2022a] [Zhang et al., 2022b].

Our experiments are conducted with a server with four A100 GPUs. We use Trans4PASS+ [Zhang et al., 2022b] as our baseline and set an initial learning rate of 8e-5, which is scheduled by the poly strategy with 0.9 power over 150 epochs. The optimizer is AdamW [Kingma and Ba, 2015] with epsilon 1e-8, weight decay 1e-4, and batch size is 4 on each GPU. Other settings and hyperparameters are set the same as Trans4PASS+ [Zhang et al., 2022b]. For each input panoramic image I in an iteration, there is a 50% chance of using it directly and the other 50% chance of using it after SGA image projection, O3D(I, αuse, βuse, γuse), where αuse / βuse / γuse uniformly sampled from 0 to αtrain / βtrain / γtrain. We set (βtrain, γtrain, αtrain) = (10◦ , 10◦ , 360◦ ). λw and λs are set as 0.3 and 0.3, respectively.

SGA validation. Most PASS datasets use a unified ERP way to process original 360◦ data, PASS models have the potential to overfit the ERP way, cannot handle 3D disturbance well and have little 3D robustness. To validate the robustness of the PASS models, we propose a novel SGA validation. nα, nβ, and nγ are the number of different angles for the yaw /

pitch / roll axis, respectively, and nα · nβ · nγ different-angle panoramic images for a certain original 360◦ data is earned. Panoramic semantic segmentation models are validated in all nα · nβ · nγ settings, and their statistics are reported as SGA metrics. In our SGA validation, "Mean" means the average of all nα · nβ · nγ traditional results (e.g., mIoU, per pixel accuracy, etc.). "Variance" means the variance of all nα · nβ · nγ results. "Range" means the gap between the maximum and minimum results of all nα ·nβ ·nγ results. Compared to traditional validation, SGA validation avoids models gain performance by fitting the ERP way of datasets and reflects objective 3D robustness. In detail, we assume that the 3D rotation disturbance is at most 5◦ / 5◦ / 360◦ of pitch (β) / roll (γ) / yaw (α) angle. We set nα = 4 (0◦ , 90◦ , 180◦ , 270◦ ), nβ = 2 (0◦ , 5◦ ), and nγ = 2 (0◦ , 5◦ ). We use the mean of them as the final performance and observe the performance difference among them to indicate the 3D robustness of models.

4.2 Performance Comparison

In this part, we first compare several recent SOTA methods with traditional metrics, and then compare the latest SOTA Trans4PASS+ in detail with SGA metrics.

Traditionalmetrics. Comparison results on Stanford2D3D Panoramic datasets with SOTA methods in traditional metrics are shown in Table 1. Following recent work, we report the performance of both official fold 1 and the average performance of all three official folds. From the results, SGAT4PASS outperforms current SOTA models by 2.8% / 1.6%mIoU, respectively, which means that our SGAT4PASS has a considerable performance margin compared to current models with traditional metrics.

SGA metrics. Comparison results on Stanford2D3D Panoramic datasets with our SGA validation metrics are shown in Table 3, and Table 2 is the detailed performance of each situation. For mean mIoU / pixel accuracy, an improvement of nearly 6% / 4% is achieved, respectively. Furthermore, our variance is about 1 100 and our fluctuation range is about 1 10 . These results show that our SGAT4PASS have much better robustness than Trans4PASS+.

4.3 Ablation Study

Effect of three modules in training process. The effectiveness of SGA image projection, SDPE, and panoramaaware loss are studied on Stanford2D3D Panoramic datasets official fold 1 with traditional metrics as shown in Table 4. (a) SGA image projection: Using it alone improves the baseline mIoU / per pixel accuracy by 1.020% / 0.820%. (b) SDPE: Using SDPE alone outperforms the baseline by 0.937% and 0.025% in mIoU and per pixel accuracy. (b) Panorama-aware loss: Using it alone improves the baseline by 1.216% and 0.250% in mIoU and per pixel accuracy.

Effect of SGA validation. We demonstrate the effect of SGA validation, which means a stronger generalizability to resist 3D rotational perturbation. We carried out experiments with two smaller disturbance settings on the pitch and roll axes ((β, γ, α) = (1◦ , 1◦ , 360◦ ) / (0◦ , 0◦ , 360◦ )), which are more favorable settings for Trans4PASS+ [Zhang et al., 2022b], because it is designed for the standard panoramic view image ((βuse, γuse, αuse) = (0◦ , 0◦ , 0◦ )). The overall statistical results are shown in Table 5. For the (β, γ, α) = (1◦ , 1◦ , 360◦ ) setting, an improvement of approximately 2.7 % / 1.7 % is obtained for the mean mIoU / pixel accuracy. Our variance is approximately 1 5 / 1 10 and our fluctuation range is approximately 12 / 13 in mIoU / pixel precision. In (β, γ, α) = (0◦ , 0◦ , 360◦ ) setting, mean mIoU / pixel accuracy gains approximately 2.6% / 1.6% improvement, variances / fluctuation is approximately 1 2 / 2 3 for SGAT4PASS. SGAT4PASS has better robustness even with little 3D perturbations. The detailed performance of these two settings and the performance of several random rotation settings are shown in Section A "Detailed Performance of SGA Validation" in the supplementary material.

4.4 Discussion and Visualizations

Performance of all semantic classes and visualizations. We show the detailed performance of all 13 semantic classes on the Stanford2D3D Panoramic datasets with both traditional and SGA metrics in Table 6, respectively. We focus mainly on the classes with significant performance gaps and mark the gap larger than 5% / 10% as red numbers / bold red numbers, respectively. There is no semantic class for which the baseline is significantly better. From the results, we can learn that the "sofa" and "door" classes improve more. An image with "door" and "sofa" is visualized in Figure 3. Rotation of the pitch / roll / yaw axis is 5◦ / 5◦ / 180◦ . The baseline prediction gap between the original and rotated input is large, which means less robustness. It predicts the door near the right boundary in Figure 3c overall right, but it is totally wrong with rotation in Figure 3g when SGAT4PASS predicts both correct. The baseline predictions for the sofa change a lot with rotation when SGAT4PASS is stable. More visualizations are shown in Section B "More Visualizations" in the supplementary material.

Different hyperparameters. λw and λs are hyperparameters in our SGAT4PASS. λs / λw determines the proportion of our constraint of spherical geometry in SDPE / panoramaaware loss. We apply them on the baseline, respectively. Traditional mIoU results are shown in Figure 4a and Figure 4b, and we choose 0.3 / 0.3 as the final λw / λs.

5 Conclusion

We have studied an underexplored but important field in panoramic semantic segmentation, i.e., the robustness of dealing with 3D disturbance panoramic input images. We have shown that using our SGA framework is key to improving the semantic segmentation quality of 3D disturbance inputs. It applies spherical geometry prior to panoramic semantic segmentation and gains considerable improvement. In detail, the SGA framework includes SGA image projection, SDPE, and panorama-aware loss. We also validated the effectiveness of our SGAT4PASS on popular datasets with the traditional metrics and the proposed SGA metrics, and studied its properties both empirically and theoretically.

Acknowledgements

This work is supported in part by National Key Research and Development Program of China under Grant 2020AAA0107400, National Natural Science Foundation of China under Grant U20A20222, National Science Foundation for Distinguished Young Scholars under Grant 62225605, Research Fund of ARC Lab, Tencent PCG, The Ng Teng Fong Charitable Foundation in the form of ZJU-SUTD IDEA Grant, 188170-11102 as well as CCF-Zhipu AI Large Model Fund (CCF-Zhipu202302).

Contribution Statement

Xuewei Li and Tao Wu contributed equally to this work.