PRNet: Point-Range Fusion Network for Real-Time LiDAR Semantic Segmentation

1 Introduction

LiDAR semantic segmentation provides a crucial point-level perception of the surrounding environments for applications, e.g. autonomous driving and moving robots. Such applications require the LiDAR semantic segmentation model to run in real-time to support timely downstream object detection and planning. However, the LiDAR point clouds come in an unstructured and sparse format, which brings challenges for efficient and effective processing. Different data representations (the 3D point, voxel, and range view) have been used to accomplish this task, which further results in differences in efficiency and accuracy as shown in Figure 1.

∗Equal contribution

†Corresponding author

The point-based methods [Qi et al., 2017a; Qi et al., 2017b; Thomas et al., 2019; Hu et al., 2020] directly consume the unstructured 3D point clouds, which are the most complete sources of 3D information. However, as shown in Figure 1, point-based methods run slowly, since it has to adopt inefficient components for searching neighbors and aggregating features within unstructured points.

The latter voxel-based methods [Choy et al., 2019; Tang et al., 2020; Cheng et al., 2021; Zhang et al., 2020; Zhu et al., 2021] discretize the point clouds into voxel cells and utilize the 2D or 3D convolution networks for feature extraction. These methods achieve relatively higher performance than point-based methods, but their efficiency (speed) still acts as a significant bottleneck for real-world applications.

The range-based methods adopt the most compact representation of the LiDAR point clouds, namely the RV representation, to ensure their efficiency. The previous rangebased methods [Cortinhal et al., 2020; Milioto et al., 2019; Xu et al., 2020; Razani et al., 2021] first project 3D points to the RV, and then perform the 2D semantic segmentation on the RV with the well-optimized 2D convolution networks, and finally transmit the 2D RV labels back to 3D points. Despite their efficiency, their performance is much poorer than the voxel-based counterparts. We analyze two major reasons in Figure 2 and propose to solve these issues as the following.

First, the information loss during the projection from 3D points to the RV is nonnegligible. As shown in Figure 2, multiple LiDAR points may be projected to the same RV pixel, but existing methods only keep the nearest point. For example, SalsaNext [Cortinhal et al., 2020] drops a significant proportion of 22% points on the SemanticKITTI dataset. To better encode the RV features, a novel range view projection (RVP) operation is proposed to aggregate features of the points in the same RV pixel through max-pooling. Moreover, the features of the RV and points are fused for the final perpoint prediction to further minimize the information loss.

Second, the semantic ambiguity results in the wrong predictions, when the RV labels are transmitted back to the 3D points. The RV squeezes the distance r dimension and the 3D points far away in the 3D space may correspond to adjacent or even the same pixel in the RV space, which implies that these 3D points would be probably assigned to the same label. To ameliorate this problem, existing methods adopt kNearest-Neighbor (kNN) as post-processing, voting for the final prediction. But sometimes the kNN also fails, as shown in Figure 2, the occluded vegetation regions are assigned by the label of traffic-sign. To address this issue fundamentally, we propose to transmit the RV features instead of labels back to 3D points and introduce a novel point-range convolution (PRConv), which transmits the RV features with dynamic per-point hyper-parameters and allows distinct 3D point features although they are corresponding to the same RV pixel.

In summary, we propose an end-to-end point-range fusion network (PRNet) for real-time LiDAR semantic segmentation. Different from the previous range-based methods that predict 2D RV semantic labels, the proposed method first projects the features from 3D points to the RV for the latter feature extraction and then fuses the RV features back to

3D points for the final per-point prediction. In this framework, the RVP operation aims to alleviate the information loss during the projection from 3D points to the RV, while the PRConv is proposed to mitigate the semantic ambiguity during the feature transmission from the RV back to 3D points. Experiments on the SemanticKITTI and nuScenes benchmarks show that the proposed PRNet outperforms existing range-based methods by a large margin. Compared with the top-ranking methods, the PRNet achieves a better speed-accuracy trade-off.

2 RelatedWork

Point-based Methods. The point-based methods directly consume raw point clouds without any quantization, but their performance and efficiency are relatively worse. The pioneering work, PointNet [Qi et al., 2017a], adopts the shared multilayer perception (MLP) to extract per-point features and the global max-pooling to integrate global features. To learn richer local structures, many subsequent works have been introduced. PointNet++ [Qi et al., 2017b] proposes the stacked set abstraction layers to learn hierarchical point features. KPConv [Thomas et al., 2019] proposes a novel spatial kernelbased point convolution to extract local structure. However, since the farthest point sampling (FPS) widely used in these networks is inefficient in both computation and memory cost, these methods are limited to indoor scenes and cannot be directly extended to autonomous driving with a large number of 3D points. RandLA-Net [Hu et al., 2020] replaces the FPS with random sampling to improve efficiency, but it is still far away from real-time processing.

Voxel-based Methods. The voxel-based methods quantize the 3D points into structured voxels, where the mature 2D or 3D convolution networks are applied. To reduce the computation and memory cost, sparse convolution [Choy et al., 2019] is applied only on these non-empty voxels. Considering that the voxelization brings quantization errors, the following works [Tang et al., 2020; Cheng et al., 2021; Xu et al., 2021] fuse the voxel features with the fine-grained point features. SPVNAS [Tang et al., 2020] further adopts neural architecture search and achieves better results with lower computation cost. AF2S3Net [Cheng et al., 2021] designs the attentive feature fusion module (AF2M) and adaptive feature selection module (AFSM) to efficiently extract local and global structures simultaneously. Recent RPVNet [Xu et al., 2021] fuses the features from the points, voxel, and RV in a single framework to alleviate quantization errors, and achieves the best results on the SemanticKITTI and nuScenes benchmarks. Besides, different 3D space partition strategies are proposed. PolarNet [Zhang et al., 2020] uses the polar grid representation and Cylinder3D [Zhu et al., 2021] follows the cylindrical partition. Although the voxel-based methods dominate the LiDAR semantic segmentation benchmarks, they cannot run in real-time on mobile platforms.

Range-based Methods. The range-based methods are more attractive because they are fast and easy-deployed by utilizing highly optimized 2D convolutions. These methods first project 3D points to the RV space and then perform semantic segmentation on the RV. RangeNet++ [Milioto et al., 2019] proposes an accelerated kNN as post-processing to deal with the semantic ambiguity from the RV predictions back to 3D points labels. SqueezeSegV3 [Xu et al., 2020] proposes the spatially adaptive convolution to apply different convolution parameters for different locations on the RV. SalsaNext [Cortinhal et al., 2020] designs a novel encoderdecoder network and adopts Lov´asz-Softmax loss that can directly optimize the mean Intersection over Union (mIoU) metric. More recently, Lite-HDSeg [Razani et al., 2021] designs lightweight harmonic dense convolutions for both efficiency and effectiveness.

3 Methodology

To achieve accurate and real-time LiDAR semantic segmentation, the method needs to extract semantic features efficiently while keeping the LiDAR point clouds information as much as possible. Inspired by the recent range-based methods that extract features at a fast speed and the point-based methods that can preserve point information, we propose a novel endto-end point-range fusion network (PRNet). The proposed method extracts semantic features mainly on the RV, and then fuses the features with 3D point features for enhancement, as shown in Figure 3. The PRNet consists of four steps: 1) the range view projection (RVP) projects the input point features to the RV feature maps; 2) a 2D FCN is applied to the 2D RV feature maps to efficiently extract semantic features; 3) the point-range convolution (PRConv) transmits features from the RV back to 3D points; 4) the point fusion (PF) module fuses the features from the RV and the 3D points to ensure complete information. Additionally, the PRNet can be stackedM times to get stronger features. Finally, the features are fed to a single fully connected layer for the final per-point semantic prediction.

In the following subsections, we first illustrate the four components of the proposed PRNet in Section 3.1, 3.2, 3.3 and 3.4. Then, the loss function is shown in Section 3.5.

3.1 Range View Projection

In the range-based methods, the 3D point features are needed to be projected to the 2D RV feature maps. Unlike the previous methods that directly ignore the occluded points, the RVP is proposed to aggregate all point features projected to the same RV pixels by max-pooling. In detail, it first transforms the kth 3D point from the cartesian space p3D k = (xk, yk, zk) to the spherical space psph k = (rk, θk, ϕk) by applying,

where rk, θk, ϕk denote the distance, zenith and azimuth angle respectively. Then, it acquires the corresponding 2D RV coordinates pRV k = (uk, vk) by discretizing θk and ϕk and ignoring rk, as the following,

where f = fup + fdown is the LiDAR vertical field-of-view. W, H are the predefined width and height of the RV feature maps.

After grabbing the coordinates on the RV, the 3D features can be transposed to the corresponding 2D position. However, there may be more than one points that fall in the same RV pixel (h,w), and an aggregation manner needs to be determined. The traditional range-based methods only select the feature from the nearest 3D point, which leads to the above-mentioned information loss. To tackle this problem, the RVP gathers the features F 3D k,c of the 3D points that fall in the pixel (h,w) by max-pooling to form the RV features F RV h,w,c ,

Theoretically, the complete 3D points information can be retained within the RV features F RV by passing information from different points through different feature channels.

3.2 2D Fully Convolutional Network

The 2D FCN is applied to the RV features for semantic feature extraction. Its architecture follows the conventional encoder and decoder structure. The encoder network has multiple down-sampling stages and does not apply down-sampling along the height dimension. The decoder network conducts up-sampling while fusing high-level and low-level feature maps. The detailed architecture is shown in the supplementary material.

3.3 Point-Range Convolution

To avoid semantic ambiguity, unlike the previous range-based methods that directly predict semantic labels on the RV, the proposed method transmits the RV features back to the 3D points and then performs per-point prediction. Considering that different 3D points may correspond to the same 2D pixel, the transmission module must be able to produce dynamic features according to its 3D and 2D positions to avoid ambiguity. Therefore, the PRConv is proposed to fulfill this function by first predicting a set of offsets and weights, and then doing a 3 × 3 convolution on the displaced positions according to the predicted offsets and weights. Intuitively, as shown in Figure 2, to aggregate features for the occluded vegetation points, the kernel position should be adjusted to the nearby non-occluded vegetation regions, and the kernel amplitude should be reduced if the corresponding kernel position falls on the traffic-sign.

Formally, for the kth 3D point, its features F 3D k are acquired by aggregating the RV features F RV within a 3 × 3 window as the following,

where pn ∈ {(-1, -1), (-1, 0), ..., (1, 1)} defines a 3 × 3 window, and wn denotes the convolution parameters applied on the nth location. To enable the PRConv to alleviate the semantic ambiguity, the kernel offset ∆pk,n is used to adjust kernel position, and the kernel weight ∆mk,n is introduced for modulating the kernel amplitude.

As illustrated in Figure 3, both ∆pk,n and ∆mk,n are predicted dynamically according to features from both the RV and the 3D points. Specifically, they are estimated following a three-step manner: 1) a 2D convolution layer is applied on the 2D feature maps F RV to get F RV ′ ; 2) the 3D features F 3D←RV ′ k for the kth point is obtained by bilinear sampling among the 2 × 2 2D neighbours of the corresponding RV position pRV k on the feature map F RV ′ ; 3) a MLP takes a concatenation of the original 3D point features F 3D k and the interpolated features F 3D←RV ′ as its input and outputs 27channel (3 × 3 × 3) vector per 3D point, where the first 18 channels denote the kernel offset ∆pk,n and the remaining 9 channels are activated by a sigmoid function to obtain the kernel weight ∆mk,n.

3.4 Point Fusion

The point fusion (PF) module fuses the features from the RV and the 3D points to enhance per-point features. For efficiency, it consists of a feature concatenation and two MLP layers. The output point features can serve as the input of the next stacked PRNet or the input of the prediction header. If more than one PRNets are stacked, we impose supervision on each to facilitate the training process. For inference, the output of the last PRNet is used for the final prediction.

3.5 Loss Function

The amounts of data for different categories are highly unbalanced in the LiDAR semantic segmentation dataset (e.g. SemanticKITTI, nuScenes). For example, the proportions of road, building, and car are hundreds times than those of motorcyclist and traffic-sign. Therefore, we apply the weighted cross entropy (WCE) Lwce that emphasizes the rare categories as the following,

where ycn (ycn ∈ {0, 1}) and ˆ ycn (ˆ ycn ∈ [0, 1]) are the groundtruth and the predicted probability of the cth class on the nth point. Fc is the frequency, and αc is the weight of the cth class. We also adopt the Lov´asz-Softmax loss [Berman et al., 2018] Lls to directly optimize the mean Intersection over Union (mIoU) metric. According to [Aksoy et al., 2020; Cortinhal et al., 2020], it can improve the mIoU performance of the LiDAR semantic segmentation task. The total loss Ltotal is the sum of the two loss terms and is defined as

4 Experiments

We evaluate the effectiveness and efficiency of the proposed PRNet on the public nuScenes [Caesar et al., 2020] and SemanticKITTI [Behley et al., 2019] single-scan LiDAR semantic segmentation benchmarks.

4.1 Experimental Setup

Datasets. nuScenes for the LiDAR semantic segmentation is a newly released benchmark with 1,000 scenes collected in Boston and Singapore. Each LiDAR scan is collected by a Velodyne HDL-32E 360◦ rotating LiDAR with 32 beams vertically. It splits 28,130 samples for training, 6,019 for validation, and 6,008 for testing. After merging similar categories and removing rare categories, 16 categories are used for the official evaluation.

SemanticKITTI contains 43,552 LiDAR scans from 22 sequences collected in Germany. Each LiDAR scan is collected by a Velodyne HDL-64E 360◦ rotating LiDAR with 64 beams vertically. The training set (19,130 scans) consists of sequences from 00 to 10 except 08, and the sequence 08 (4,071 scans) is used for validation. The rest sequences (20,351 scans) from 11 to 21 are only provided with LiDAR point clouds and are used for the online leaderboards. The dataset is annotated with 28 categories, but it merges categories with different motion states to acquire 19 valid categories for the single-scan LiDAR semantic segmentation benchmark.

Evaluation Metric. To evaluate the proposed PRNet and its competitors, we follow the official metric, namely mean Intersection over Union (mIoU), as the following,

where TPc, FPc, FNc are the true positive, false positive and false negative of the cth category, respectively. C is the total number of classes.

Network Setup. In the experiments, we adopt two stacked PRNets, and each has a similar network architecture but different parameters. The dimensions of the input point features for these two nets are 5, and 64, respectively. The first 5 feature channels contain x, y, z, intensity and r. As shown in Figure 3, the first MLP layer of each PRNet outputs 64 feature channels. The details of the 2D FCN can be seen in the supplementary material. The two stacked PRNets output 64, 96 point feature channels, respectively. Finally, the segmentation results are acquired by applying an FC layer to the output of the last stacked PRNet. Note that PRNet-2C just doubles the RV feature channels of PRNet-1C.

For SemanticKITTI, the parameters of LiDAR vertical field-of-view are set as fup = 3◦ and fdown = -25◦ . The input size of the RV branch is 64 × 2048. For nuScenes, we adopt the same configuration except fup = 20◦ and fdown = -40◦ .

Training Details. All experiments are conducted with PyTorch FP32 on NVIDIA RTX 2080Ti GPU. The proposed PRNet is trained from scratch for 48 epochs with a batch size of 16 on 8 GPUs. Stochastic gradient descent (SGD) serves as the optimizer with a weight decay of 0.001, a momentum of 0.9, and an initial learning rate of 0.02, which is decayed by 0.1 every 10 epochs. Following the convention, the data augmentation strategies include random flipping along the x and y axes, random global scale sampled from [0.95, 1.05], random rotation around the z axis, random Gaussian noise N(0, 0.02), and instance CutMix [Xu et al., 2021].

4.2 Comparisons With the State-of-the-Arts

Results on nuScenes. We report the performance of the proposed PRNet on the newly released nuScenes validation set. As shown in Table 1, our PRNet-2C achieves the best performance among all competing methods, and even outperforms the most competitive RPVNet for most categories. In terms of running time, the PRNet-2C is much faster compared with all methods except the SalsaNext, but SalsaNext is slower and has poorer mIoU performance than the PRNet-1C.

Results on SemanticKITTI. The proposed PRNet is compared with the state-of-the-arts on the SemanticKITTI test set. As shown in Table 2, the methods are grouped as pointbased, voxel-based, and range-based methods from top to bottom. We find that PRNet-2C outperforms all point-based and range-based methods by a large margin, and it can be comparable with the top-ranking voxel-based methods, while it runs much faster. Though the proposed PRNet-2C performs poorer compared with the top-ranking RPVNet on some hard categories, it runs 4 times faster with performance comparable with RPVNet on most categories.

Limitation Discussion. For the categories that have fewer training samples and are easily confused with other categories, the proposed method performs worse than the voxelbased RPVNet, e.g. motorcyclist that is easily confused with bicyclist and motorcycle. The reason is that the proposed range-based method mainly encodes context on the 2D RV, but this can be remedied by introducing images.

4.3 Ablation Studies

We make ablative analyses on the SemanticKITTI validation set to figure out the effectiveness of the proposed components.

Effects of the Framework. This analysis is very important to illustrate the insight of our motivation. It starts with the baseline model, SalsaNext [Cortinhal et al., 2020], which is an open-sourced and top-ranking range-based method. As shown in Table 3, we progressively add the proposed components to the SalsaNext until it becomes the same as the proposed PRNet. It can be discovered that: 1) it achieves +1.2 mIoU gains when the post-processing kNN is replaced by the proposed PRConv (a, b); 2) the RVP outperforms the traditional range projection strategy that only keeps the nearest point within an RV pixel, by +1.0 mIoU (b, c). Based on the official code of SalsaNext, we find that only 78% of the whole 3D points are kept during projection to the RV, while the RVP keeps as much 3D points information as possible; 3) the feature fusion of the RV and the 3D points further improves the mIoU by +0.9 (c, d); 4) our 2D backbone is more efficient than that of SalsaNext (d, e, f); 5) two stacked PRNets perform better than a single one (e, f).

Variants of the PRConv. We set the baseline as the bilinear sampling (BS) that acquires the point features by interpolating the four neighbors of the corresponding RV pixel. Afterwards, we add the kernel offset and the kernel weight one by one to figure out the proposed PRConv. As shown in Table 4, the PRConv without the kernel offset and the kernel weight improves the +0.4 mIOU, compared with the baseline. Moreover, adding the kernel offset into PRConv leads to the biggest mIoU jump of +1.8. Finally, the complete PRConv equipped with the kernel offset and weight outperforms the baseline by +3.0 mIoU.

5 Conclusion

In this paper, we present an end-to-end PRNet for real-time LiDAR semantic segmentation, which leverages the strengths of the point-based and range-based methods. The PRNet extracts features mainly on the RV and produces semantic predictions in the 3D space by fused features originating from both the 3D space and the RV space. To minimize the information loss, we propose the RVP operation for the RV projection. To avoid semantic ambiguity, a novel PRConv is proposed to transmit the RV features back to the 3D points dynamically. Experimental results on the SemanticKITTI and nuScenes benchmarks demonstrate the effectiveness and superiority of the proposed components. The PRNet has resolved two long-plagued issues for the range-based methods and demonstrates the potential of the range-based methods for both effectiveness and efficiency. Future works lie in integrating image information to further improve its performance.