APR: Online Distant Point Cloud Registration through Aggregated Point Cloud Reconstruction

1 Introduction

As LiDAR sensors have a precise and accurate 360° view, they are installed on new vehicle models for obstacle detection and avoidance to navigate safely. It is of great interest to share and align outdoor point clouds among neighboring vehicles via broadband wireless communication, which can greatly extend the visual field and improve the point density on distinct objects. Because vehicles may be distant (e.g. 20 to 50 meters apart), the corresponding point clouds are rather different in terms of point density and point of view about the same object in the scene. For example, Figure 1 (a) illustrates points about a target vehicle extracted from two nicely aligned point clouds obtained from two vehicles only 20 meters apart 1 , respectively. It can be seen that they have quite distinct perspectives about the target (i.e. blue and orange points are obtained from the side front and the rear perspectives, respectively). Despite such disparity of distant point clouds, if they can be well aligned, it would certainly enhance various downstream tasks such as object detection and semantic segmentation.

∗Corresponding author

1 Two point clouds separated by 20.48 meters are picked from the KITTI dataset for illustration without loss of generality.

One practical online distant point cloud registration scheme for moving vehicles has to meet the following three requirements: 1) it has to be able to deal with the density and view disparity of such point clouds; 2) it should be cost efficient with respect to both computation and storage overhead for online inference; 3) it has to achieve superior accuracy as the result is crucial to autonomous or assisted driving decision and driving safety applications. For instance, it is extremely critical for driving safety applications to adopt a rigid registration criterion, e.g., 0.5° of rotation error and 0.5 meters of translation error (corresponding to a maximum translation error of less than one meter at all locations within 50 meters).

In the literature, most point cloud registration schemes target at close point clouds (e.g., 10 meters apart), which have similar point density and share overlapped sensor perspective on target objects. Among these methods, feature extractors [Choy et al., 2019; Bai et al., 2020; Ao et al., 2021; Huang et al., 2021; Poiesi and Boscaini, 2021] aim at improving feature quality. Outlier rejection methods [Bai et al., 2021; Pais et al., 2020; Choy et al., 2020] try to identify false correspondences. End-to-end methods in turn create new registration pipelines. These methods have made great effort in merging of traditional methods [Aoki et al., 2019; Yew and Lee, 2020], feature matching procedure [Weixin et al., 2019; Li et al., 2020; Sarode et al., 2019], using transformer to extract contextual information [Wang and Solomon, 2019a; Wang and Solomon, 2019b; Ali et al., 2021], and performing staged registration based on local patch similarity assumption [Lu et al., 2021; Yu et al., 2021; Qin et al., 2022]. The accuracy when directly applying these methods to large-scale outdoor distant point clouds, however, is unsatisfactory. For example, Figure 1 (b)-(d) depict the results of SOTA point cloud registration methods on the example two distant point clouds shown in Figure 1 (a). As a result, to the best of our knowledge, there is no existing scheme successfully addressing the online distant point cloud registration problem.

In this paper, we propose a novel feature extraction framework, called Aggregated Point Cloud Reconstruction (APR), for online distant point cloud registration on moving vehicles. To defeat the density and view disparity of distant outdoor point clouds, it would be ideal if both vehicles have a full view and a better understanding about the same environment. Inspired by this insight, the main idea of APR is to train a powerful feature extractor by embedding a representation of a denser aggregated point cloud with a complete view about the current environment into the features, referred to as APR features. Then, such features of two corresponding distant point clouds can be used for registration.

The APR design faces two main challenges as follows. First, training such a powerful feature extractor to contain a full view of the environment is non-trivial. In APR design, the extractor is trained via an autoencoder structure. Specifically, the encoder can be a state-of-the-art feature extraction backbone (e.g., FCGF or Predator), and metric learning loss is applied to the extracted features, making APR features preferable for registration based on feature similarity. Furthermore, the decoder decodes the feature map of a single point cloud and derives a reconstructed point cloud, which is compared against the aggregated point cloud (APC), defined as a series of point cloud frames of the corresponding vehicle aligned together. As a result, the capability of the encoder to guess a denser geometry is enhanced, so that APR feature contains rich environment information. Particularly, for memory-heavy backbones such as Predator, the encoder and decoder are asymmetrical to avoid out-of-memory (OOM) issue during training.

Second, it is challenging to perform online distant point cloud registration on two moving vehicles as it requires the registration algorithm to be not only accurate but fast. Given a consecutive series of point cloud frames collected on each vehicle, a straightforward method is to take multiple frames from each vehicle as input to perform registration, which would incur heavy memory and computation costs since SLAM or multi-way registration is needed to register consecutive sweeps. In contrast, in our design, as the encoder has the capability to guess the features of APC, only two point cloud frames of a pair of vehicles are used to perform online pairwise registration. Point cloud series of each vehicle are only used to generate aggregated point clouds during offline training.

We implement APR on previous SOTA feature extractors, i.e., FCGF [Choy et al., 2019] and Predator [Huang et al., 2021]. We distill two low-overlap point cloud datasets, i.e., LoKITTI and LoNuScenes, with ≤ 30% overlap from KITTI and nuScenes and conduct extensive experiments. Results demonstrate that APR can effectively improve the performance of previous feature extractors, granting an increase of registration recall (RR) by 2.6% on LoKITTI and 5.2% on LoNuScenes when registering two point cloud frames ranging from 5 meters to 50 meters apart.

We highlight our main contributions made in this paper as follows:

• We propose an autoencoder design as a feature extraction framework, where the descriptiveness of the encoder could be enhanced despite the variety of decoder design, which is either symmetrical or asymmetrical.
• We introduce a new type of point cloud reconstruction target, where instead of using the input frame, we use several frames in vicinity to describe the environment in different views and densities, effectively confronting density variation and view disparity.
• We conduct extensive experiments and results demonstrate that APR achieves SOTA performance for distant point cloud registration.

2 RelatedWork

In this section, we first discuss traditional and learning-based feature extractors, which are closely related to our feature extractors; Then we move on to end-to-end methods and reconstruction-based methods to introduce recent progress on registration pipelines.

2.1 Traditional Feature Extractors

Traditional methods [Johnson and Hebert, 1999; Rusu et al., 2009; Tombari et al., 2010] represent earlier exploration of feature matching based on local shapes. SpinImages (SIs) [Johnson and Hebert, 1999] matches point clouds based on projection images; FPFH [Rusu et al., 2009] extracts rotationinvariant histograms of local geometries; SHOT [Tombari et al., 2010] functions by combining loca reference frames (LRFs) with geometry histograms. However, traditional methods usually ask for surface normal, which is hard to obtain in real-time scenarios, and are easily outperformed by learning methods due to their limited discriminative power.

2.2 Learning-based Feature Extractors

Patch-based LearningMethods. The pioneering work for patch-based learning methods is 3DMatch [Zeng et al., 2017], which applies 3D convolutions on local areas to extract local features for registration. PPF-Net [Deng et al., 2018] utilizes PointNet [Qi et al., 2017] to extract robust point-pair features. PerfectMatch [Gojcic et al., 2019] uses smoothed density value (SDV) to further improve feature robustness. Recent progress include DIP [Poiesi and Boscaini, 2021], which uses a combinational loss of chamfer loss and hardest-contrastive loss [Choy et al., 2019]; SpinNet [Ao et al., 2021] combines LRF with SO(2) invariant convolution to achieve better rotation invariance. However, these methods generally have limited receptive field due to the use of local patches, and high computation time due to repeated local patch computation. They generally don't satisfy the real-time requirement for our problem.

Fully Convolutional Methods. In order to naturally incorporate global and local information, fully convolutional methods are proposed. FCGF [Choy et al., 2019] first applies metric learning to dense convolutional features, achieving SOTA performance while being orders of magnitude faster than patch-based methods. D3Feat [Bai et al., 2020] improves upon FCGF through adopting KPConv [Thomas et al., 2019] and training a joint extractor-detector backbone. Predator [Huang et al., 2021] further approaches the low-overlap problem using overlap attention module at bottleneck, achieving SOTA performance. We implement our method based on these methods, as their fast and robust nature perfectly matches our requirements for a backbone.

2.3 End-to-end Registration

End-to-end registration methods generally modify components of the traditional registration pipeline, converting the pipeline into a network that can be trained in an end-to-end manner.

Superpoint MatchingMethods. Staged registration based on superpoints have seen success on high-overlap point clouds [Lu et al., 2021; Yu et al., 2021; Qin et al., 2022], where registration is first performed on downsampled point cloud (i.e., superpoints), then cascaded onto denser point cloud patches around the matched superpoints. However, these methods generally struggle to register distant point clouds, since the patch similarity assumption is broken under huge density variance and view disparity.

OtherMethods. Some methods such as PointNetLK [Aoki et al., 2019] and RPM-Net [Yew and Lee, 2020] enhance optimization algorithms [Lucas and Kanade, 1981; Gold et al., 1998] with features extracted from PointNet [Qi et al., 2017]. Some methods such as DeepICP [Weixin et al., 2019] and IDAM [Li et al., 2020] refine the feature matching process in a learnable manner, reducing false positives. Other pipelines, including DCP [Wang and Solomon, 2019a], PRNet [Wang and Solomon, 2019b] and RPSRNet [Ali et al., 2021], use transformers to grab the contextual information between features. However, they are generally dedicated to indoor or synthetic point cloud datasets, and are generally inable to scale up to the level of outdoor LiDAR point clouds.

2.4 Reconstruction-based Registration

We are aware of other methods that are also trying to merge chamfer loss and reconstruction into registration pipelines.

Feature-metric Registration [Huang et al., 2020] uses the reconstruction of the input frame, which basically asks for no information loss. DIP [Poiesi and Boscaini, 2021] uses chamfer loss only for aligning two local patches. Compared to their focus on features rather than geometries, our decoder network directly learns the offset bias of the new points from features, focusing more on reconstructing local geometry.

3 ProblemModeling

3.1 SystemModel

We consider conducting online distant point cloud registration on moving vehicles. Each vehicle is equipped with a LiDAR sensor and can continuously generate a time series of point cloud frames (e.g., ten point cloud frames per second) while moving. In addition, neighboring vehicles can efficiently exchange point cloud data in real time via broadband wireless communication techniques [Wang et al., 2020; Perfecto et al., 2017]. Vehicles have sufficient amount of memory for point cloud data storage but a restricted computational capability with an onboard embedded system. During offline training of our scheme, the rough distance estimation between a pair of point cloud frames in the point cloud series is required. It should be noted that, during online registration, we do not require any side channel information about how vehicles move via GPS or inertial sensors (e.g., accelerometers and gyroscopes).

3.2 ProblemDefinition

We denote the sequence {Xvi 1 ,Xvi 2 , . . . ,X vi t } as the time series of t point cloud frames of vehicle vi, where Xvi k = {pi ∈ R3 |i = 1, 2, ...,N} for k ∈ [1, t] is the k-th point cloud frame of N points. For a pair of vehicles v1 and v2 to register their current point cloud frame, e.g., Xv1 k = {pi ∈ R3 |i = 1, 2, ...,N} of N points and Xv2 k = {qi ∈ R3 |i = 1, 2, ...,M} of M points, the distant point cloud registration problem is to find the optimal rotation matrix R ∈ SO(3) and translation vector t ∈ R3 that align the pair of point clouds, i.e., Xv1 k RT + tT aligns with Xv2 k . For the ease of explanation, we refer to the point cloud frames to be registered as the key frames and point cloud frames before and after one key frame as non-key frames.

4 SystemDesign

The core idea of Aggregated Point Cloud Reconstruction (APR) is to leverage an autoencoder structure to train a powerful encoder as the feature extractor. Instead of reconstructing the original key frame, the decoder in the autoencoder structure reconstructs an aggregated point cloud. Therefore, with the well-trained encoder, features can be extracted from one single key frame but can effectively represent an environment point cloud with a dense and panoramic view, which are used for online registration. To this end, as depicted in Figure 2, APR has distinct pipelines for offline training and for online inference, respectively. More specifically, the training pipeline consists of the following three components:

Point Cloud Series

Figure 2: The training and inference pipelines of APR. The offline training pipeline consists of three components, i.e., APG, KFE and NPR. APG generates the current aggregated point cloud (APC) based on nearby non-key frames of a vehicle in a non-trainable manner. KFE and NPR constitute an autoencoder, where the encoder extracts per-point features and the decoder takes such features to reconstruct the current APC. The Chamfer Distance between these two point clouds is used as a complementary loss against the metric learning loss to train the extractor (i.e., FCGF and Predator). During the online inference, only the current key frames of a pair of vehicles are exchanged and used to perform feature-based registration with the encoder at each vehicle.

Key-frame Feature Extraction (KFE). KFE is the encoder in the autoencoder structure which is a fully convolutional network (FCN). The encoder acts as the feature extractor that extracts features from a key frame for both online registration during inference and non-key-frame reconstruction during training. APR is a universal feature extraction framework and can take an existing per-point feature extractor as its encoder. For instance, in our current implementation, two SOTA fully convolutional methods, i.e., FCGF [Choy et al., 2019] and Predator [Huang et al., 2021] are adopted to satisfy the real-time requirement for online registration. More point cloud feature extractors such as D3Feat [Bai et al., 2020] would be adopted in the future.

Aggregated Point Cloud Generation (APG). APG takes neighboring non-key frames of the current key frame as input and aligns them to form an Aggregated Point Cloud (APC) as the reconstruction target of the decoder.

Non-key-frame Point Cloud Reconstruction (NPR). NPR acts as the decoder of the autoencoder which explicitly extracts local geometries from the feature map by generating several new points around the point corresponding to a feature vector. The generated points are compared with the APC generated by the APG using the Chamfer Distance loss.

During online inference, only the current key frames are exchanged between a pair of vehicles. Then the key frames are used to extract APR features and perform conventional feature-based registration at each vehicle.

4.1 Aggregated Point Cloud Generation

The APG component is used to generate APC as the reconstruction targets for the autoencoder during offline training. Specifically, given the point cloud series of vehicle vi, i.e., {Xvi 1 ,Xvi 2 , . . . ,X vi t } and a key frame Xvi k , APG first samples 2ψ non-key frames centered at the key frame Xvi k from the point cloud series, i.e., ψ frames before Xvi k and ψ frames after Xvi k with frames separated at a short distance interval of α meters. Then, these sampled non-key frame and Xvi k are aligned by ground-truth position label, or through a conventional multi-way registration method [Choi et al., 2015] when precise relative position is not available. The frames are further cropped with a scope sphere centered at the key frame to discard distant points out of interest. Finally, voxel downsampling is conducted on the APC to enhance the robustness against density variation. Figure 3 illustrates an APC using six non-key frames (denoted as blue points) centered at one key frame (denoted as orange points) with frames separated at a distance interval of ten meters.

4.2 Non-key-frame Point Cloud Reconstruction

Our key insight is that APC naturally suffers less from density variation and self-occlusion, as depicted in Figure 3. Through the reconstruction of local geometry, ground truth correspondence features are encouraged to be similar in order to reconstruct similar local shape of APCs. This can be interpreted as a new side-channel for feature similarity supervision which is less influenced by density and view disparity.

Specifically, the encoder (feature extractor) in APR is trained to guess the local geometry and encode such information into a per-point feature map; The decoder in APR is trained to take the feature map, and decode it into a denser point cloud. Formally, given a key frame of N points Xk = {pi ∈ R3 |i = 1, 2, ...,N} and the corresponding ldimensional features F = {fi ∈ Rl |i = 1, 2, ...,N}, the decoder can be formulated in two ways, either symmetrical or asymmetrical.

Symmetrical Design. In this case, the decoder Dsym is an FCN that shares identical structure with the encoder except for the input and output dimensions. The decoder takes the feature map and outputs the reconstructed offsets as the output feature O = Dsym(F) ∈ RN×3ϕ , where ϕ is referred to as the point generation ratio.

Asymmetrical Design. Alternatively, the decoder Dasym can be a small MLP that takes a single feature vector as input, providing a lightweight and universal option. For each feature fi, the decoder locally generates the set of location offsets as output Oi = Dasym(fi) ∈ R3ϕ .

Fusion. We interpret Oi as ϕ segments of 3-dimensional vectors with each vector being an offset b1i , b2i , ..., bϕi ∈ R3 . By adding these location offsets to the corresponding original point pi in the key frame, a reconstructed point cloud is obtained: Rk = {pi + bji ∈ R3 |i = 1, 2, ..., N; j = 1, 2, ..., ϕ}. The collected offsets are denoted as ˆO = {bji ∈ R3 |i = 1, 2, ...,N; j = 1, 2, ..., ϕ}. We contract the index i and j into a single index in the following section for convenience of display.

4.3 Loss Formulation

To train the autoencoder, the Chamfer Distance (CD) between the APC generated by the APG and the reconstructed environment generated by the NPR is used as a complementary loss against the metric learning loss involving the extracted features from a pair of point clouds. L2 regularization is also applied to the generated offset lengths.

Specifically, given a pair of point clouds Xv1 k = {pi ∈ R3 }, Xv2 k = {qj ∈ R3 }, and their corresponding offsets ˆOv1 = {ol ∈ R3 }, ˆOv2 = {om ∈ R3 }, CD loss and regularization loss are formulated as:

Metric Learning (ML) is generally utilized to train effective fully-convolutional feature extractors. In this paper, we use the original Hardest Contrastive Loss for FCGF [Choy et al., 2019], and the Circle Loss [Sun et al., 2020] plus Overlap Loss and Matchability Loss for Predator [Huang et al., 2021], same as their original paper.

Let LML denote the metric learning loss of an embedded point cloud feature extraction scheme (e.g., FCGF [Choy et al., 2019] and Predator [Huang et al., 2021]) and the final loss is formulated as L = LML + λ1LCD + λ2LL2, where λ1, λ2 controls the ratio between loss terms.

5 Performance Evaluation

Distant Point Cloud Datasets. Previously, only closerange registration datasets have been extracted from KITTI [Geiger et al., 2012] and nuScenes [Caesar et al., 2020]. To create distant point cloud datasets, following 3DLoMatch proposed in Predator [Huang et al., 2021], we pick point cloud pairs with ≤ 30% overlap from KITTI and nuScenes, creating LoKITTI and LoNuScenes, respectively. We also subdivide KITTI and nuScenes w.r.t. the distance between two LiDAR centers, denoted as d (in meters). Dataset slices with d ∈ [d1, d2] in KITTI and nuScenes are dubbed KITTI[d1, d2] and nuScenes[d1, d2], respectively. In addition, previous close-range datasets with d = 10, referred to as KITTI and nuScenes, are also considered. We adopt three registration criteria for assessing a successful registration, i.e., loose (5◦ & 2m) [Yew and Lee, 2018], normal (1.5◦ & 0.6m) and strict (0.5◦ & 0.3m). We report three metrics, including relative rotation error (RRE), relative translation error (RTE), and registration recall (RR). For alignment of nonkey frames in APG, we use only ground-truth pose from SemanticKITTI [Jens et al., 2019] and nuScenes [Caesar et al., 2020], while the effect of low-precision position labels are simulated in the robustness analysis in Secion 5.2. We set default parameters as ψ = 3 and α = 10.

Training Strategy. Although we report arbitrary results on [d1, d2] datasets, it is hard for the network to converge when directly trained with d2 ≥ 30 or d1 ≥ 10. As a result, we first pre-train a model on a dataset with lower distance, where d ∈ [5, 20]. Then the pre-trained model is further finetuned on [5, d2] (d2 ≥ 30) to guarantee convergence. For cases where d2 < 30, no finetuning is applied. Finally, the resulting model is used to report results on [d1, d2] datasets.

Decoder Choice. We implement both symmetrical design (denoted with postfix (s)) and asymmetrical design (denoted with postfix (a)) on FCGF, and found that symmetrical design generally performs better. However, symmetrical Predator is not trainable due to memory constraints, so only asymmetrical design is reported for Predator.

Visualization. We demonstrate the registration performance of all methods on KITTI in Figure 4.

5.1 Parameter Configuration

Effect of Point Generation Ratio ϕ. We first examine the effect of the point generation ratio ϕ, varying it from one to eight. Table 2 lists the performance of FCGF+APR(a/s) and Predator+APR(a) on KITTI[5, 20] validation set. It can be seen that the performance of FCGF+APR(a/s) peaks when ϕ = 4, while the performance of Predator+APR drops when ϕ < 4. We can see that ϕ = 4 is a good choice for all feature extractors.

Effect of Decoder Size. We fix the point generation ratio ϕ to four and vary the size of the per-point decoder in asymmetrical designs. Specifically, the asymmetrical decoder is an MLP with flexible hidden layer size, e.g., (29 , 28 ) reveals a 3layer MLP with l, 512, 256, ϕ × 3 dimensions from input to output. Table 2 lists the performance of FCGF+APR(a) and Predator+APR(a) on KITTI[5, 20] validation set. It can be seen that the decoder with the size of (29 , 28 ) can achieve the best RR performance. Therefore, we set the decoder hidden layer size in asymmetrical design to (29 , 28 ) in the following experiments.

5.2 Performance Comparison

Improvements over Baselines. We compare our method with baseline feature extractors under all three criteria on all four datasets. Table 1 lists the registration results. Compared to the baselines, APR can effectively improve the registration performance both under normal overlap and low overlap, either symmetrical or asymmetrical. Symmetrical designs excel under loose criterion but lose advantage under rigid criterion. Though Predator+APR(a) does not excel in all tests with RRE or RTE slightly falling behind in some cases, it generally outperforms other methods in most scenarios. On average, APR improves the RR of FCGF and Predator by 4.4%, 3.5%, respectively on KITTI, and 0.6%, 6.9%, respectively on nuScenes. The average RR improvements for both backbones on LoKITTI and LoNuScenes are 7.1% and 4.6%, respectively.

Impact of Distance d for allMethods. Due to the nature of LiDAR sensors, points on near objects are denser than those on distant objects. Consequently, density and view disparity are mainly caused by increased distance between both LiDARs. Table 3 shows the RR of all candidate methods under increasing ranges of d on KITTI[d1, d2] dataset under normal registration criterion (1.5°,0.6m). It can be seen that SpinNet, D3Feat and Geotransformer gradually fail with increasing d, as they suffer from bad convergence caused by improper loss or structure designs. The APR-empowered methods generally receive a performance boost that is proportional with the distance d, and Predator+APR(s) surpasses all other methods on all distances. We conclude that APR can effectively confront the density and view disparity between two separated point clouds, and sets a new SOTA for distant point cloud registration problem.

Robustness Analysis. We investigate the robustness of APR, assuming APG could fail to align some non-key frames with only noisy pose annotation available, as in OdometryKITTI. To simulate different degrees of failure during training, we impose random rotation (θ ∈ [-π, π]) on a random axis on Ndisturb out of 2ϕ non-key frames. Table 4 lists the RR of APR on both KITTI[5, 20] and nuScenes[5, 20] dataset with Ndisturb ∈ [0, 5]. Under the optimal parameter setting, we have ψ = 3, i.e., 6 non-key frames, 3 on each side. It can be seen that asymmetrical design performs better, whose RR first rises then decreases with Ndisturb, peaking at 95.4% for FCGF+APR(a), and 99.5% for Predator+APR(a), respectively on KITTI dataset. Similar trends can be observed on nuScenes dataset. When Ndisturb ≥ 4, RR for both asymmetrical methods drops slightly. All versions of APR, are comparable with not-disturbed version (Ndisturb = 0), and better than native methods (see the last column). We conclude that APR is robust against APG failures.

Impact of Density Variation. In this experiment, we examine the effect of density variation alone, without view disparity. We choose d ∈ [5, 10] to minimize the difference in point of view, then randomly downsample one point cloud out of every point cloud pair, creating simulated density variation between a pair of point clouds. Table 5 lists the RR of FCGF, Predator and their asymmetrical APR-empowered version on nuScenes[5, 10] dataset under normal registration criterion. The relative improvement of RR for APR-empowered methods compared to native methods increases as the downsample ratio decreases, meaning that APR is effective in confronting point cloud density variation.

6 Conclusion

We have proposed APR, a novel feature extraction framework for online distant point cloud registration on moving vehicles. APR leverages an autoencoder design to obtain a better feature extractor through reconstruction of the aggregated point clouds during training. Our method is able to force the feature extractor to guess local geometry information without changing any of the extractor design or imposing extra inference time. The extracted APR features are more robust against density variation and view disparity, significantly improving the accuracy of distant point cloud registration. APR outperforms all other extractors by a large margin, increasing average registration recall of SOTA extractors by 7.1% on LoKITTI and 4.6% on LoNuScenes.