BiCo-Net: Regress Globally, Match Locally for Robust 6D Pose Estimation

1 Introduction

The problem of 6 Degree-of-Freedom pose estimation aims to predict the orientation and location of one detected object instance in 3D space from a canonical model via recovering a rigid transformation from the object space to the camera space. Such problem has been widely encountered in fields of engineering such as robotic grasping and autonomous driving. A large number of deep methods including regression based [Xiang et al., 2018; Park et al., 2019] and keypointbased [Tekin et al., 2018; Peng et al., 2019] only rely on extracting texture information from RGB images, which are sensitive to objects with poor textures. Alternatively, with the development and wide application of depth sensors, exploring 6D pose estimation on RGB-D images becomes popular in recent years, where depth images can provide complementary geometry information to RGB images.

∗Corresponding authors

The pioneering works [Xiang et al., 2018; Li et al., 2018] on RGB-D images are in a two-stage structure: deep pose estimation and post-processing refinement with iterative closest point (ICP), which leads to less efficiency during inference. Residual learning based refinement in [Wang et al., 2019a] is proposed to improve efficiency with orders of magnitude faster than the ICP, which can ensure real-time inference. From a practical perspective, there remain two compounded challenges for learning a robust 6D pose function to address the problem: 1) severe occlusion and 2) unavoidable systematic noises during depth imaging [Barron and Malik, 2013].

Most of the existing methods [Wang et al., 2019a; He et al., 2021] concern on coping with the former by improving feature encoding on integrating textural and shape features into discriminative representation, while very few work pays attention to the latter. In [Zhou et al., 2021], a point refinement network is the first attempt to explicitly polish point clouds via completion and denoising, whose features are combined with those encoded from raw point clouds and RGB images for better multi-modal feature fusion to regress final poses. However, performance gain of PR-GCN in [Zhou et al., 2021]

over the pose regression baseline can be sensitive to point cloud generation, which itself remains an active and challenging task especially under incomplete and noisy observations.

We argue that encoding pose-sensitive local features and modeling a statistical distribution of pose inliers are two key factors for accurate and robust performance in 6D pose estimation. On one hand, pose estimation dependent on local texture and geometry can perform stably when missing a part of object regions, which is thus robust against occlusion, but those local features are sensitive to the quality (e.g. noises and resolution) of the acquired data. On the other hand, the distribution of pose predictions on local features can be explored to alleviate negative effects of systematic noises in depth imaging. In this paper, we propose a novel deep model for 6D pose estimation on RGB-D images, i.e. a Bi-directional Correspondence Mapping Network (BiCo-Net) as shown in Figure 1, by simultaneously addressing both challenges in a unified and implicit manner.

Given the point cloud (generated from the depth image) and the RGB image of an object under the observation pose, the feature output of DenseFusion feature encoder [Wang et al., 2019a] is decoded to generate the corresponding oriented points in the canonical space. To effectively exploit object model priors as a reference, a clean and complete model point cloud under the observed pose is similarly produced in an encoder-decoder learning style from the corresponding oriented points under the canonical pose, which is in an opposite direction of the aforementioned point cloud generation. Inspired by the success of point-pair features [Drost et al., 2010], a set of oriented point pairs from the input (e.g. the camera space) can be randomly sampled to produce accurate pose predictions by matching with those corresponding ones in the output space (e.g. the model space). As pose prediction relies only on one pair of local oriented points, such a characteristic can encourage robustness against sparse and occluded point clouds. Moreover, owing to revealing a global distribution of rigid pose transformation favored by multiple point pairs, selecting and combining pose predictions of the bi-directional correspondence mapping can alleviate negative effects of outliers in noisy point clouds. For imposing pose-sensitive information, features of the bi-directional point cloud generation can be regularized by a typical pose regression as [Wang et al., 2019a].

In general, the whole network consists of an ensemble of two parts: direct pose regression on a concatenation of global features and point-wise features as [Wang et al., 2019a] and pose computation via locally matching of oriented point pairs in-between canonical and observed poses, which can further refine pose predictions. Extensive experiments on three popular benchmarks, i.e. YCB-Video [Xiang et al., 2018], LineMOD [Hinterstoisser et al., 2011] and Occlusion LineMOD [Brachmann et al., 2014], can verify superior performance of the proposed BiCo-Net to the state-of-the-art methods, especially for severe occluded scenes. Main contributions of this paper lie as follows:

• This paper proposes a novel 6D pose estimation method - the BiCo-Net based on locally matching oriented point pairs between the model and camera space and direct pose regression.
• The proposed BiCo-Net is implicitly robust against occlusion and sparse point distribution owing to exploiting the pose-sensitive characteristic of each single pair of oriented points under different poses.
• Negative effects of outliers in noisy depth images can be mitigated via selection and an ensemble of redundant pose predictions.
• Our BiCo-Net can gain state-of-the-art performance on three benchmarks, especially on the more challenging Occlusion LineMOD dataset.

2 RelatedWorks

Keypoint-based 6D Pose Estimation. A typical keypointbased pose estimation algorithm is designed in a two-stage pipeline: first localizing 2D projection of pre-defined key points in 3D space, and pose predictions can be generated via 2D-to-3D key point correspondence with a PnP [Lepetit et al., 2008]. Existing methods can be categorized into two groups: object detection based [Rad and Lepetit, 2017; Tekin et al., 2018] and dense heatmap based [Oberweger et al., 2018; Pavlakos et al., 2017]. The former performs well on localizing sparse key points of object foreground, but are sensitive to occlusion [Oberweger et al., 2018]. The latter group of methods are more robust to inter-object occlusion in a dense correspondence mapping style. PVNet [Peng et al., 2019] is proposed to detect 2D keypoints via voting on pixel-wise predictions of the directional vector that points to keypoints and is robust to truncation and occlusion. PVN3D [He et al., 2020] extends the 2D keypoints into 3D space by building 3D-3D correspondence and then uses the least-squares fitting [Arun et al., 1987] to generate the pose prediction. Learning of directional vectors pointing to 3D keypoints under camera space suffers from the variation of object pose and the vast 3D search space. In contrast, the bi-directional correspondence mapping in our method makes point-wise predictions on 3D oriented points between the model and camera space directly to build up dense correspondence, instead of their projection in 2D images or the directional vector pointing to 3D keypoints, which enhances local feature discrimination via direct regression based regularization in a pose-sensitive manner.

Dense Regression-based 6DPose Estimation. An alternative group of algorithms to cope with occlusion is to produce dense pose predictions for each pixel or local patches with hand-crafted features [Liebelt et al., 2008; Sun et al., 2010], CNN patch-based feature encoding [Doumanoglou et al., 2016; Kehl et al., 2016] and CNN pixel-based feature encoding [Wang et al., 2019a; Zhou et al., 2021], whose final pose output is selected via a voting scheme. DenseFusion [Wang et al., 2019a] fuses RGB and Depth information for each point and uses point-wise features to regress dense poses. Zhou et al. proposes PR-GCN [Zhou et al., 2021] to handle incomplete and noisy point clouds in practice via designing a PRN to complete and denoise observed point clouds and use graph convolution to better integrated the RGB and Depth information. In [Wang et al., 2019b], normalized object coordinate space is proposed to build up 3D-3D correspondences for each pixel in category level then recover the

6D pose and size by the least-squares fitting. The proposed BiCo-Net is designed to both regress dense poses from pointwise features and locally match pose-sensitive oriented points in a unified framework. The complementary characteristics of dense pose regression and dense correspondence mapping can be fully utilized to gain robust pose predictions. Moreover, compared with the least-squares fitting, our point-pair pose computation which uses the correspondence of two oriented points can perform more robust under heavy occlusion.

3 Methodology

The problem of 6D pose estimation of an object given the RGB-D image and its canonical CAD model is to estimate their rotation R ∈ SO(3) and translation t ∈ R3 , which can be defined as a rigid transformation T = [R|t] from the model space with respect to the camera space. The whole pipeline of our BiCo-Net is illustrated in Figure 2, which can be divided into three modules: feature encoding on segmented object instances (see Sec. 3.1) in a blue block, generation of oriented points via Bi-directional Correspondence Mapping (see Sec. 3.2) in a green block, and an ensemble of pose predictions by point pair matching and point-wise regression (see Sec. 3.3) in a yellow block.

3.1 Instance Segmentation and Feature Encoding

Following existing methods [Wang et al., 2019a; Zhou et al., 2021], BiCo-Net first employs an off-the-shelf instance segmentation method for RGB images (e.g. Mask RCNN [He et al., 2017] in our experiments) to segment the object of interests, which produces a cropped image patch I and one scene point cloud under the camera space S = {(si, nsi ) ∈ R6 }Ni=1 , where si ∈ R3 denotes 3D coordinates converted from the masked depth region and nsi ∈ R3 denotes its normal vectors physically computed by PCA. Both I and S are fed into CNN-based and MLP-based feature encoders respectively to extract texture and geometric features from heterogeneous data sources, which are further fused by the DenseFusion module introduced in [Wang et al., 2019a] to obtain pointwise features F S = {f s i ∈ R512 }Ni=1 . Similarly, for exploiting the model priors, we randomly sample one clean model point cloud ˜m from the canonical CAD model and compute point-wise normal vectors ˜nm to form ˜ M = {(˜mj, ˜nmj ) ∈ R6 }Mj=1 as input of an MLP-based feature encoder to generate point-wise features F ˜ M = { ˜ f m j ∈ R512 }Mj=1 .

3.2 Bi-directional Correspondence Mapping

Given features F S from visual observation I and S, the BCM-scene (BCM-S) in the top row of Figure 2 aims to regress the corresponding oriented point cloud under the model space M = {(mi,nmi ) ∈ R6 }Ni=1 , where mi = R-1 (si - t) and nmi = R-1nsi . With the point-wise features F S as input, we employ a MLP with {512,1024} neurons and an average pooling (AvgPool) to generate a global feature gS = AvgPool(MLPS(F S )) ∈ R1024 . Finally, ˆM = BCMS(F S , gS ) for superior robustness to only using F S , where generated points ˆM = {( ˆmi, ˆnmi )}Ni=1 . Similarly, the BCM-model (BCM-M) in the bottom row of Figure 2 aims to reconstruct a clean point cloud under the camera space ˜ S = {(˜ sj, ˜nsj ) ∈ R6 }Mj=1 from the features under the model space F ˜ M, where ˜ sj = R˜mj + t and ˜nsj = R˜nmj . To this end, a global feature pS ∈ R1024 aggregated from F S together with F ˜ M are learning to regress ˜ S, while ˆ S = BCMM(F ˜ M, pS ) is the generated point cloud under the camera space by the BCM-M branch. Moreover, to impose pose sensitive information for generation of point clouds in the BCM-S and BCM-M, a direct point-wise pose regression (PR) in the middle row of Figure 2 on F S , g ˜ M, and pS to predict ˆ T PR = PoseReg(F S , g ˜ M, pS ), where ˆ T PR = {( ˆRi, ˆ ti)}Ni=1 and g ˜ M = AvgPool(MLPM (F ˜ M)).

Loss Functions. We use the Euclidean distance to supervise both BCM branches as follows:

where λ is a trade-off parameter, (mi,nmi ) and (˜ sj , ˜nsj ) are ground truth oriented points of the BCM-S and BCM-M

branches, while ( ˆmi, ˆnmi ), (ˆ sj, ˆnsj ) are the generated points. To ensure the pose consistency between BCM branches and direct regression branch, we replace the ground truth pose in (mi,nmi ), (˜ sj , ˜nsj ) with the mean of point-wise predicted pose [ ˆRi|ˆ ti] for symmetric objects. For supervising ˆ T PR with T = [R|t] in the pose regression branch, we use the ADD Loss [Xiang et al., 2018] for asymmetric objects and ADD-S Loss for symmetric objects:

where K is the number of points sampled from the surface of the CAD model, [R|t] and [ ˆRi|ˆ ti] are ground truth and point-wise predicted poses respectively. The total loss of our BiCo-Net can thus be written as:

3.3 Point Pair Matching and Prediction Ensemble

With generated point clouds ˆM and ˆ S under the model and camera space respectively, our goal is to learn a rigid transformation T or T -1 between camera and model space. Encouraged by Point-pair feature (PPF) [Drost et al., 2010] to describe object poses by matching local features generated from oriented point pairs, any pair of oriented points in S and its corresponding ones in ˆMcan determine object pose, while similar situation is observed for point pairs between ˜ M and ˆ S. Specifically, object pose can be readily obtained by randomly sampling a point-pair (sr, si) from scene points S or (˜mr,˜mj) from model points ˜ M and then matching it with the corresponding pair ( ˆmr, ˆmi) or (ˆ sr, ˆ sj) generated by our BCM branches. The transformation from ( ˆmr, ˆmi) to (sr, si) is defined as the following [Drost et al., 2010]:

where T ˆmr→x denotes a transformation that translates ˆmr into the origin and rotates ˆnmr on to the x-axis, and the same definition of Tsr→x for transforming (sr, nsr ). When ˆmr and sr are aligned to the x-axis, there is a α angle difference about the x-axis between Tsr→xsi and T ˆmr→x ˆmi, which encourages to use Rx(α), a rotation with respect to the x-axis, to align Tsr→xsi and T ˆmr→x ˆmi. As a result, the rigid transformation Φ : (sr, si, ˆmr, ˆmi) → T can be written as:

Note that, 6D object pose by locally matching a pair of oriented points under the camera and model space can be readily computed for desirable real-time inference.

As our method only relies on each single point pair to estimate 6D object pose, it allows sparse and imbalanced point distributions, which is thus able to achieve good performance for severe occlusion. For alleviating noises in point clouds and increasing inference speed, we use the FPS algorithm to downsample S and ˜ M to a subset of Z points, which are then to generate Z 2 point pairs to compute pair-wise pose candidates Tz = [Rz|tz] for the BCM-S and BCM-M respectively. For avoiding unreliable pose candidates from point-pairs constructed by neighboring points, pose predictions will be filtered out with the following error measure:

and preserving the top 10% of candidates as the pose prediction sets T BCM-S and T BCM-M of these two branches.

An Ensemble of Pose Predictions. As mentioned in Sec. 1, we obtain three sets of pose predictions (i.e. T PR from direct pose regression; T BCM-S and T BCM-M via locally matching with point pairs), from three branches of the BiCo-Net. For achieving superior robustness via using the complementary information of three sets, we consider applying the average pose of T PR ∪ T BCM-S ∪ T BCM-M as the final pose output.

4 Experiments

4.1 Datasets and Settings

Datasets. To evaluate our BiCo-Net comprehensively, experiments are conducted on three popular benchmarks - the YCB-Video dataset [Xiang et al., 2018], the LineMOD [Hinterstoisser et al., 2011], and the more challenging Occlusion LineMOD [Brachmann et al., 2014]. The YCB-Video dataset has 92 videos in total, each of which shows a subset of 21 objects with varying textures and sizes under cluttered indoor environment. Following existing works [Wang et al., 2019a; Zhou et al., 2021], we adopt 16,189 frames from 80 videos with an additional 80,000 synthetic images provided by [Xiang et al., 2018] for training and extract 2949 key frames from the remaining 12 videos for testing. The LineMOD contains 15,783 images belonging to 13 low-textural objects placed under different cluttered environments. We use the standard training/testing split as [Xiang et al., 2018; Wang et al., 2019a]. The Occlusion LineMOD provides 6D pose labels of 8 objects selected from the LineMOD and includes 1214 images with multiple heavily occluded objects, which is made more challenging.

Performance Metrics. Following [Wang et al., 2019a; Zhou et al., 2021], we adopt the average distance (ADD) [Xiang et al., 2018] and ADD-Symmetric (ADD-S) as performance metrics. 6D pose predictions are considered to be correct if the ADD/ADD-S is smaller than a predefined threshold. For the YCB-Video dataset, we vary from 0 to 10cm to plot an accuracy-threshold curve and report the area under the curve (AUC). We also report the result of ADD-S <2cm as [Wang et al., 2019a; Zhou et al., 2021]. For the LineMOD and the Occlusion LineMOD, we use ADD-S for symmetric objects (i.e. eggbox and glue) and ADD for the remaining objects having an asymmetric geometry while taking 10% of the diameter as the threshold.

4.2 Implementation Details

The numbers of scene/model points, i.e. , N/M, are set to 1000/1000. In point-pair pose computation, we downsample the scene points and model points to Z = 100 points by the FPS which thus generates Z 2 = 10, 000 pose candidates from point pairs. The hyper-parameter λ in the losses of BCM-S and BCM-M branches is empirically set to 0.05. We use the Adam optimizer with a 10-4 learning rate to train our model for 50 epochs, and the learning rate decays 0.3 per 10 epochs.

4.3 Comparison with State-of-the-art Methods

Comparative evaluation of the proposed BiCo-Net and stateof-the-art methods on the YCB-Video, LineMOD and Occlusion LineMOD datasets are showed in Tables 1, 2, and 3. In general, our method can consistently achieve state-of-the-art performance in all benchmarks. Specifically, on the YCBVideo dataset, our method achieves the best performance on both metrics in Table 1, and similar results on the LineMOD in Table 2 can also be observed. Compared to moderate improvement on the YCB-Video and LineMOD, the proposed BiCo-Net can gain accuracy of 69.5% on the more challenging Occlusion LineMOD, which is significantly superior to the state-of-the-art methods as illustrated in Table 3. Such results can verify the effectiveness of our BiCo-Net for 6D pose estimation on RGB-D images. In addition, we measure the inference time on average of the proposed method: the forward time of BiCo-Net is 16ms; the point-pair pose computation time is 29ms; the segmentation network takes 30ms. As a result, the average time for processing a frame for inference is 75ms with a GTX 1080 Ti GPU, which is comparable to existing methods (e.g. 60ms for the DenseFusion and 68ms for the PR-GCN) to meet desirable real-time inference in practical applications.

4.4 Ablation Studies

Robustness against Inter-Object Occlusion. To evaluate the robustness of our method against occlusion, we follow

[Wang et al., 2019a; He et al., 2020] to define occlusion level by using the percentage of invisible points on the object surface. We compare our BiCo-Net with several methods, including PoseCNN+ICP [Xiang et al., 2018], DenseFusion [Wang et al., 2019a], and PVN3D [He et al., 2020], on the YCB-Video with accuracy of ADD-S <2cm under different levels of occlusion, whose results are shown in Figure 3. With the percentage of invisible part increasing, our method can perform stably well when performance of comparative methods decrease or even collapse, which again confirms superior robustness of the proposed BiCo-Net to its competitors even with heavily occluded objects.

Effects of An Ensemble of Filtered Pose Predictions. To learn a correspondence mapping using multiple instances, a typical robust scheme is least-square fitting based RANSAC to optimize the hypothesis with the maximum inliers. We conduct one experiment in terms of the AUC metrics of the YCB-Video dataset to obtain 79.1%/89.1% on the output of the BCM-S/BCM-M branch, which is significantly inferior to the results using point-pair matching in our BiCo-Net, reaching 94.9%/95.0% only with T BCM-S /T BCM-M (i.e. without direct pose regression). Such a result confirms the rationale of ensembling pose predictions in T BCM-S and T BCM-M , owing to the better robustness against outliers in noisy point clouds than the RANSAC. Moreover, we take the average of T PR ∪ T BCM-S ∪ T BCM-M as the final pose prediction of our BiCo-Net in an ensemble manner, whose effectiveness is verified on the YCB-Video and Occlusion LineMOD respectively (See the last row of Table 4). Moreover, without direct pose regression, as results of T BCM-S /T BCM-M are only 94.9%/95.0% on the AUC metrics of the YCB-Video dataset, the direct pose regression branch can benefit generation of point clouds in the BCM branches (i.e. 95.7%/95.6% for Z = 100 in Table 5).

Effects of Bidirectional Correspondence Mapping. For evaluating the effectiveness of the BCM-S and BCM-M branches of our method, we conduct experiments on the YCB-Video dataset and the Occlusion LineMOD. The baseline method takes scene points S and cropped image patch I as input and only performs point-wise pose regression by T PR = PoseReg(F S , pS ). The average of T PR of each object instance is utilized as its pose prediction. As shown in Table 4, the method introducing the BCM-S branch can improve 0.7% and 1.6% on two datasets respectively, indicating that BCM-S effectively improved the discrimination of local feature coding owing to introducing point-wise pose sensitive regularization. The BCM-M branch can outperform the baseline by 0.8% and 4.0% on two datasets respectively. This can be credited to exploiting shape priors of the CAD model provides an ideal reference for the pose regression network, which alleviates the partiality of scene point clouds due to (self-)occlusion. The combination of BCM-S and BCMM further gains an improvement of 1.1% and 4.5% on both datasets, indicating that these two branches provide complementary information, which further supports our claim about an ensemble of pose predictions.

Evaluation on Size of Point-pairs. We evaluate pose predictions via point-pair matching by taking an average of T BCM-S and T BCM-M as the output pose respectively with varying size Z of points, and report results in Table 5. As our method only relies on simple geometric properties of two local points for pose computation, even with sparse input points, T BCM-S /T BCM-M can gain comparable performance to T PR (see the first row of Table 4).

5 Conclusion

This paper introduces a novel neural network for 6D pose estimation based on an ensemble of direct regression and locally matching pairs of oriented points under the camera and model space. The proposed BiCo-Net can achieve robust performance on severe inter-object occlusion and systematic noises in scene point clouds, owing to our design of exploiting pose sensitive information carried by each pair of oriented points and an ensemble of redundant pose predictions. Experiment results can verify the effectiveness of each module in our method and the state-of-the-art performance.

Acknowledgments

This work is supported in part by the National Natural Science Foundation of China (Grant No.: 61771201, 61902131), the Program for Guangdong Introducing Innovative and Entrepreneurial Teams (Grant No.: 2017ZT07X183).