Learning to Estimate Object Poses without Real Image Annotations

1 Introduction

Monocular object pose estimation aims to estimate the 6DoF pose (i.e., 3D translation and 3D rotation) of an object from a color image. Fast and accurate pose estimation has a wide range of applications such as robotic manipulation and augmented reality. Current state-of-the-art methods [Peng et al., 2019; Li et al., 2019; Zakharov et al., 2019; Hu et al., 2020] usually leverage learning-based techniques, which require a large amount of real annotated training data. However, annotating 6D poses is very time-consuming and labor-intensive [Hinterstoisser et al., 2012].

There exist some works exploring how to learn object pose estimation without real image annotations. Some works [Sundermeyer et al., 2018; Manhardt et al., 2019] attempt to use synthetic images for training, which comes with no cost of data annotation. Unfortunately, these methods do not generalize well to the real data due to the sim-to-real domain gap. To solve this problem, [Wang et al., 2020] trains pose estimation networks on real unannotated RGBD data with a self-supervised learning method. They compare the rendered image with the input image and optimize the network parameters to minimize the difference. While being differentiable, the supervision is indirect and the loss functions are highly nonlinear, which makes the learning process prone to local minima. As a result, there is a huge performance gap between [Wang et al., 2020] and fully-supervised methods.

∗Authors contributed equally

†Corresponding author

In this work, we propose a simple yet effective learning framework that leverages depth information to learn object pose estimation. Without using any real image annotations, we achieve competitive results on par with the state-of-the-art methods that use real annotations for training. The basic idea is to use synthetic images to train an initial pose estimator and then finetune the initial pose estimator on unannotated RGBD data with a depth-based pose refiner (Figure 1). The key findings are as follows:

• The main difference between synthesized and real images is in the background. Therefore, the synthetically trained pose estimator will have better performance on the real data if we use a two-stage pipeline that first detects the object with an object detector and then es-

timates the object pose from the cropped image.

• With the above two-stage design, the pose estimator trained only on synthetic data produces reasonable pose estimates on real data in most cases, though they are not very accurate.
• If depth maps are available, the inaccurate poses can be refined by a depth-based pose refiner, e.g., ICP [Zhang, 1994], yielding accurate pose estimates. Furthermore, failed pose estimates can be identified by a depth-based pose evaluator.
• Finally, the optimized poses output by the depth-based pose refiner can be used as pseudo ground truth to supervise the RGB-based pose estimator and largely boost its performance on the real data.

A practical advantage of our framework is the flexibility to allow a combination of any pose estimator and pose refiner. Neither component needs to be fully differentiable which is a necessary condition in [Wang et al., 2020]. This means that we are able to apply our learning framework to train twostage pose estimation approaches [Rad and Lepetit, 2017; Peng et al., 2019; Song et al., 2020]. These two-stage methods usually include a RANSAC-based Perspective-n-Point (PnP) algorithm which is non-differentiable.

In summary, we present a simple yet effective framework for learning object pose estimation without real image annotations. Without using any real annotated data, we achieve competitive performance with state-of-the-art methods on public benchmarks. Specifically, on the LINEMOD [Hinterstoisser et al., 2012] and Occluded LINEMOD [Brachmann et al., 2014] datasets, we achieve an accuracy of 88.4% and 47.3% in the ADD(-S) metric, respectively, without using any real annotation provided by the datasets. The best results reported previously [Song et al., 2020] on these two datasets are 91.4% and 62.2%, respectively, which used real annotations provided by the datasets for training. Compared to recent self-supervised methods [Wang et al., 2020; Sock et al., 2020], we achieve a significant performance gain (88.4% vs. 60.6% on the LINEMOD dataset and 77.1% vs. 48.6% on the YCB-Video [Xiang et al., 2018] dataset).

2 RelatedWork

6D Pose Estimation. One line of works adopts a two-stage paradigm: They first establish correspondences between the 2D image and the 3D model using deep neural networks, then solve the PnP problem to compute the 6D pose. [Rad and Lepetit, 2017; Peng et al., 2019; Wang et al., 2021b] try to detect projections of a set of predefined 3D points related to the 3D model. [Song et al., 2020] propose a hybrid representation to express different geometric information, which includes not only points but also lines. To address occlusion, some works [Park et al., 2019; Zakharov et al., 2019; Li et al., 2019] attempt to predict pixel-wise object coordinates to create dense correspondences for 6D pose estimation. Another line of works directly regresses the 6D pose from an image. [Xiang et al., 2018] trains a network to regress the 6D pose. There also exist some works learning a pose embedding. For instance, [Sundermeyer et al., 2018]

attempts to learn an implicit representation of object orientations. More recently, [Hu et al., 2021] introduces 6D pose estimation for objects in space. Category-level object pose estimation [Wang et al., 2019] problem is also gaining more interest, which aims to estimate 6D poses for unseen objects. The majority of above methods rely on real annotated data.

Learning Object Pose Estimation with Synthetic Images. Since annotating 6D poses is expensive [Hinterstoisser et al., 2012], a recent trend is to exploit 6D pose estimation with synthetic data. Some methods [Sundermeyer et al., 2018; Zakharov et al., 2019] train networks only using synthetic data. However, such methods generalize poorly on the real data due to the synthetic-real domain gap. To overcome this problem, one branch of works [Hodaˇn et al., 2019] proposes photorealistic and physically-based rendering techniques to bridge the domain gap. Another branch aims to employ domain adaptation techniques. [Sundermeyer et al., 2018] try to augment synthetic images with modifications (e.g., random background and varying lighting conditions), so as to learn domain-invariant attributes. These techniques can only mitigate the problem to some extent.

Self-supervised Learning in 6D Pose Estimation. Recent works exploit the field of self-supervised learning for 6D pose estimation. [Deng et al., 2020] proposes a self-supervised learning framework, where robots autonomously annotate real data for training. However, it requires robot interaction, which also needs a high cost. To overcome this limitation, Self6D [Wang et al., 2020] proposes a self-supervised learning framework which can train a pose estimation network on real unannotated RGBD data. They explore the consistency between the object rendered by the estimated pose and the observed object. However, the loss functions are highly nonlinear, which makes the learning process prone to local minima. Therefore, there is still a performance gap between Self6D and fully-supervised methods. [Sock et al., 2020; Wang et al., 2021a] propose self-supervised learning with color images. They utilize the pose and perceptual consistency to supervise pose estimation networks. Unfortunately, the consistency does not strictly reflect the pose quality, as the RGB domain can be affected by many conditions. More recently, [Wen et al., 2021; He et al., 2022] introduce selfsupervised learning for unseen objects. We focus on instance object pose estimation and follow the setting where unlabeled RGBD data is available during training. Our method almost eliminate the performance gap.

3 Method

Our learning framework is illustrated in Figure 2. The proposed framework consists of a pose estimator, a pose refiner, and a pose evaluator. During training, we first train our pose estimator on synthetic data. Afterwards, given a set of real unannotated images, the synthetically trained pose estimator estimates their poses as the initial poses for the pose refiner. Then the pose refiner utilizes the corresponding depth maps to optimize these imprecise initial poses. The proposed pose evaluator assesses refined poses' quality. Finally, the selected high-quality poses are used as supervision for our pose estimator. During inference, the pose estimator computes the 6D

pose of the target object from a color image. Compared to the setting of fully-supervised methods [Peng et al., 2019], our method replaces expensive manual 6D pose annotations with cheap depth maps during training time. In the following sections, we first look at the design of the pose estimator in Sec. 3.1 and then go into the details of the pose refiner and the pose evaluator in Sec. 3.2, which enables our method to learn pose estimation from depth maps.

3.1 Pose Estimation

The task of the pose estimator is to detect the object and determine its 3D rotation R and 3D translation t from a color image. Here we use PVNet [Peng et al., 2019] as the pose estimator in our framework. Note that we can also adopt other pose estimators [Rad and Lepetit, 2017; Song et al., 2020]. A straightforward way is to train PVNet on the synthetic data and apply it to the real data. However, we found that this strategy yields poor pose estimates on the real data, which cannot be refined to produce high-quality results by a pose refiner. One reason is that the background of the synthetic data is clean, while it is complex in the real data. To overcome this problem, we propose a two-stage pipeline, which first detects the object and then estimates the object pose from the cropped image patch. Adding an object detector also has other advantages: (1) It makes our pose estimator more scale-invariant, since an object detector crops the object and resizes the cropped object to a fixed size. (2) It is easier for our pose estimation networks to extract features due to the elimination of the background.

Specifically, we first detect the object and crop the object and resize it to a fixed size using CenterNet [Zhou et al., 2019]. Then a network is trained to detect projections of 3D object keypoints on the cropped image. After recovering these 2D key-point positions, we compute the 6D pose by solving the PnP problem. More specifically, to detect 2D object keypoints in an image, the network predicts pixel-wise object semantic labels and votes. Then we follow PVNet to use a RANSAC-based algorithm to locate 2D keypoints based on predicted semantic labels and offsets.

3.2 Learning Pose Estimation with DepthMaps

We aim to learn pose estimation with depth maps instead of 6D pose annotations. Compared to manually annotated 6D poses, the depth maps can be easily collected at scale. Specifically, we use a depth-based pose refiner to optimize inaccurate poses from the synthetically trained pose estimator. Then the proposed pose evaluator accesses the refined poses' quality and high-quality poses will be used as the supervision for the pose estimator.

Pose Refiner. Given a set of inaccurate poses, the pose refiner's task is to optimize for refined poses which best explain the observations. In fact, our framework can choose any pose refiner, and here we adopt the Iterative Closest Point algorithm (ICP) [Zhang, 1994] as our pose refiner, which is a commonly used pose refinement method in object pose estimation. Specifically, we first render a depth map Dr and a normal map N r , leveraging the 3D model and the estimated pose from the last step. To compute the optimization loss, we obtain pixels within the object region based on the predicted semantic mask. Using the rendered depth and observed depth, we can get the 3D observed point and rendered point for each pixel. The residual for each pixel is the smallest distance from the observed point to the plane of the rendered point, where the plane is defined by this rendered point and its normal. More specifically, the residual item defined on pixel p is

where π-1 is a back projection function. These residual items are minimized using gradient descent to get our expected pose. However, the ICP algorithm is sensitive to the quality of the input pose, which may make a result pose converge to the local minima. To overcome this problem, we generate a set of hypothesis poses by perturbing the input pose. Then we refine these poses based on Eq. (1) and select the best fit.

Pose Evaluator. Although the refined poses from the pose refiner are quite accurate, there still exist some failure poses. These imprecise poses may give wrong supervision to the pose estimator, making the training process unstable and decreasing performance. Therefore, we propose a pose evaluator to filter these failure poses. An intuitive idea is to evaluate the fit of the estimated pose with the observed scene and filter out poses of bad fit. We utilize geometric consistency to measure the quality of the estimated poses. Specifically, we use the optimization target Eq. (1) of our pose refiner as the measure for pose quality. Denoting this measure as Cgeom, we take poses whose Cgeom > C ∗ as failure poses, where C ∗ is a predefined threshold related to the object's diameter.

3.3 Implementation Details

Specifically, for M classes of objects and K keypoints, our network takes the H × W × 3 image as input and outputs H×W×(K×2×M+M+1) tensor. In the output tensor, K × 2 × M represents the vectors, while M + 1 represents the class probabilities. Following PVNet [Peng et al., 2019], we set K as 9. We set M as 13 to ensure the comparison fairness and the corresponding backbone is ResNet-34 in the row "Ours-AllObjects" of Table 1. We take M as 1 in all other experiments and the network backbone is ResNet-18. In practice, we set the pose quality threshold C ∗ to 0.2 times the diameter of the object.

Given supervision poses, we first compute supervision vectors to keypoints and semantic labels. These vectors are used to supervise the network using the smooth ℓ1 loss. For semantic labels, we adopt softmax cross-entropy loss. To ensure the quality of initial poses for the pose refiner, the networks are first trained on synthetic data. We take the initial learning rate as 1e-3 and halve it every 20 epochs. After pretraining, we set the initial learning rate as 5e-4 and halve it every 10 epochs to finetune the networks to employ learning with unannotated RGBD data. Every 5 epochs, we update the supervision poses of real training data using the pose estimator at the time and the pose refiner. All networks are trained until they are converged and it takes about 4 hours to train a pose estimation network and 2 hours to finetune it on 4 TITAN Xp gpus.

4 Experiments

4.1 Experiments Setup

Training Data. We train the pose estimator on the real training data without using annotations. Specifically, we follow Self6D [Wang et al., 2020] to use 15% of the real data on the LINEMOD dataset and 10% of the real training data on the YCB-Video dataset. The LINEMOD and YCB-Video datasets provide depth maps collected by RGBD sensors. Our synthetic data is rendered by BlenderProc.

Evaluation Datasets. We evaluate our method on the LINEMOD [Hinterstoisser et al., 2012], Occluded LINEMOD [Brachmann et al., 2014] and YCB-Video [Xiang et al., 2018] dataset, which are widely used benchmark datasets for object pose estimation. These datasets exhibits many challenges: the varying lighting conditions, image noise, and occlusions. We follow the setting of Self6D, which evaluate all objects of LINEMOD and Occluded LINEMOD datasets and 5 objects on the YCB-Video datasets.

Metric. We take ADD Metric as the evaluation metric, which is a standard metric for 6D object pose estimation.

It measures the average distance of the transformed model points by the estimated and the ground truth poses. We claim a pose is correct if the measure distance is less than 10% of the model's diameter. For symmetric objects, we rely on the ADD-S metric, where the measure distance is the average distance to the closest model point.

4.2 Comparison with the State-of-the-art

Performance on the LINEMOD Dataset. We compare our method with state-of-the-art RGB-based pose estimators on the LINEMOD dataset in Table 1. These methods can be divided into two camps based on whether real annotated data is used during training. Note that some methods [Li et al., 2019; Zakharov et al., 2019] train one network for all objects. To ensure the comparison fairness, we include the results under the same setting in the row "Ours-AllObjects". The results show that our method significantly outperforms methods without annotations by a large margin of 27.8%. Note that our method's performance of most objects is as good as or even better than methods with annotations except for the object Hole. Hole.'s 3D model is rather noisy than other objects in the dataset, making the refinement difficult.

Robustness to the Occluded LINEMOD Dataset. We evaluate networks that are trained for the LINEMOD dataset on the Occluded LINEMOD dataset. As shown in the Table 2, Our method outperforms synthetically trained methods and even achieves competitive results compared to state-of-the-art methods, which need an amount of real annotated data.

Generalization to the YCB-Video Dataset. To further show our method's generalization to other datasets, we conduct experiments on the YCB-Video dataset. We outperform Self6D in terms of all objects and improve the mean recall from 48.6% to 77.1% as shown in Table 3.

4.3 Flexibility of the Learning Framework

To demonstrate the flexibility of our framework, we implement it with an alternative pose estimator and refiner. We re-implement BB8 [Rad and Lepetit, 2017], which predicts the pixel locations of 3D bounding box's corners, using the heatmap representation, while the original BB8 directly regresses the coordinates. We also implement an additional refiner which renders a depth map using a differentiable renderer [Chen et al., 2019] and optimizes the pose by minimizing the geometry consistency defined in Self6D [Wang et al., 2020]. We refer to them as "BB8hm" and "DiffRefiner", respectively. The results are listed in Table 4. It can be observed that the proposed learning framework always significantly improves the results no matter what pose estimator and pose refiner are adopted.

4.4 Ablations and Analysis

Object Detector. To demonstrate the benefit of adding an object detector, we compare the pose estimator with the one of no detector. We train both of them on synthetic data and compare the pose estimation results on the LINEMOD dataset. As shown in the columns "w/o det." and "w/ det." below "Synthetic" in Tab, 5, adding an object detector strongly enhances the pose estimation performance of the pose estimator. Although our object detector is only trained on synthetic data, the pose estimator with detected boxes gives competitive results compared to the one with ground-truth boxes. To prove the claim that one of the main difference between the synthetic and real data lies in the background, we train both pose estimators on real data. The results are shown in the column "Real Anno." in Table 5. The two-stage design has smaller performance drop (92.8% vs. 70.5%) when trained on real annotated data and synthetic data than the single-stage design (89.1% vs. 56.6%), which supports our claim.

Pose Evaluator. To analyze the impact of the pose evaluator, we compare the proposed pipeline with the same pipeline of no pose evaluator, which means all refined poses are used as supervision for the pose estimator. We conduct experiments on the YCB-Video dataset and the results are listed in the columns "w/o filter" and "w/ filter" in Table 6. This demonstrate the proposed pose evaluator improves pose estimation performance.

Learning Framework. We explore the influence of the proposed learning framework on the LINEMOD, Occluded LINEMOD, and YCB-Video dataset. The results of the LINEMOD dataset are listed in the columns "w/ det." below "Synthetic" and "Real Unanno." in Table 5. Qualitative results of the LINEMOD dataset are shown in Figure 3. The results of the Occluded LINEMOD dataset are listed in columns Table 6: Ablation studies on the YCB-Video dataset. Syn. means only synthetic data are used to train the networks. w/o filter means all refined poses are used as supervision. w/ filter is our proposed pipeline, where only poses that are considered as success poses are used to supervise the networks.

"Syn." and "Ours" in Table 2. The accompanying qualitative images are shown in Figure 4. For the YCB-Video dataset, the results are listed in the rows "Syn." and "w/ filter" in Table 6. The results on above three benchmarks show the effectiveness of our learning framework. For some objects, our models even achieve better performance than models trained on real data with ground truth annotations. The possible reason is that there are error annotations in the LINEMOD training data (e.g., cat sequence: frame 34, 37, 41, etc.)

5 Conclusion

In this paper, we introduced a simple yet effective learning framework utilizing the accurate results of the depth-based pose refinement method to supervise the RGB-based pose estimation network. We achieve a significant performance gain compared to recent self-supervised methods. Without using any real labeled data, our method achieves competitive results on par with current leading approaches with real annotations. Furthermore, an amount of experiments are conducted to prove the flexibility of the proposed framework and analyze the impact of main components in the proposed framework. The limitation is that our method needs depth maps and CAD models during training on unlabeled data. It would be interesting to overcome the need for them in the future.

Acknowledgments

The authors would like to acknowledge the support from the National Key Research and Development Program of China (No. 2020AAA0108901) and the ZJU-SenseTime Joint Lab of 3D Vision.