Pre-defined Keypoints Promote Category-level Articulation Pose Estimation via Multi-Modal Alignment

1 Introduction

Articulated objects are common in daily life, encompassing household items like scissors and cabinets, as well as office tools such as laptops. These objects are composed of multiple rigid components linked by joints, setting them apart from standard rigid objects through their kinematic constraints. This unique structure complicates articulation pose estimation, making it more challenging than for rigid objects. Accurate and efficient pose estimation of articulated objects remains a crucial hurdle in various downstream applications, particularly in robot manipulation [Xiong et al., 2023], human-object interactions [Liu et al., 2021], embodied AI [Yu et al., 2023; Guo et al., 2024], and augmented reality [Amin and Govilkar, 2015].

∗Corresponding author: Liu Liu.

Traditionally, the 6D object pose estimation problem has been addressed by matching points between 3D models and images. In recent years, the advent of low-cost RGB-D sensors has made it possible to infer poses for low-texture objects. In low-light environments, RGB-D methods [Wang et al., 2019] have demonstrated higher accuracy compared to pure RGB methods. Furthermore, as an alternative to directly regressing keypoint coordinates, keypoint-based methods have proven to be highly effective for pose estimation [Li et al., 2021], and can be seamlessly applied to multimodal input for 6D object pose estimation. Despite progress, category-level articulated pose estimation remains a challenging and critical task, as existing methods still encounter several unresolved issues:

(i) Multi-Modal Fusion. 3D object models provide geometric information, while RGB images offer texture and color details. Traditional fusionmethods (e.g., [Wang et al., 2019]) integrate features from these modalities but often fail to address their distinct data distributions and feature representations. This oversight can result in information redundancy or mismatches, leading to learning difficulties or convergence failures. Although subsequent works, such as [Shivakumar et al., 2019], attempt to address this issue using depth maps generated from projections, these maps lack the rich color and texture information of RGB images, as they only represent 2D plane projections. Similarly, RGB images lack depth information, capturing only 2D color distributions. Multi-modal fusion thus remains a significant challenge, further complicated by factors like viewpoint changes, background noise, and varying lighting conditions.

(ii) Keypoint Estimation Manner. Keypoint-based methods are widely used in domains like human and animal pose estimation. Recent advances include heatmap-based [Sun et al., 2020] and sequence-based [Lin et al., 2022] techniques, which, despite differing mechanisms, belong to the regression paradigm. However, these methods face similar challenges: 1) The distribution of distances between visible points and keypoints is overly scattered, resulting in nearby points having a stronger sensitivity to positional features, making them easier to regress. 2) Dependency on global features: Reliance on global features makes them vulnerable to noise and occlusion, leading to inaccurate keypoint predictions.

To address the first issue, we propose a geometry-guided, image-enhanced fusion module that aligns point cloud and image features. Point clouds provide geometric structure, while images offer rich texture. By projecting point clouds into depth maps to locate regions of interest in image features, we extract pose-sensitive features via interpolation and selection. A tailored fusion module then filters redundancy and noise, ensuring semantic consistency across modalities.

To address the second issue, we reformulate the task as proposal integration and scoring (see Fig. 1). The network regresses a radius per visible point, denoting its Euclidean distance to the keypoint, and maps it into a 3D accumulator space to generate robust, independent proposals per grid, even under occlusion. Additionally, pre-defined keypoints enhance performance by yielding more uniform radius distributions compared to heuristically generated ones.

In summary, to address the category-level articulated object pose estimation problem under multi-modal representation, we propose a Pre-defined Keypoint multi-modal AliGnmEnt framework, named PAGE. The core of our method lies in a novel geometric-color alignment process, pre-defined keypoint estimation mechanism, and proposal integration scoring strategy. In practice, to validate the effectiveness of our method, we conduct extensive experiments on multiple datasets, including synthetic, semi-synthetic, and real-world datasets. We believe these experimental results demonstrate the superior performance and robustness of our PAGE. Our main contributions can be concluded as threefold:

• We propose the PAGE, a novel Pre-defined Keypoint multi-modal AliGnmEnt framework for category-level articulation pose estimation.
• In PAGE, Color features are leveraged to complement geometric information, forming the enhanced geometric-color alignment. To avoid the weakness of traditional heuristics and regressing methods, we propose the customized Keypoint estimator and proposal integration scoring scheme.
• The experimental results from synthetic (ArtImage), semi-synthetic (ReArtMix), and real-world datasets (RobotArm) all demonstrate the effectiveness and robustness of the proposed PAGE.

2 RelatedWork

2.1 Pose Estimation fromDifferent Modalities

Pose estimation methods can be categorized by input data type: 1) RGB-Based: These methods estimate 6D poses from color or monochrome images, typically via 2D detection or keypoint localization [Baek et al., 2019]. 2) Depth-Based: Depth images are converted into partial point clouds using camera parameters. Approaches include reinforcement learning [Liu et al., 2023] and regression models [Li et al., 2020]. 3) RGB-D-Based: These methods fuse depth and RGB images through various levels and paradigms. For example, DenseFusion [Wang et al., 2019] employs dense pixel-wise feature embedding.

In summary, RGB-only methods are sensitive to lighting, causing performance drops in low illumination, while depthonly methods struggle to differentiate objects with similar shapes but different colors, leading to ambiguities. To overcome these issues, we adopt an RGB-D manner. Instead of traditional ways, we emphasize feature alignment to reduce redundancy and improve downstream task performance.

2.2 Keypoint Based Pose Estimation

Keypoint-based methods are widely applied in various fields, such as human [McNally et al., 2022], animal [Gong et al., 2022], and object estimation [Xue et al., 2021]. The general workflow of these methods typically begins with a keypoint regression network, where the core task is to estimate the keypoint locations of an object in different spaces. The object's pose is then inferred based on the correspondences of these keypoints. The key advantage of these methods lies in their heuristic algorithm (e.g., FPS, BBox) design, aiming to generate geometrically dispersed keypoints. To enhance robustness, this process often integrates classical algorithms, such as RANSAC [Besl and McKay, 1992; Horn et al., 1988; Fischler and Bolles, 1981]. Keypoint-based approaches have been proven to be one of the most accurate solutions for 6D pose estimation, and recent research has further advanced the development of these methods. Generally speaking, it can be divided into the following paradigms: 1) Detection. i.e., detect the position of key points through images or sensor data (such as [Qiao et al., 2017], [Nguyen et al., 2022]). 2) Matching. i.e., match key points between different images or frames for tracking or alignment (e.g., [DeTone et al., 2018], [Rockwell et al., 2024]). 3) Generation. i.e., Generate the location or heatmap of key points ( [Zhu and Ye, 2024]).

Although previous methods vary in their implementation, most can be regarded as regression methods. These methods rely on clear and well-defined input-output mappings and are heavily dependent on high-quality data and robust global features. Furthermore, this paradigm is highly sensitive to outliers, which can lead to biased network convergence and a lack of robustness in noisy and occluded scenarios.

3 Problem Statement

In this paper, our goal is to address the Category-level Articulation Pose Estimation task based on RGB-D modalities (CAPE task). The core idea is to predict the pre-defined keypoints through a scoring mechanism to determine the articulated pose. To this end, we define a new paradigm for the CAPE task, named PAGE. Specifically speaking, given the partial point cloud P of an articulated object A = {δk}Kk=1 (where {δk} represents the k-th rigid part) and its corresponding 2D observed image I as the input, the output objectives of our PAGE are as follows: (i) per-part 3D rotation R(k) ∈ SO(3); (ii) per-part 3D translation t(k) . Together, the rotation and translation parameters form the 6D pose estimation result, denoted as T = {R(k) , t(k) }Kk=1 ∈ SE(3).

The proposed PAGE framework operates as follows: point clouds within the intra-category are processed by a backbone optimized using geometry-sensitive loss functions to generate pre-defined keypoints GT. Then the partial point cloud P ∈ RN×3 = {pi}Ni=1 is projected into a depth image using the camera mapping parameter M, establishing a 1-to-1 mapping with the 2D image I. This filters redundant information, yielding point-wise color features FC, which are processed by the GC-Fuser to produce re-modulated features FCG. These features are used to either decode the articulation segmentation mask (optional) or directly predict the radius ri for each visible point pi. Per-part keypoints K(k) = {κi}3i=1 are predicted as follows: if the error between the predicted radius and the corresponding grid's distance is within a threshold η, the corresponding proposal count is incremented by 1. Then each grid's score Si is computed, and the highest-scoring grid's center is selected as the keypoint. Finally, the ICP algorithm outputs the estimated 6D pose {R(k) , t(k) }Kk=1 .

4 Methodology

4.1 Pre-defined Keypoint Estimation

As highlighted in Sec. 2, the core of traditional keypointbased methods lies in their heuristic algorithms, which aim to generate geometrically scattered keypoints, typically placed near the object's surface. These keypoints exhibit a nonlinear distribution, meaning they aren't arranged along a single line but are scattered in multiple directions, aiding in the subsequent pose transformation recovery through correspondences between keypoints and image points. However, we observe that the distance distribution between heuristic keypoints and visible points follows a distinct divergent pattern, a phenomenon that has not been adequately addressed in previous studies. Therefore, this work shifts its focus to the generation of pre-defined keypoints. We find that by training a Graph Network, a set of geometrically sensitive scattered keypoints can be selected, which significantly enhances the accuracy and efficiency of object pose estimation. Compared to previous heuristic keypoint algorithms, this method provides a more precise prediction of articulated object poses.

As shown in Fig. 2, our goal is to train the network such that the radius (i.e., the Euclidean distance between visible points and keypoints) distribution of each keypoint set is as similar as possible. To achieve this, we draw inspiration from the design of [Qiu et al., 2020] and employ GraphNet. Note that the input module consists of the complete point clouds of the same category, while the output is a keypoint distribution that is spatially almost consistent. This approach leads to a pre-defined keypoint generator that is specific to the current category. High-quality 3D keypoints should possess geometric sensitivity in order to more accurately capture the pose of articulated objects. Therefore, during the GT keypoint estimation process, we consider the following key factors: 1) Minimum Distribution Difference: Heuristically generated keypoints often exhibit a large variance in the distance distribution to visible points. This causes the model to develop a preference for utilizing point cloud regions near the keypoints (where features are denser) and avoiding regions farther away (where features are sparser), leading to local optimality or significant prediction errors. 2) Divergent Layout: Keypoints should avoid clustering in close or identical locations. Instead, they should be distributed as evenly as possible to more comprehensively represent the geometric features of the object. 3) Geometric Inclusiveness: The distribution of keypoints should not be too far removed from the object, as this would lead to a lack of shape meaning and may degrade the object representation to mere points.

Thus, we propose the KP-Estimator, which conducts customized design and constraints as follows: 1) We aim for the generated keypoints to have similar radius clusters (i.e., the Euclidean distance clusters between each keypoint and each visible point), meaning that the radius distribution for each pair of keypoints should be as close as possible. To achieve this, we introduce the Wasserstein loss Lwass [Wu et al., 2024], which goes beyond WassGAN-GP [Gulrajani et al., 2017] and in the same spirit inspired by involving both a critic loss and gradient penalty. The Wasserstein loss Lwass between two keypoint radius sets Ri and Rj is defined as:

Where D(Ri) and D(Rj) represent the histogram distributions of radius, Ri and Rj represent the radius group of the respective keypoints κi and κj. D(R) denotes the joint distribution of Ri and Rj, while ∇R is the gradient calculated from R, and λ is the gradient penalty hyperparameter. Unlike WassGAN-GP [Gulrajani et al., 2017], which applies the gradient penalty only when the generator learns to imitate the GT, we apply the gradient penalty to all sets of radius values. This helps to enhance the smoothness and stability of the predictions, thereby optimizing overall performance.

2) To prevent keypoints collapse, we introduce the per-part separation loss, Lsep, which ensures that multiple keypoints do not converge at the same location.

where ζ is the threshold. Note that ζ is slightly larger than the distance between keypoints sampled by FPS, because our key points are not distributed on the surface of the object, but have a certain distance.

3) Finally, to ensure comprehensive coverage of the rigid part's shape geometry by the learned keypoints, we introduce the per-part coverage loss, Lcov. This loss is based on the difference between the volume V(·) of the keypoints, P (k) , and the input point cloud, X(k) , while also penalizing keypoints that are distant fromX(k) :

The total loss, Lttl, for KP-Estimator, which estimates selfsupervised per-part 3D keypoints, is defined as the weighted sum of the Wasserstein loss Lwass, separation loss Lsep, and coverage loss Lcov, with corresponding weights λ1 = 0.6, λ2 = 0.2, and λ3 = 0.2, as follows:

Through this approach, KP-Estimator is trained to identify a set of dispersed keypoints with uniformly distributed radii. This results in a unified set of keypoints applicable to all objects in the dataset, which can subsequently be utilized in keypoint-based 6D pose estimation methods. Experimental results show that the keypoints selected by KP-Estimator improve the accuracy of most evaluation metrics (details can be found in the ablation study).

4.2 Geometric-Color Alignment

Feature alignment and fusion is not a trivial task, as many existing methods directly perform fusion while neglecting the importance of proper alignment. To address this, we propose a novel multi-modal alignment and fusion method, called GCA (Geometric-Color Alignment). Specifically, the proposed GCA consists of three steps: (1) Depth-RGB Image Registration, (2) Redundancy Minimization (i.e., feature selection), and (3) Geometry-Color Fusion (GC-Fuser).

(1) Depth-RGB Image Registration. Firstly, the partial PC will be projected into a 2D depth image with the help of the camera mapping matrix (denoted as M). Afterward, we conduct the registration by bilinear interpolation, and then the 1-1 correspondence between depth and RGB images can be easily established at different resolutions. Mathematically, for a specific point p in the point cloud, we can calculate its corresponding position p′ (u, v) in the depth image. This process can be expressed by the projection equation p′ = M ×p. (2) Redundancy Minimization. With the 1-1 registration between Depth and RGB images, our target is to extract RGB features strongly correlated with each visible point. Formally, taking the sampled position p′ and the image feature map FI as inputs, generating point-wise color feature representations FC for each sampled position. Thus, we filter out the redundant mutual information. In practice, since the sampling position may lie between adjacent pixels, we employ interpo-

lation to obtain the color features at continuous coordinates.

(3) Geometry-guided Color Fuser. To address the aforementioned fusion issue, we propose the GC-Fuser (Geometry-guided Color Fuser) that adaptively estimates the importance of color features on a per-point basis, guided by geometry features. The specific process is outlined in Fig. 3: first, geometry features FG and per-point color features FC are projected into modality-aware feature spaces independently. Then, we further conduct mutual information minimization between them to explicitly promote the learning of complementary information. Next, these two branches are fused into a compact feature representation, which is then compressed through another fully connected layer, ultimately generating an adaptive weight map w. We normalize this weight map to the range [0,1] using the softmax function. This procedure can be formulated as:

where W, U, V denote the learnable weight matrices in our GC-Fuser. S() represents Softmax and H() represents the output activation.

After obtaining the adaptive weight map w, we combine the geometry features FG ∈ RN×C1 and semantic-related color features wFC ∈ RN×C2 in a concatenation manner. The specific process can be expressed as:

where || represents concatenation, C = C1 + C2.

4.3 Proposal Integration Scoring

The distance voting method [Wu et al., 2022] has been proven to be an effective position inference algorithm. By adhering to the inclusion similarity between the geometric structure of visible points and the complete point cloud structure, it leverages local structures to infer global pose information. Building on this, we further propose a novel 3D proposal integration scoring method. The core idea is: in the 3D accumulator space, the number of valid proposals from visible points is counted for each grid, and the confidence score is calculated by aggregating these proposals.

For each rigid part, we output a set of keypoint proposals along with their corresponding aggregation scores independently. To more intuitively illustrate the proposal integration scoring mechanism, we draw an analogy between each visible point in the point cloud and a Member of Parliament (MP), whose primary task is to raise proposals for each grid in the 3D accumulator space. In this procedure, considering the partial point cloud P ∈ RN×3 (input), we utilize the fused features FCG to output 3N channels, which represents the predicted distances (radii) from each visible point to the three pre-defined keypoints. Furthermore, we define the resolution of the 3D accumulator space as ρ, which corresponds to the edge length of the smallest unit grid voxel. If the absolute error between the computed radial distance and the estimated keypoint radius is less than or equal to ρ, the proposal is considered valid, and the proposal count for the corresponding grid is incremented by 1; otherwise, the proposal is deemed invalid, with the count remaining at 0. We adopted the confidence function from [Tekin et al., 2018] and applied it to our proposal function, as expressed mathematically below:

Where 1(·) is the indicator function.

Subsequently, the proposal count for each grid is aggregated to compute the confidence score. Here, the confidence score is defined as the ratio of the current grid's proposal count to the total proposal count. The grid with the highest score is selected as the final result (Eq. 8), and its center point is considered as the elected keypoint ˆK (Eq. 9).

Overall, when the network is sufficiently trained, the majority of the proposals generated by local feature points are able to accurately localize the target keypoint, and the final center point estimation converges to the ground truth. Even in the presence of a small amount of noisy votes, no significant deviation occurs, which demonstrates the robustness of our method. From another perspective, our aggregation step effectively estimates the probability density of the center point position in the spatial domain (i.e., the keypoint) and determines its location accordingly. Notably, this process is invariant to rigid transformations (i.e., SE(3) invariance), ensuring stability even when the object pose changes, and maintaining a prominent peak in the distribution.

5 Experiments

Datasets, Baselines, and Metrics. We evaluate our PAGE framework on the ArtImage [Xue et al., 2021], ReArtMix [Liu et al., 2022], and RobotArm [Liu et al., 2022] datasets, covering synthetic, semi-synthetic and real-world scenarios. For performance comparison, we benchmark against four RGB+Depth-based methods: A-NCSH [Li et al., 2020], Densefusion [Wang et al., 2019], ASMM [Zhang et al., 2025]. The evaluation metrics include degree error for 3D rotation, distance error for 3D translation, and 3D Intersection over Union (IoU) for measuring 3D scale.

Implementation Details. During the pre-processing, the input RGB images are scaled into 224×224 resolution and the point clouds are downsampled into 2,048 points before being fed into the network. Please note that the input modality of all methods have been adjusted to depth + color, and re-training was conducted under the same experimental settings compared to ours. When training the backbone (PointNet++ [Qi et al., 2017] ), the Adam optimizer was employed with an initial learning rate of 0.001 and a weight decay of 0.0001. The learning rate decayed by a factor of 0.5 every 20 epochs. All experiments were conducted on four NVIDIA GeForce RTX 4090 GPUs, each with 24GB of memory.

5.1 Comparison with the SOTAMethods

In this section, we conduct comparative experiments in the synthetic dataset (i.e., ArtImage) with the classical methods, aiming to verify the effectiveness of our PAGE. Tab. 1 demonstrates the quantitative results. 1) When considered as a whole, our method has comprehensively refreshed the SOTA results, which proves the effectiveness of the coordination of each proposed module. 2) When considered separately, we get the best pose estimation result lies in category laptop, with

(4.0◦ , 1.7◦ ), (0.014m, 0.019m) for rotation error and translation error. This can be attributed to our keypoint generation strategy, which can better generalize to objects with uniform geometric scales. Considering the 3D IoU metric, our prediction errors are significantly better at each part compared to the baseline model (A-NCSH), with the average value 57.5% vs. 48.5%. Qualitative results in Fig. 4 (left) further highlight that our predictions closely align with GT, confirming the robustness and precision of our approach.

5.2 Ablation Study

Pre-defined Keypoints. As mentioned in Sec. 4.1, we propose that methods of pre-defined keypoints can outperform the heuristic methods. Therefore, we conduct detailed ablation experiments in this section. Results are shown in Tab. 2 (I - III). It is noted that the FPS and BBox based keypoint generation methods are adopted from [Zhao et al., 2020]. From the quantitative results, we can conclude that: our method significantly outperforms the heuristic methods (i.e., FPS and BBox based methods) by a large margin, which reported (1.9◦ and 5.4◦ ) and (0.013m, 0.032m) with our methods. All in all, the proposed method not only addresses the distribution of keypoints but also takes into account the geometric and structural relationships of the articulation, optimizing through three geometry-sensitive loss functions (see Sec. 4.1). This is crucial for modeling the pose of objects based on keypoints.

Fusion Manner. In this work, we propose a novel multimodal learning method coined geometric-color alignment as detailed in Sec. 4.2. To verify the effectiveness of the proposed module, we conduct the ablation experiments in Tab. 2 (IV - VI). Note that configuration IV represents the direct concatenation of depth and RGB features. Configuration V represents the method from densefusion [Wang et al., 2019]. It can be inferred that our method achieves state-of-the-art performance, while the traditional method (i.e., direct concatenation) suffers from severe performance degradation due to noisy feature alignment.

5.3 Generalization Capacity

Experiments on Semi-Synthetic Scenarios. ReArtMix dataset is used to evaluate our method, which incorporates semi-synthetic scenarios. The detailed results are presented in Tab. 3. Our approach achieves the best performance on the Drawer category, with rotation error of only 1.2◦ , and translation error of 0.013m and 0.017m. This demonstrates that our method is equally effective in Semi-Synthetic Scenarios, showcasing its robustness and adaptability across diverse object categories and scenes. Qualitative results for the five categories are illustrated in Fig. 4 (Right).

Experiments on Real-world Scenarios. We further train and evaluate PAGE on the 7-part RobotArm dataset in realworld scenarios. As shown in the quantitative results (Tab. 4), our method achieves superior performance in per-part pose estimation compared to the baseline A-NCSH. Specifically, PAGE significantly reduces both rotation and translation errors. For rotation errors, PAGE achieves notable improvements across parts 1 to 7, with average rotation errors reduced to 7.5◦ compared to 12.1◦ of A-NCSH. Similarly, for translation errors, PAGE maintains an average error of 0.044m, outperforming the baseline's 0.126m. While accumulative errors are observed in deeper kinematic structures (5th, 6th, and 7th parts), our method demonstrates greater robustness against these challenges compared to the baseline, which exhibits more significant accumulative errors in both rotation and translation. Qualitative results are illustrated in Fig. 5.

6 Conclusion

In this work, we propose a novel framework, PAGE, to address category-level articulation pose estimation through multi-modal alignment. Our approach focuses on sequentially aligning geometry and color features, followed by effective feature fusion. To overcome the limitations of heuristic keypoints, we introduce a tailored pre-defined keypoint estimation method that enhances pose estimation performance. Additionally, a proposal integration scoring strategy is employed to accurately determine the 6D pose of articulated objects. Experimental results demonstrate that PAGE achieves state-of-the-art performance on the synthetic ArtImage dataset and exhibits strong generalization capabilities on the semi-synthetic ReArtMix dataset and real-world multi-hinged articulated object datasets (e.g., RobotArm). These results underscore the robustness and adaptability of our method across diverse scenarios.

Acknowledgments

This work was supported in part by National Natural Science Foundation of China under Grant 62302143 and Anhui Provincial Natural Science Foundation under Grant 2308085QF207