SeqPose: An End-to-End Framework to Unify Single-frame and Video-based RGB Category-Level Pose Estimation

1 Introduction

Category-level object pose estimation is a fundamental computer vision task and crucial for 3D perception and understanding [Liu et al., 2024]. It aims to accurately estimate the locations, orientations, and metric sizes of the objects w.r.t. the camera in given images, depth maps and point clouds [Zheng et al., 2024; Sun et al., 2024]. In practice, automatic estimation of the rotation and translation of an object can facilitate applications, such as interactions in augmented reality [Su et al., 2019; Wen et al., 2023], robotic manipulation [Liu et al., 2023a; Yu et al., 2023; Liu et al., 2023b; Sun et al., 2022] and autonomous driving [Hoque et al., 2023; Sun et al., 2023]. Recent advances in RGBD-based categorical object pose estimation have achieved remarkable progress [Wang et al., 2019; Liu et al., 2023c; Chen et al., 2024; Lin et al., 2024; Wang et al., 2024]. However, RGBDbased methods largely depend on depth sensors [Zhang et al., 2024], which limits their applications in practice, especially for RGB-only settings without depth devices [Liu et al., 2024]. Recently, RGB-based categorical pose estimation has also demonstrated its wide application scenarios equipped with mobile devices, such as AR headsets, smartphones, etc., and become a promising trend [Liu et al., 2024].

∗Corresponding author: yqzhang@gdut.edu.cn

RGB-based methods [Manhardt et al., 2020; Lee et al., 2021; Fan et al., 2022; Wei et al., 2024] have been proposed in recent years. Concretely, the RGB-based category-level pose estimation methods can be roughly classified into two main paradigms: 1) two-stage direct regression methods, and 2) multi-stage normalized object coordinate space (NOCS) based methods (see Figure 1 (a)). Particularly, two-stage direct regression methods achieved pose estimation by leveraging multiple separate models, such as object detector and pose regressor. Multi-stage NOCS-based methods estimate the shape variations by capturing the correspondence between the observed objects and NOCS representation. Object pose is calculated using the Umeyama [Umeyama, 1991] and Perspective-n-Point (PnP) [Lepetit et al., 2009] algorithms.

However, existing RGB-based approaches struggle with multi-stage settings and heavily rely on intermediate results of single-frame object detection and depth estimation. Furthermore, the dependency on off-the-shelf techniques, including 2D object detectors, depth estimators, non-differentiable NOCS shape alignment, etc., leads to cumulative errors, degrading the performance of accuracy and stability of the pose and size estimation. Additionally, the multi-stage settings and extra dependencies complicate the whole pipeline and impose significant limitations on their practical applications and deployment in real-world scenarios. It thus motivates us to streamline a compact yet efficient end-to-end framework for category-level 6D pose and size estimation.

To achieve this goal, we propose SeqPose, a SpatialtEmporal sparse Query network for RGB-based categorylevel object Pose estimation. Our key insight is that: 1) Despite the limited information and ill-posed problem setting, advanced feature extractors can be used to effectively map RGB inputs to implicit robust semantic and geometry feature space without explicitly introducing object detectors or depth estimators; 2) The idea of using the spatial-temporal correlations for RGB-based categorical object pose estimation remains underexplored. It is promising to introduce spatialtemporal information as a strong prior, to correct geometry errors and stabilize the pose and size estimation results; 3) The intra-frame semantic and geometry feature can be aggregated by leveraging a set of sparse 3D query anchors and propagating inter-frame correspondence, which thereby unifies single-frame and video-based RGB category-level object pose estimation. Our SeqPose comprises three main modules: (a) A hierarchical multi-scale feature fusion module that leverages the DINOv2 encoder and a depth decoder to integrate robust intra-frame semantic and geometric features. (b) A spatial-temporal sparse query network for aggregating intra-frame features by a set of inter-frame correlated 3D query anchors. (c) A pose prediction head to directly estimate pose and size from the learned queries. To this end, our SeqPose can be trained end-to-end and enables shape-prior-free categorical pose estimation. Experimental results show that our model can achieve superior performance in estimating 6D pose and size for category-level objects on both single-frame and video-based RGB benchmark datasets. Our main contributions can be summarized as follows:

• To the best of our knowledge, we are the first to unify single-frame and video-based RGB category-level object pose and size estimation into a single-stage end-toend framework.
• We build a DINOv2-based encoder-decoder network to learn implicit semantic categorical prior and instancespecific geometric representations by integrating hierarchical intra-frame features.
• A spatial-temporal sparse query-based network is presented to aggregate intra-frame semantic and geometry

features and propagate inter-frame correlations by leveraging implicit 3D query anchors.

2 RelatedWork

RGBD-based Method. Initially, Wang et al. [Wang et al., 2019] introduced the NOCS to align all objects with the same category in a shared 3D canonical representation. Specifically, in [Wang et al., 2019], Mask R-CNN [He et al., 2017] is adopted to segment objects and an extra head is designed to predict the 2D projected NOCS map. Then, the given depth map is used to capture the 3D-3D correspondence by backprojecting the predicted NOCS map. However, the intraclass shape variations significantly impact the performance of 6D pose and size estimation for shared shape prior-based methods. Therefore, Chen et al. [Chen et al., 2020a] proposed to learn an implicitly canonical shape space (CASS) using a variational auto-encoder. Similarly, Tian et al. [Tian et al., 2020] proposed to reconstruct the observed 3D object to model the deformation. Furthermore, Wang et al. [Wang et al., 2021a] presented a cascaded relation and recurrent reconstruction network to accurately recover the instance 3D model in the shared canonical space, Chen and Dou [Chen and Dou, 2021] developed a structure-guided prior adaptation scheme to adapt the 3D canonical prior model.

The major challenges for category-level pose estimation are the intra-class shape and texture variations. Matching the instance-specific shape to the mean shape prior efficiently and accurately becomes the focus for RGBD-based approaches. Moreover, the shared canonical space, i.e., NOCS for each category should be obtained in advance, and instance-specific geometry representation , such as patch-wise point cloud and depth map, should be calculated online or acquired offline.

RGB-based Method. In practice, depth information may be absent. It is crucial for achieving high-precision categorylevel pose estimation using RGB images in real applications [Liu et al., 2024; Chen et al., 2020b; Manhardt et al., 2020; Fan et al., 2022; Wei et al., 2024]. Specifically, Chen et al. [Chen et al., 2020b] proposed a VAE-based model to reconstruct the RGB image conditional on the pose. Manhardt et al. [Manhardt et al., 2020] presented CPS++, a selfsupervised model for class-level pose and metric size estimation. Lee et al. [Lee et al., 2021] proposed a two-branch network to simultaneously regress the metric scale and NOCS map. Similarly, Fan et al. [Fan et al., 2022] developed OLDNet to predict the depth and NOCS representation from an RGB image for pose estimation. Wei et al. [Wei et al., 2024] decoupled the 6D pose and size estimation by designing a 2D-3D correspondence learning and a metric scale recovery module. Zhang et al. [Zhang et al., 2024] proposed the Laplacian mixture model to represent the object shape for pose estimation. Li et al. [Li et al., 2024] presented a coarse-to-fine method that leveraged geometry supervision for coarse 3D feature extraction and refined the feature using pose and size constraints.

Generally, RGB-based methods depend heavily on object detection, depth estimation, and non-differentiable sharedshape-prior matching and optimization, which separates the pipeline into multiple stages and hinders these approaches from being integrated into an end-to-end framework imposing limitations in practical application and deployment.

LargeVisionModels for Pose Estimation. Recently, large vision models (LVMs) have achieved remarkable progress, driven by rapid advancements in self-supervised learning techniques and the availability of large-scale datasets. DINO [Caron et al., 2021; Oquab et al., 2023], as a representative LVM, has gained considerable attention due to its extraordinary performance in producing semantically consistent patch-wise features. The DINO feature has been explored for detection [Liu et al., 2023d], depth estimation [Yang et al., 2024a; Yang et al., 2024b], pose estimation [Chen et al., 2024], etc. Specifically, Chen et al. [Chen et al., 2024] proposed to learn categorical SE(3)-consistent features by integrating semantic category priors extracted using DINOv2, with object-specific geometric features. Our implemented model is closely related to [Chen et al., 2024]. However, instead of introducing categorical shape priors, we propose to learn the intra-frame hierarchical representations of the semantic category priors and instance-specific geometric features produced by the DINOv2 encoder [Oquab et al., 2023] and DepthAnythingV2 decoder [Yang et al., 2024b], respectively. Moreover, a spatial-temporal sparse query network is designed to learn implicit 3D anchors and estimate the 6D pose and size by aggregating the intra-frame features with spatial-temporal correlated queries.

3 Method

3.1 Model Overview

Given an input sequence of monocular RGB images [I(1) , I(2) , ..., I(t) , ..., I(T) ], our goal is to estimate the 3D location, orientation and size of objects in each image directly. The overview of the proposed SeqPose is illustrated in Figure 2. It mainly consists of three modules: (a) A hierarchical feature fusion module is presented to learn and integrate robust semantic and geometry features using DINOv2 and the depth decoder of DepthAnythingV2, respectively. (b) A spatial-temporal sparse query network is designed to aggregate the semantic and geometry features by learning a set of inter-frame correlated 3D query anchors. Finally, (c) A 9-DoF prediction head is introduced to estimate the pose and size directly from the learned queries using the bipartite matching algorithm [Wang et al., 2021b].

3.2 Hierarchical Semantic and Geometry Feature Fusion

Instead of using a CNN-based object detector, we adopt the encoder of DINOv2 [Oquab et al., 2023] and decoder of DepthAnythingv2 [Yang et al., 2024b] to produce intra-frame latent semantic and geometry feature. Our key assumption is that features produced by the DINOv2 encoder can be regarded as robust semantic priors to understand objects. Moreover, DepthAnythingv2 also demonstrates its superiority in recovering geometry by estimating accurate depth maps. In particular, the encoder of DINOv2 is responsible for extracting semantic features of an input image:

where I(t) is the t-th frame in a given RGB sequence. s(t) i denotes i-th level of features produced by the encoder of the DINOv2 (DINOv2enc(·)). Then, the decoder of DepthAnythingv2 with convolutional layers (Depth dec (·)) is exploited for iteratively upsampling the hierarchical geometry-aware depth features, which can be expressed as:

where g(t) i refers i-th level of geometry features produced by the Depth dec (·), which is a hierarchical feature pyramid structure with skip connection ADD(·) and residual convolutional blocks FUSEi(·). We ignored the upsampling and reshaping operations to simplify the Equation. Finally, the semantic and geometry features are integrated by:

where z(t) i refers to a type of 2.5D features that integrate the i-th level of intra-frame semantic and geometry features. CONVi, CAT denote 1-by-1 convolutional layer and concatenation operation, respectively.

To learn semantic and geometry features, three separate pixel-wise dense prediction heads are introduced, including a depth head, a normal head and a semantic head to predict depth map ˆP(t) depth , normal map ˆP(t) norm, and semantic mask ˆP(t) seg, respectively:

3.3 Spatial-temporal Sparse Query Network

Our proposed framework's core is a spatial-temporal sparse query (STSQ) network, which learns to capture the implicit correspondence between a set of learnable 3D query anchors and the integrated intra-frame 2.5D features. Inspired by the query-based models in object detection [Carion et al., 2020; Wang et al., 2021b; Wang et al., 2023], our proposed STSQ, as illustrated in Figure 2, consists of three main components: 1) A sparse query-based memory queue to learn a set of spatial-temporal correlated 3D query anchors. 2) A temporal query cross-attention module for modeling inter-frame interactions between 3D query anchors across different time steps. 3) A geometry-aware cross-attention block is introduced for aggregating intra-frame hierarchical 2.5D features guided by the estimated depth and normal map.

(a) Sparse Query-based Memory Queue. We initialize a set of implicit 3D query anchors Qm to all zeros at time step 0 as the sparse query-based memory queue. The number of queries in the memory queue is set to N ×K, where N denotes the number of historical frames, K denotes the top-K candidate query anchors stored in the memory queue for each frame of a sequence. During training, the memory queue Qm is updated according to 1) input frames within the sequence [t -∆t, t -1], and 2) the top-K queries within each frame ranked by the confidence score predicted by the pose prediction head. This can be formulated as:

where TOPK(·) denotes the selection of top-K candidate queries for each frame in the sequence [t -∆t, t -1]. Q(t) c is the implicit 3D query anchors for current frame. Furthermore, the top-K object queries of the previous frame in the memory query will be selected and concatenated with the 3D query anchors Q(t) c to ensure high recall:

(b) Temporal Query Cross-attention. Once we get the sparse query Q(t) m in the memory queue and the implicit 3D

query anchors ˆQ(t) c , the temporal queries will be propagated to the current frame by using a cross-attention block:

˜Q(t) c = CA(ψquery( ˆQ(t) c ), ψkey( ˆQ(t) m ), ψvalue( ˆQ(t) m )), (7) where CA(·) denotes the multi-head cross-attention layer. ψquery(·), ψkey(·) and ψvalue(·) refer the linear projections for query, key, and value input, respectively. ˆQ(t) m = CAT(Q(t) c ,Q(t) m ) are treated as the key, and value for introducing temporal interactions. The temporal correlations between the memory queue and the current query set are captured in the attention matrix generated by using a softmax function. Before feeding the sparse queries into the crossattention layer, a pose-aware layer normalization operation is adopted for implicit camera pose information compensation. Notably, when the model operates under the singleframe (SF) setting, the historical memory queue is disabled. Therefore, ˆQ(t) m = ˆQ(t) c = Q(t) c . We refer the reader to the supplementary document for more details.

(c) Geometry-aware Cross-attention. To aggregate the spatial-temporal feature, we present a geometry-aware crossattention block to query intra-frame 2.5D features z(t) . For simplicity, we ignore the subscript i in the following descriptions. The geometry-aware features are regarded as key and value and fed into the multi-head cross-attention block to aggregate intra-frame long-range contextual features. Considering computational and memory efficiency, a deformable cross-attention block (DeformCA(·)) [Zhu et al., 2021] is adopted to integrate multi-scale intra-frame features by learning a small set of sampling locations:

where ϕquery(·), ϕkey(·) and ϕvalue(·) are the linear projections for query, key, and value, respectively. In our implementation, the DeformCA(·) first learns to sample a small set of 3D locations in the camera coordinate system for the 3D query anchors, and then projects the 3D coordinates to the 2D image plane using the camera intrinsic.

3.4 Pose Prediction Head and Loss Function

We adopted a set prediction head to estimate 3D translation, 3D rotation and object metric size. Similar to [Carion et al., 2020; Wang et al., 2021b], the Hungarian algorithm is utilized to find the optimal bipartite matching between the predicted object queries and the ground-truth (GT) object sets. Concretely, a feed-forward network (FFN) is introduced to regress the 3D translation, 3D rotation, and metric size. A linear projection layer predicts the confidence score of the corresponding class label:

where ˆµ ∈ R3 , ˆ γ ∈ R6 , and ˆσ ∈ R3 denote the 3D translation, rotation and metric size concerning width, height and length of object queries. φ(·) is the linear projection to predict the class label ˆ y. Following [Zhou et al., 2019], we predict 6D continuous parameters for estimating 3D rotations rather than 4D quaternion values in our implementation.

To train our model, multiple loss functions are used, including 1) The Hungarian matching loss for optimizing the bipartite matching, in which we use the focal loss Lclass for supervising the category prediction, and the L1 loss for pose and size estimation. 2) The auxiliary smooth L1 losses Ldepth and Lnormal for depth and normal map estimation. 3) The cross-entropy (CE) loss Lseg for semantic segmentation. The overall loss can be defined as:

where λ1 to λ7 are the weights to trade off different loss terms. Lposition, Lrot and Lsize refer to loss terms for estimating 3D translation, rotation and metric size, respectively. Due to the space limitation, we refer readers to the supplementary document for more details on loss functions.

4 Experiments

4.1 Datasets and Evaluation metrics

Following [Manhardt et al., 2020; Wei et al., 2024; Zhang et al., 2024], we trained and evaluated our proposed SeqPose on the REAL and Context-Aware MixedEd ReAlity (CAMERA) datasets [Wang et al., 2019] to verify the effectiveness of our method on both video and non-video datasets. The CAMERA dataset is rendered from CAD models of 1,085 object instances against real backgrounds, it contains 300,000 synthetic images, of which 25,000 constitute the CAMERA25 dataset for testing. The REAL dataset captures 42 object instances in the real world and includes 18 videos under different real scenes. It comprises 4,300 images for training, 950 for validation, and 2,750 constitute the REAL275 for testing. The above datasets contain six categories including bowl, bottle, can, camera, mug, and laptop.

We adopt the commonly used metric, i.e. mean Average Precision (mAP) across all object categories to evaluate the performance. Specifically, we calculate the mAP by setting thresholds of 3D Intersection-Over-Union (3D IoU) at 25%, 50% and 75%, i.e. IoU25, IoU50 and IoU75, the 3D bounding box is derived from the estimated pose and size. We also compute the mAP by setting a rotation error and a translation error within specific degrees and centimeters, i.e. the mAP at 10◦ , 10cm, 10◦ 10cm, 10◦ 5cm and 5◦ 5cm.

4.2 Implementation Details

We implemented SeqPose based on the DETR3D [Wang et al., 2021b] framework. For each dataset, we trained our model with the same hyperparameter settings. Concretely, the model is trained for 24 epochs with batch size 16 and optimized using the AdamW [Loshchilov and Hutter, 2019] optimizer. The learning rate is initialized to 4e-4 and updated following a cosine annealing policy. We set all the loss weights in Eq. (10) as 1.0, except for λ3=10.0 to penalize the rotation errors. The input size is 238×322. Moreover, we trained our model under single-frame (SF) and multi-frame (MF) settings on the REAL dataset, separately. For the CAMERA dataset, only the SF version of our model is trained, as the dataset exclusively consists of synthetic single-frame RGB images. Additionally, we set the frame length to three for memory queue updating under the MF setting. For the SF model, only 3D query anchors of the current frame are used to estimate the results. All experiments are implemented using PyTorch on a workstation equipped with four NVIDIA GeForce RTX 4090 GPUs. During testing, our model achieves an inference speed of 6.2 FPS. For more details on the implementation, please refer to our supplementary document.

4.3 Main Results

Quantitative results. We first compared our SeqPose with SOTA methods for RGB-based category-level object pose estimation on the CAMERA25 (Single-Frame) and REAL275 (Video) datasets, including Synthesis [Chen et al., 2020b], CPS++ [Manhardt et al., 2020], MSOS [Lee et al., 2021], ODL-Net [Fan et al., 2022], FAP-Net [Li et al., 2024], LaPose [Zhang et al., 2024] and DMSR [Wei et al., 2024]. The quantitative results are listed in Table 1. It shows that our models can outperform SOTA methods in terms of 10◦ 10cm and 5◦ 5cm. In particular, for translation estimation in terms of mAP@10cm, our model achieves 26.0% and 35.9% improvements compared to the SOTA method DMSR [Wei et al., 2024] on the REAL275 and CAMERA25, respectively. Moreover, our model also improves the performance on rotation estimation, achieving a significant performance gain of 5.4% and 0.9% in terms of mAP@10◦ on the REAL275 and CAMERA25, respectively. For 3D IoU metrics, our model also achieves competitive results. Concretely, the mAP@IoU50 of our SeqPose is 12.8% and 19.8% higher than DMSR on REAL275 and CAMERA25, respectively. Even for the strictest metric, i.e. IoU75, our models also realize 2.4% and 6.1% improvement compared with DMSR.

Figure 3 presents a detailed analysis in terms of Average Precision (AP) for each category against different types of error thresholds, including 3D IoU(%), rotation (◦ ) and translation (cm). It shows that our model delivers considerable superiority in comparison with the SOTA method DMSR w.r.t. different error thresholds across various object categories, especially for slender objects, i.e. "bottle", and complex-shaped objects, i.e. "camera" on both REAL and CAMERA datasets. We believe this is due to the powerful generalization ability of the DINOv2 and depth decoder. The robust semantic and geometry features can be generalized for objects with large intra-class texture and shape variations. On the other hand, our model indicates slightly poor AP values on the "laptop" class in terms of the 3D IoU and translation error. This may be related to the articulation of the "laptop" that exhibits more DoF in a pose with more flexible shapes. Since our model does not explicitly incorporate shape priors as constraints, the implicit 3D query anchors may be confused by the articulation states and variations of centroid/corner points. It is worth further investigation to apply our model to articulated objects leveraging kinematic constraints in future work.

Qualitative results. We further present Figure 4 to visualize the estimated object pose and size via projected 3D bounding boxes and object coordinate axes compared with the SOTA methods, i.e., LaPose [Zhang et al., 2024] and DMSR [Wei et al., 2024]. Specifically, for each dataset, the first two examples are consecutive frames. The DMSR results for the "camera" class show a large shift in rotation and scale across different frames. In contrast, our model can produce more stable and consistent results by introducing spatial-temporal information. Moreover, the results of DMSR in the third, fourth and fifth columns indicate object instances are missing or wrongly located. It may related to the cumulative errors of 2D object detectors in multi-stage settings. In contrast, our model performs better with a higher recall for accurately locating objects. In particular, the poses and metric sizes of small objects shown in the first, second and last columns can be accurately estimated with tight 3D-oriented bounding boxes. Moreover, It also shows inaccurate pose and size estimation for the "laptop" and "mugs" classes. We refer the reader to the supplementary document for more visual examples, failure cases and discussion.

4.4 Ablation Study

Ablation on proposedmodules. To further validate the effectiveness and contribution of the proposed modules, we performed several ablations to analyze our model on the REAL275 dataset in terms of the LVM backbone, auxiliary prediction heads and STSQ module, respectively. The quantitative results are listed in Table 2. Specifically, we trained baseline models by directly using EVAViT encoder [Li et al., 2022], and DINOv2 encoder [Oquab et al., 2023] without introducing the Depth decoder of DepthAnythingv2 [Yang et al., 2024b]. For auxiliary prediction heads, models without segmentation head (w/o Seg. Pred.), depth prediction head (w/o Depth Pred.), and normal prediction head (w/o Norm. Pred.), are trained separately for verifying the effectiveness of different supervision for feature learning. The w/o STSQ Memory refers to the SeqPose model without introducing the spatial-temporal correlated sparse query memory queue.

Table 2 shows that the SeqPose with DINOv2 encoder achieved a better mAP than EVAViT. It can be treated as a strong baseline and demonstrates the effectiveness of the DINOv2 in extracting rich and robust semantic prior features. Moreover, for auxiliary prediction heads, the performance can be further improved by introducing extra supervision to learn semantic and geometry-aware features. In particular, the model fine-tuned without introducing the depth estimation head, i.e. w/o Depth Pred., achieved better performance across all the metrics but a slightly lower mAP@10cm than other baseline models. For the STSQ module, it shows the model without introducing spatial-temporal correlated sparse query memory queue (w/o STSQ Memory) degrades the pose and size estimation performance in terms of all the metrics.

Our overall model achieved better performance than all the baseline models. It demonstrates the full model can benefit from: 1) The DINOv2 encoder produces robust intra-frame semantic features; 2) The auxiliary prediction heads can enhance the performance by encouraging the model to produce geometry-aware features compensating for intra-frame features; and 3) The spatial-temporal query memory queue further improve pose and size estimation performance by leveraging inter-frame feature interactions and correlations.

Ablation on the number of frames and queries. To further evaluate the effectiveness of the STSQ module, we performed another two sets of ablations on the number of frames in the query memory queue and the number of 3D anchor queries of the current frames on the REAL dataset. Quantitative results are listed in Table 3. Concretely, we ablate the performance of the STSQ memory queue by increasing the number of frames. Table 3 shows that the performance can be improved by increasing the number of frames to three, but degraded by setting more frames. It demonstrates the effectiveness of inter-frame query propagation by introducing temporal contextual relations. However, the performance may be degraded when the number of frames is larger than five. The reason may be related to the limitation of the STSQ module in capturing much longer temporal correlations for global range querying. Therefore, in our implementation, we set the number of historical frames as three in the memory queue.

Table 3 shows that the performance concerning pose and size estimation can be significantly improved when adding more implicit 3D anchors to 600, but slightly degraded by setting the number of 3D query anchors to 1200. Considering the trade-off between computational complexity and accuracy, we finally set the number of queries to 600.

5 Conclusion

We propose SeqPose, a novel end-to-end framework that unifies single-frame and video-based category-level object pose and size estimation. The DINOv2 encoder and depth decoder are incorporated to integrate intra-frame hierarchical semantic and geometric features. Furthermore, a spatial-temporal sparse query network is developed to propagate inter-frame correlations and aggregate intra-frame features by leveraging a set of learnable 3D query anchors. A pose prediction head is employed and optimized using the bipartite matching algorithm for set predictions. The whole framework is trainable in an end-to-end manner, enabling shape-prior-free pose and size estimation. Experimental results demonstrate our model can outperform SOTA methods in both single-frame and video-based categorical pose estimation on the CAMERAandREALdatasets. In futurework, it would be valuable for further investigation to generalize our model for challenging objects and scenes, e.g., articulated and unseen objects.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China (NSFC) under grants: 62302104 and 62476063, the National Key Research and Development Program of China under Grant 2025YFE0101100, the Natural Science Foundation of Guangdong Province under grants: 2023A1515012884 and 2025A1515011293, and the Science and Technology Program of Guangzhou under grant: SL2023A04J01625. We acknowledge that Yuzhu Ji and Mingshan Sun contributed equally to this work.