Open-Vocabulary Object 6D Pose Estimation via Modulated Textual Semantics

1 Introduction

6-degree-of-freedom (6D) pose estimation aims to recover the 3D rotation and 3D translation of an object relative to the camera coordinate system, serving as a fundamental component in robotic manipulation [Liu et al., 2023] and augmented reality [Tang et al., 2019]. Although existing instance-level and category-level methods [Peng et al., 2019; Labb´e et al., 2020; Wang et al., 2019; Ji et al., 2025] achieve highprecision pose estimation, they rely heavily on large-scale training data with object-specific or category-normalized priors, limiting their applicability in open-world scenarios.

To overcome these limitations, recent research has shifted toward novel object pose estimation, aiming to generalize to objects unseen during training. However, most existing approaches [Liu et al., 2022; Wen et al., 2024; Deng et al., 2025; Huang et al., 2025] typically require high-quality 3D models or multi-view reference images of the target object during inference. While effective, preparing such data for every novel object incurs significant preparation and computational overheads, hindering their deployment in realtime applications. To reduce this dependence, a more practical paradigm is to infer the 6D pose using only a single reference view [Corsetti et al., 2024; Lee et al., 2025; Liu et al., 2025]. Given the limited visual information in this setting, augmenting the sparse observation with auxiliary representations has emerged as a critical solution to ensure robustness. Recent methods [Corsetti et al., 2024; Corsetti et al., 2025] demonstrate that textual prompts can be used to facilitate pose estimation of novel objects. These methods exhibit remarkable adaptability, performing effectively even when reference and query images are captured from completely different environments. However, treating text as a rigid prior limits its robustness in complex scenarios. Without adapting to the visible content, such static semantics fail to capture reliable cross-view feature correspondences essential for precise pose estimation.

In this work, we propose a unified framework that modulates static textual embeddings into adaptable semantic guidance for RGB-D open-vocabulary 6D pose estimation. Unlike prior works that directly use generic text priors, we introduce a TFMmodule. TFM employs a set of learnable latent queries to interact with frozen text embeddings, optimized to transform static textual descriptions into task-adaptive representations. To ensure these representations are robust to clutter, we propose a FGSA strategy. FGSA employs a metric learning objective [Oord et al., 2018] to strictly align the modulated textual representations with the visual foreground, while explicitly suppressing interference from visually similar background regions. Furthermore, we design the SACF module to bridge the visual gap caused by distinct environments in the reference and query views. By fusing the visual features of both the reference and query views with modulated semantics as an invariant anchor, SACF enables consistent feature aggregation across different views. Finally, the fused features are utilized to predict object segmentation and establish semantic-aware matching. Unlike methods [Corsetti et al., 2024; Corsetti et al., 2025] that rely solely on visual similarity for matching, we introduce a textual semantic consistency score to dynamically suppress mismatches.

In summary, our main contributions are as follows:

• We propose a framework that modulates static textual embeddings into adaptable semantic guidance for openvocabulary object 6D pose estimation.
• We introduce the TFM module for dynamic textual semantics modulation, the FGSA strategy for discriminative foreground grounding, and the SACF module for semantically-guided cross-view feature fusion.
• The experimental results on RGB-D open-vocabulary object pose estimation benchmarks demonstrate that our method achieves superior performance.

2 RelatedWork

Recent research in 6D pose estimation has primarily focused on handling novel objects that were not seen during training. Depending on the reference information available at inference time, existing approaches are generally divided into three main groups: model-based methods, multi-reference view methods and single-reference view methods. Below, we provide a brief overview of all three approaches.

Model-based novel object pose estimation. CAD modelbased approaches typically require high-quality 3D models of the target objects during inference to establish correspondences with the input image. OVE6D [Cai et al., 2022] establishes a universal framework for pose estimation from depth images by encoding object viewpoints relative to CAD models. MegaPose [Labb´e et al., 2022] adopts a coarse-to-fine strategy, first classifying a coarse pose from a massive set of synthetic templates and then refining it via a render-andcompare network. To improve efficiency, GigaPose [Nguyen et al., 2024b] decouples pose prediction and renders only a small number of templates using CAD models, thereby significantly accelerating the inference process. Although these methods achieve high accuracy, render-and-compare is computationally intensive and makes it difficult to obtain CAD models for novel objects in real-world scenarios.

Multi-reference view novel object pose estimation. To eliminate the dependency on CAD models, several methods utilize video sequences or multiple reference images. Early learning-based approaches like LatentFusion [Park et al., 2020] construct a latent 3D representation from a small set of reference views and regress the pose via a renderand-compare strategy. To improve robustness against occlusion and clutter, FS6D [He et al., 2022b] employs a prototypical network to aggregate features from multiple support views for pose estimation. OnePose [Sun et al., 2022] and OnePose++ [He et al., 2022a] reconstruct a sparse 3D point cloud from a video sequence using Structure-from-Motion and match 2D keypoints in the query image to this 3D representation. FoundationPose [Wen et al., 2024] establishes a unified framework for both model-based and model-free settings. In its model-free configuration, it leverages a neural field representation synthesized from a small number of reference images to perform robust tracking and estimation. Although effective, collecting complete video scans or calibrated multi-view image sets for each new object still incurs significant preparation overhead.

Single-reference view novel object pose estimation. This category targets the most challenging setting where only a single reference view is available. Due to the limited visual information available, directly recovering the 6D pose is a highly ill-posed problem. To overcome this, recent approaches attempt to augment the sparse observation with auxiliary representations. For instance, NOPE [Nguyen et al., 2024a] proposes to learn a canonical representation directly from a single image to infer the object pose. However, due to the inherent scale ambiguity in monocular estimation, it is primarily limited to estimating 3D rotation rather than the full 6D pose. Reconstruction-based methods like Any6D [Lee et al., 2025] attempt to lift the single reference image into a pseudo-3D model to facilitate 3D-2D matching. In addition, some methods explore text prompts as an additional source of semantic information. Oryon [Corsetti et al., 2024] and Horyon [Corsetti et al., 2025] leverage Vision-Language Models (VLMs) to guide the pose estimation process using text descriptions. These methods exhibit remarkable adaptability to environmental changes. However, they typically treat text embeddings as static priors, failing to adapt semantics to the specific visual context of the target object. In contrast, our framework modulates these static embeddings into taskadaptive semantic cues, ensuring robust cross-modal alignment and reliable feature correspondences.

3 Method

We follow Oryon [Corsetti et al., 2024] to address the problem of open-vocabulary 6D pose estimation, where the target object is defined solely by a user-provided textual prompt and is unseen during training.

Formally, given a textual description T and two RGB-D images {(Ir,Dr), (Iq,Dq)} captured from different scenes, our framework aims to jointly predict object segmentation masks, establish cross-view correspondences, and recover the relative pose. An overview of the proposed pipeline is shown in Fig. 1. First, the textual description T is encoded by a frozen text encoder [Radford et al., 2021] to obtain textual embeddings. These embeddings are then refined by the TFMmodule, which employs learnable queries to transform generic semantics into task-adaptive representations. The input images Ir and Iq are processed by a frozen image encoder [Radford et al., 2021] to extract dense visual feature maps. To ensure the modulated semantics are discriminative, we introduce the FGSA strategy. FGSA leverages foreground supervision to guide the calibrated textual semantics toward object-related regions while explicitly pushing away background distractors via a contrastive objective. This strategy operates only during training, introducing no additional computational overhead at inference time. Then, the SACF module utilizes the calibrated semantics as an invariant anchor to aggregate visual features across the reference and query views. Based on the fused representations, the framework predicts segmentation masks of object regions and computes semantic-aware cross-view correspondences. Finally, following [Corsetti et al., 2024], the predicted matches are projected into 3D space using the depth maps Dr, Dq, and camera parameters, and the relative 6D pose is then solved via a point cloud registration algorithm [Bai et al., 2021].

3.1 Text Feature ModulationModule

CLIP [Radford et al., 2021] text embeddings align with global image representations and work well for image-level retrieval, but these frozen embeddings often lack the finegrained detail and task-specific adaptation required for feature matching and 6D pose estimation. To address this, we propose the TFM module, which transforms static userprovided descriptions into adaptable, task-aware semantic representations without fine-tuning the heavy text encoder.

Let Ft denote the initial text embeddings extracted from the description using a frozen text encoder [Radford et al., 2021]. Rather than using these fixed features directly, we initialize a set of learnable latent queries QL from a Gaussian distribution. These queries act as a flexible medium to extract and reorganize semantic information from the fixed text embeddings. The TFM module is structured as a stack of M transformer blocks. As illustrated in Fig. 1, unlike standard decoder architectures, we design our block to prioritize the extraction of semantic cues. Formally, let Q(m-1) denote the input queries to the m-th block. The modulation starts with a multi-head cross-attention (MCA) layer, where Q(m-1) serves as the query, and the text embeddings Ft serve as both key and value. The features are then processed sequentially by a multi-head self-attention (MSA) layer and a feed-forward network (FFN). Each sub-layer adopts a postnormalization design, with a residual connection followed by layer normalization (LN):

The output of the final block is denoted as the calibrated text embeddings Fct. Driven by the end-to-end training objective, these learnable queries are optimized to selectively aggregate semantic cues relevant to the target object, yielding a compact representation that is better suited for the subsequent visual matching task than the original generic embeddings.

3.2 Semantic-Anchored Cross-View Fusion

Reliable 6D pose estimation requires accurate correspondences between reference and query views. To achieve this, we propose the SACF module, which aggregates multi-view features guided by the shared textual calibrated semantics. As shown in Fig. 1, SACF consists of two stages: Image Feature Modulation and the Cross-view Exchange Former.

First, we inject the calibrated textual semantics Fct from the TFM module into the visual features Fiv of both views (i ∈ {r, q}). We adopt the cross-attention architecture following [Liu et al., 2021; Cho et al., 2024] to perform image feature modulation and obtain semantically enhanced feature maps ˆFiv , ensuring that subsequent fusion is grounded in the object's specific semantic attributes described by the user. After image feature modulation, the next step is to establish interaction between the two views. Specifically, the features ˆFiv are first augmented with learnable 2D positional embeddings P and processed by depth-wise convolutions to incorporate local geometric context:

Then, inspired by [Cui et al., 2022], we project Fig into queries, keys, and values, and design a bidirectional exchange mechanism that swaps queries between the reference and query views to enhance shared semantic regions across the two views. We formulate the exchange former as:

Subsequently, we upsample features Hrattn and Hqattn to obtain the fused features Hr and Hq , which are then used to predict the object segmentation mask and compute cross-view feature correspondences.

3.3 Foreground-Grounded Semantic Alignment

The TFM module implicitly calibrates textual semantic embeddings for pose estimation, but without explicit constraints, it may overfit to generic scene contexts rather than the target object. To this end, we propose the FGSA strategy. As illustrated in Fig. 1, FGSA operates only during training and consists of two components: a feature selector and a semantic alignment objective.

Feature Selector. The feature selector is designed to identify the visual foreground and mine hard background negatives, enabling explicit contrastive supervision. Given an image feature map Fv and the binary object mask Mgt, we first extract the foreground representation. Specifically, we apply RoIAlign [He et al., 2017] on the object bounding box derived fromMgt, followed by global average pooling (GAP)

to produce a compact foreground vector vfg. To identify visually confusing background regions, we employ a sliding window strategy over Fv. The window size is dynamically adapted to the object's scale to ensure consistency. This generates a set of candidate background windows Wbg = {wi}, excluding those overlapping with the foreground. For each window wi, we extract its feature representation using the same RoIAlign and GAP operations as the foreground to ensure feature space alignment. Let vi denote the extracted vector for window wi. We then compute the visual similarity between each background candidate and the foreground vector:

where sim(·) is the cosine similarity. We then select the top-k background windows with the highest scores si as the hard negative samples, denoted as Vbg = {vjbg }kj=1 . These regions share similar visual statistics with the object and are most likely to confuse the model.

Semantic Alignment Objective. The goal of FGSA is to align the calibrated text embeddings Fct with the visual foreground while pushing away hard negatives. We formulate this using a contrastive InfoNCE loss [Oord et al., 2018]. Here, we treat Fct as anchors, vfg as positive keys, and the hard background features Vbg as negative keys. The semantic alignment objective is formulated as:

where f (n) ct denotes the n-th calibrated query vector in Fct, and s(t, v) = exp(⟨t, v⟩/τ ) defines the exponentiated cosine similarity with temperature τ .

By integrating LSA during training, the TFM module is guided to learn calibrated text semantics that are strictly grounded in object regions, reducing background interference and better supporting robust 6D pose estimation.

3.4 Semantic-Aware Matching

During inference, we estimate the 6D pose by establishing correspondences between reference and query views. Unlike prior works [Corsetti et al., 2024] that rely solely on visual similarity, which can be ambiguous in cluttered scenes, we leverage calibrated textual semantics to verify match reliability. Specifically, we propose a semantic-aware matching score that combines visual and semantic similarity.

Given the fused feature maps Hr , Hq and predicted masks Mrpred ,Mqpred , we first extract the valid features V r and V q . For a pair of feature points hi ∈ V r and hj ∈ V q , the visual similarity is computed as Svis(i, j) = ⟨hi, hj⟩. To incorporate semantic guidance, we compute a consistency score by measuring the alignment of each visual feature with the textual representation. We derive a holistic semantic prototype ¯tct = Mean(Fct) to serve as a global reference. In practice, since the fused features Hr and Hq have different dimensions from the modulated textual features, we perform this verification by retrieving feature vectors from the feature map Fv at the spatial locations corresponding to hi and hj. The semantic consistency score is defined as:

where fv(pi) denotes the visual feature vector sampled from Fv at the position pi of point i, and σ(·) is the sigmoid function. This acts as a soft gate that suppresses matches where either point fails to align with the user-provided description. The final matching score is a weighted combination:

where λsem balances the contribution of visual and semantic cues. We retain matches with Sfinal(i, j) > θconf using a confidence threshold θconf. The filtered matches are back-projected to 3D using depth maps and camera intrinsics, and the pose is solved via the PointDSC [Bai et al., 2021] algorithm.

3.5 Loss Functions

Our framework is trained in an end-to-end manner using a multi-task objective. The total loss function Ltotal is composed of matching loss Lmatch, segmentation loss Lseg and semantic alignment loss LSA.

Matching Loss (Lmatch). We establish the set of groundtruth correspondences Pgt by projecting pixels from the reference view to the query view using the ground-truth depth and relative camera poses. To avoid ambiguous many-to-one matches, we employ a symmetric dual-softmax loss [Cheng et al., 2021; Sun et al., 2021], which enforces cycle consistency, requiring the matched features to be mutual nearest neighbors in both views. Let S be the similarity matrix where Sij = ⟨hri , hqj ⟩/τ. We compute the matching probability from reference to query (Pr→q) and from query to reference (Pq→r) using the softmax function:

The matching loss is defined as the negative log-likelihood of the joint probability for all valid ground-truth pairs:

By maximizing the product of bidirectional probabilities, this loss strictly penalizes ambiguous matches and encourages a one-to-one correspondence between Hr and Hq .

Segmentation Loss (Lseg). We utilize the Dice Loss for segmentation supervision:

where i iterates over the reference and query views, and Mipred ,Migt denote the predicted probability map and the binary ground-truth mask for view i, respectively.

Semantic Alignment Loss (LSA). As described in Sec. 3.3, this loss explicitly aligns the calibrated textual embeddings with the visual foreground features while pushing away mined hard background negatives.

Finally, the total loss is defined as:

where λseg and λalign are balancing weights.

4 Experiments

4.1 Datasets and EvaluationMetrics

Datasets. Under the standard novel-object 6D pose estimation setting, we train on synthetic data and evaluate on real-world datasets without fine-tuning on test objects. Our model is trained and evaluated on benchmarks established by Oryon [Corsetti et al., 2024] based on the ShapeNet6D [He et al., 2022b], REAL275 [Wang et al., 2019], and ToyotaLight [Hodan et al., 2018] datasets. We term them Shape6DT, REAL-T, and TL-T, respectively, which focus on determining the relative pose of a text-prompted object captured in two distinct scenes. Shape6D-T is a large-scale synthetic dataset containing approximately 800K images, where the training text descriptions are constructed from object names and synonyms available in the metadata. We evaluate our model on REAL-T and TL-T, each of which contains 2K image pairs selected from real-world scenes with mild occlusions and challenging lighting variations. The textual descriptions for REAL-T and TL-T are constructed by combining specific visual attributes with object categories.

Evaluation Metrics. Following the standard BOP protocol [Hodan et al., 2018], we report Visual Surface Discrepancy (VSD), Maximum Symmetry-Aware Surface Distance (MSSD), and Maximum Symmetry-Aware Projection Distance (MSPD), and calculate their mean as the Average Recall (AR). We also use the ADD(S)-0.1d [Xiang et al., 2017] metric to evaluate pose recall with a 10% diameter threshold, which implicitly handles object symmetry. For segmentation, we employ the mean Intersection-over-Union (mIoU).

4.2 Implementation Details

For the visual and text encoders, we utilize the CLIP [Radford et al., 2021] backbone, which is frozen during training. The TFM module consists of M = 4 transformer layers. We train our model in an end-to-end manner on 4 NVIDIA RTX 4090 GPUs. Optimization is performed using the AdamW optimizer [Loshchilov and Hutter, 2017] with an initial learning rate of 1 × 10-4 and a weight decay of 1 × 10-4 . We employ a cosine annealing scheduler with a warm-up period of 1,000 iterations. The batch size is set to 16, and the model is trained for 25 epochs. The loss weights are empirically set to λseg = 1.0, λalign = 0.5 and the number of hard negative samples in FGSA is set to k=3. For the semantic-aware matching, the balancing weight is set to λsem = 0.2, and we filter matches with a confidence threshold θconf = 0.3. To prevent overfitting, we apply data augmentations including random color jittering, Gaussian blur and flipping.

4.3 Comparison with State-of-the-Art

We primarily compare our method with state-of-the-art single reference-view approaches, including the point cloud registration method ObjectMatch [G¨umeli et al., 2023], the crossview matching method Oryon [Corsetti et al., 2024], and the 3D model reconstruction method Any6D [Lee et al., 2025].

Quantitative results. Table 1 shows a comprehensive comparison on the REAL-T and TL-T datasets. In the practical setting where segmentation masks must be predicted, our method achieves significant improvements. Specifically, on the REAL-T dataset, our method achieves an AR of 40.7, surpassing the previous SOTA Oryon by 8.5 points. On the challenging TL-T dataset, which features severe lighting variations, our method achieves an AR of 37.6 compared to 30.3 for Oryon. This demonstrates that our calibrated textual semantics offer more targeted and robust guidance for pose estimation in complex environments. Our framework also achieves higher mIoU scores of 74.2 and 72.3 on the REALT and TL-T datasets, respectively. Notably, as shown in Table 1, when the baseline methods are equipped with masks predicted by our framework, their performance improves consistently. For instance, Oryon's AR on REAL-T rises from 32.2 to 35.5, and Obj. Mat.'s AR increases from 22.4 to 27.4.

In the setting using ground-truth masks, our method continues to outperform competitors on most metrics. On REAL-T, we achieve an AR of 52.9, outperforming the reconstruction-based method Any6D (51.0) and matchingbased method Oryon (46.5). This indicates that beyond better segmentation, our proposed method establishes more reliable feature correspondences for pose estimation. The slightly inferior performance compared with Any6D on some metrics is attributed to the fact that Any6D adopts an explicit 3D shape reconstruction strategy, which enforces strong geometric consistency. However, such generated geometric structures often suffer from visual artifacts, causing alignment deviations on the image plane. In contrast, our method achieves superior performance on projection-aware metrics.

Qualitative results. Figure 3 shows the results on the REAL-T and TL-T datasets using predicted masks. Compared to Oryon, our framework demonstrates superior robustness. For example, Oryon generates a noisy mask for the "brown mug", failing to filter out semantically similar background distractors. In contrast, our method produces a clean mask and an accurate pose, validating the effectiveness of the proposed framework. Figure 4 presents the results using ground-truth masks. Any6D is sensitive to sparse view information and struggles with complex textures. In contrast, our method can effectively aggregate multi-view features under the guidance of calibrated textual semantics, thereby estimating more accurate 6D poses.

Robustness to segmentation errors. To evaluate the robustness to segmentation errors, we quantified the errors in the masks generated by our framework using the IoU metric relative to the ground truth. Specifically, we evaluate the AR using masks under varying IoU thresholds. As shown in Fig. 5, our method consistently outperforms Oryon across all IoU thresholds on both the REAL-T and TL-T datasets. Moreover, the AR scores of the proposed framework exhibit greater stability compared to those of Oryon.

4.4 Ablation Study

Effectiveness of individual components. Table 2 validates the contribution of each module. The baseline method adopts fixed text embeddings and performs fusion only via image feature modulation to predict segmentation and matching results, yet this configuration yields inferior performance across all metrics compared to the full implementation of SACF. Incorporating TFM boosts AR to 36.8 by transforming fixed global text priors into task-adaptive feature representations, thereby providing more flexible semantic guidance for subsequent feature fusion. Applying FGSA with SACF significantly boosts mIoU to 74.0 by effectively suppressing background distractions. The full model achieves the best performance with an AR of 40.7, validating the effectiveness of modulated textual semantics for 6D pose estimation.

Impact of Semantic-Aware Matching Weight. We verify the contribution of semantic consistency by varying the balancing weight λsem in Eq. (10). As shown in Table 3, introducing semantic guidance (λsem > 0) initially boosts performance, peaking at λsem = 0.2 with AR scores of 40.7 and 37.6 on REAL-T and TL-T, respectively. This confirms that calibrated textual semantics serve as a critical constraint to reject ambiguous visual matches. However, performance degrades significantly when λsem exceeds 0.2. This indicates that while semantic cues are essential for verification tasks, overemphasizing semantics degrades the accuracy of 6D pose estimation, implying that semantic consistency should act as a soft constraint rather than a dominant factor.

5 Conclusion

In this paper, we presented a framework for RGB-D openvocabulary 6D pose estimation, guided by modulated textual semantics. To overcome the limitations of using static text embeddings for visual matching, we introduced the Text Feature Modulation (TFM) module, which transforms frozen descriptions into task-adaptive semantic representations. Then, we proposed the Foreground-Grounded Semantic Alignment (FGSA) strategy to explicitly ground these semantics in object regions while suppressing background distractions. Furthermore, the Semantic-Anchored Cross-View Fusion (SACF) module was designed to leverage these calibrated semantics as an invariant anchor, enabling robust feature aggregation across distinct views. Extensive experiments on standard benchmarks demonstrate that our method achieves superior performance in both segmentation quality and pose estimation accuracy. Future work will explore integrating image caption and depth estimation to extend our framework to an RGB-only setting, enabling broader applications and greater practicality.

Acknowledgements

This work was supported in part by the National Natural Science Foundation of China under Grant U24B20155 and Grant 62506175, in part by the Basic Research Program of Jiangsu under Grant BK20250666, and in part by the Jiangsu Province Higher Education Institution Basic Science (Natural Science) Research General Program under Grant 5KJB520038.