DDPA-3DVG: Vision-Language Dual-Decoupling and Progressive Alignment for 3D Visual Grounding

1 Introduction

The goal of Visual Grounding (VG) [Deng et al., 2021; Liu et al., 2024] is to precisely locate the object referred to by a natural language description. In recent years, the progress of 3D data acquisition devices has sparked growing interest in using point clouds for various vision and language tasks, such as Dense Captioning (DC) [Yuan et al., 2022; Wang et al., 2022] and Visual Grounding (VG) [Yang et al., 2020; Li and Sigal, 2021]. Among these, 3D visual grounding (3D VG) plays a crucial role in environmental perception, ranging from visual language navigation to autonomous robotics [Chen et al., 2021; Chen et al., 2022; Gao et al., 2023; Liu et al., 2023; Wang et al., 2023a]. However, bridging the gap between the differing modalities of natural language and 3D scenes remains challenging.

∗Corresponding author.

Existing methods primarily follow two paradigms: the two-stage paradigm [Zhao et al., 2021; Yang et al., 2021; Zhang et al., 2023; Guo et al., 2023; Yuan et al., 2021] and the one-stage paradigm [Luo et al., 2022; Jain et al., 2022; Wu et al., 2023]. In the two-stage paradigm, a universal 3D detector first generates numerous candidates. These proposals interact extensively with the text features to produce language-modulated visual representations, which are then scored by a regression head to yield the final prediction. Conversely, the one-stage paradigm typically uses Farthest Point Sampling (FPS) to select a fixed set of points. Under the guidance of text features, the points with the highest confidence are progressively identified to achieve grounding.

Accurate alignment between visual representations and language descriptions is central to the success of 3D VG. Nonetheless, current approaches either keep the information in natural language and 3D scenes fully entangled or, conversely, fragment it too finely for alignment-both resulting in suboptimal performance. For example, in two-stage approaches, only object-level features are utilized, making fine-grained target object localization difficult. In contrast, one-stage approaches rely exclusively on point-level features, which can be too noisy to capture precise semantic cues and often produce diffuse confidence distributions. A similar challenge occurs with natural language: if treated solely at the sentence level, crucial distinctions between target and reference objects may be masked. While recent work [Wu et al., 2023] partially addresses imbalance and ambiguity by explicitly modeling language as a dependency tree, the problem of effectively extracting meaningful guidance from natural language and representing visual features at an appropriate granularity for cross-modal alignment remains unsolved.

In this paper, we propose a novel method, dubbed DDPA3DVG, that transforms 3D scene information into multigrained visual features under text guidance, while also decoupling natural language descriptions into coherent semantic components to improve target localization. We then align these dual-decoupled features from coarse to fine, achieving more precise 3D visual grounding. As illustrated in Figure 1, when dealing with complex 3D scenes containing multiple similar target and reference objects, existing methods that rely on fully coupled features often suffer from ambiguous alignments. By decoupling both the scene and language into more focused elements, our approach makes the alignment process more systematic and precise, especially when fusing text-guided localization with multi-granularity visual representations. Specifically, we divide the 3D scene into object-level, voxel-level, and point-level features, while splitting natural language into target object-related features and reference object-related features. These dual-decoupled features are then gradually fused from coarse to fine, allowing the network to effectively match target and reference descriptions with their corresponding visual features. Moreover, to effectively capture both global and local context, we employ static queries distributed across the entire scene alongside position-learnable dynamic queries. During the alignment process, these mixed queries interact progressively with the dual-decoupled features, facilitating a more comprehensive understanding of 3D spatial information and linguistic cues while minimizing interference from irrelevant objects.

In summary, our main contributions are:

• Dual-Decoupling Strategy: We decouple both the 3D scene and natural language to extract multi-granularity visual features and semantically rich text features, facilitating more effective cross-modal alignment.
• Progressive Alignment Framework: We progressively fuse these dual-decoupled features in a coarse-to-fine manner, which significantly improves localization precision for the target object.
• Experiments on three challenging benchmarks, ScanRefer, Nr3D, and Sr3D, demonstrate that our method achieves state-of-the-art performance, validating its effectiveness in 3D visual grounding.

2 RelatedWork

2.1 3DVision-Language Tasks

Vision and language are essential modalities for understanding and interacting with the world. In particular, 3D Vision- Language tasks [Luo et al., 2022; Azuma et al., 2022] integrate 3D visual perception with linguistic information, enabling more comprehensive analyses of complex environments. One representative example is 3D Visual Question Answering (3D-VQA) [Ma et al., 2022; Azuma et al., 2022], which relies on point clouds as input and requires models to interpret object attributes and spatial relationships in order to provide contextually accurate answers. Another example is 3D Dense Captioning (3D DC) [Yuan et al., 2022; Wang et al., 2022], aimed at generating text descriptions for each object by identifying its position, attributes, and spatial relations within the scene. By contrast, 3D Visual Grounding [Luo et al., 2022; Wu et al., 2023] specifically targets localizing a particular object based on a given linguistic expression. In this work, we concentrate on advancing the model's ability to integrate multi-grained 3D vision representations and decoupled linguistic elements, thereby attaining more precise grounding performance.

2.2 3DVisual Grounding

3D visual grounding focuses on locating objects within a 3D scene based on free-form natural language descriptions. Two main paradigms have emerged in this area: two-stage methods [Yang et al., 2021; Guo et al., 2023; Yuan et al., 2021] and one-stage methods [Luo et al., 2022; Jain et al., 2022; Wu et al., 2023]. Two-stage approaches leverage off-the-shelf, pre-trained object detectors to generate object candidates, then match these proposals to linguistic features to select the target region. For example, 3DVG-Transformer [Zhao et al., 2021] adopts a Transformerinspired architecture to incorporate contextual cues, thereby enhancing proposal generation and cross-modal disambiguation. Multi3DRefer [Zhang et al., 2023] extends the ScanRefer [Chen et al., 2020] dataset and task to support multiple object references, introducing new evaluation metrics and benchmarks to accommodate more general grounding scenarios.

In contrast, one-stage approaches directly regress the bounding box of the target object, bypassing explicit proposal generation. 3D-SPS [Luo et al., 2022] was among the first to refine localization guided by linguistic cues, while EDA [Wu et al., 2023] relies on implicit text decoupling to separate key linguistic components for greater alignment precision. However, EDA solely decouples text features, leaving visual features unaddressed. In this work, we present DDPA-3DVG, which explicitly decouples both visual and text representations, enabling more effective cross-modal alignment for enhanced 3D grounding accuracy.

3 Method

3.1 Overview

An overview of our DDPA-3DVG framework is illustrated in Figure 2. First, we use a Vision Encoder and a Language Encoder to extract the visual and text features, respectively. A Cross-Modality Encoder then fuses these features (Sec. 3.2). Next, the language descriptions are split into semantically meaningful elements, generating text features for both the target-related component and reference-related component.

Leveraging these decoupled text features, we decompose the 3D scene into multi-grained visual representations (Sec. 3.3). To further align the resulting decoupled visual and text features, we progressively fuse them with the input queries via cascaded reference and target decoders (Sec. 3.4). Finally, the fused queries localize the target object through a position alignment loss and a semantic alignment loss (Sec. 3.5).

3.2 Feature Encoder

For the language modality, we employ a pre-trained RoBERTa [Liu et al., 2019] model to encode the input sentences. On the visual side, we generate point-level features f v p ∈ RN×D through PointNet++ [Qi et al., 2017], where N represents the number of points in the point cloud and D their feature dimensionality. Following [Jain et al., 2022], we employ a Cross-Modality Encoder-consisting of multiple cross-attention layers-to integrate the extracted visual and text features. The output, f t ∈ Rl×D , where l is the length of the input sentence, is then used in the subsequent sentence decoupling stage; additionally, a set of static queries Qs ∈ RNs×D is initialized for the following progressive alignment phase.

3.3 Vision and Language Dual-decoupling

Language Decoupling

Drawing inspiration from EDA [Wu et al., 2023], we use existing NLP tools [Schuster et al., 2015; Wu et al., 2019] to explicitly classify words in sentences into five categories:

1) Main Object (Target): The object to be localized. 2) Attributes: The target object's properties (e.g., color, shape). 3) Auxiliary Objects (References): Additional objects that help localize the target. 4) Pronouns: Referents for the main object. 5) Relations: Spatial relationships between the main object and auxiliary objects.

While EDA builds a dependency tree to decouple sentences, we further classify all words into two components based on these categories: the target component (composed of "main objects" and "attributes"), and the reference component (including "auxiliary objects", "pronouns", "relations" and any remaining ones). First, we encode the five word categories as multi-hot vectors, where the corresponding word position is marked as one. This process produces the codes {Cmain, Cattri, Cauxi, Crel , Cpron, C∅}, where C∅ represents words that do not belong to any specific category. The target and reference components are then encoded as Ctgt = Cmain +Cattri and Cref = Cauxi +Crel +Cpron +C∅.

Next, we obtain the decoupled target and reference text features by applying the respective components to the text features f t :

This decomposition effectively separates the target object from the reference elements, enhancing the network's ability to interpret objects and their interrelationships within the scene.

Vision Decoupling

To capture both coarse and fine spatial cues, we propose multi-grained visual representations [Yuan et al., 2025a; Yuan et al., 2025b] at the object, voxel, and point levels. This approach benefits from high-level features for swift object identification and low-level features for precise localization.

Multi-granularity Visual Representations. Objectlevel features, commonly used in two-stage 3D VG approaches [Jain et al., 2022], are extracted by applying a pretrained 3D detector to the scene. The resulting bounding boxes yield class and box embeddings, concatenated to form the object feature f vo .

Point-level features follow the one-stage 3D VG paradigm, where each point in the cloud is encoded by PointNet++, and FPS is applied to get point proposals. Local neighborhoods are then aggregated to refine each point feature f vp .

To bridge object- and point-level representations, we introduce voxel-level features. We divide the point cloud into a W × L × H voxel grid:

where (x, y, z) are a point's coordinates and (i, j, k) its voxel index. Each voxel's embedding ¯f vv is taken as the average of its points' features. A shallow MLP calculates a usefulness score for each voxel, retaining only the top R×Nvoxel based on this score:

Position embeddings are then added, followed by a selfattention layer for aggregating global context:

where xv is the voxel's position, ⊙ is element-wise multiplication. Finally, our multi-grained visual features are:

Text-guided Vision Decoupling. Since f v represents the entire scene without text context, we further decouple it under the guidance of language description. We first compute the cosine similarity between each element in f v and the target or reference text features f t tgt and f t ref , then select the top-K features with the highest similarity scores:

where ⊗ means the calculation of cosine similarity. The decoupled features thereafter become f v tgt = [f vo tgt , f vv tgt , f vp tgt ] and f v ref = [f vo ref , f vv ref , f vp ref ]. This filtration discards distracting elements in the scene and improves alignment accuracy by focusing on the most salient features.

3.4 Progressive Alignment

After the dual-decoupling of vision and language, we obtain f t tgt , f t ref , f v tgt , and f v ref . We then propose a progressive alignment module composed of two cascaded decoders: 1) A Reference Decoder that progressively fuses f t ref with the multigrained visual features f v ref . 2) A Target Decoder that further refines f t tgt with f v tgt .

Within each decoder, text features are merged with the object-, voxel-, and point-level features from coarse to fine. To capture both global and local spatial cues, we synthesize queries from two sources: 1) static queries derived from Farthest Point Sampling, initialized by Qs, and 2) dynamic queries, learnable embeddings randomly initialized. During training, dynamic queries adaptively focus on crucial regions (e.g., the target object), while static queries help maintain broad scene coverage.

For example, consider a scenario where the text describes a reference object ("it is under a big painting that is gray, yellow, and brown") and a target object ("this is a light brown couch"). In the Reference Decoder, dynamic queries converge around the "painting" and obtain its spatial relationship "under" relative to the "couch." In the Target Decoder, the mentioned attributes of the couch guide the dynamic queries to accurately localize the correct furniture piece.

3.5 Prediction Head and Alignment Loss

After the progressive alignment process, the final queries are used to predict the target object's bounding box via a singlelayer MLP and estimate the position labels, Cpred, for each word through another MLP.

To enforce cohesive visual-text alignment, we define two losses: position alignment loss and semantic alignment loss. The position alignment loss is formulated as follows:

where the ground truth language distribution, Clang ∈ {Cref , Ctgt}, is determined by the text position labels of each component. Cobj represents the semantic distributions of object features for the i-th candidate among k objects and is obtained by applying a softmax operation to Cpred.

The semantic alignment loss measures coherence between object embeddings, o, and text embeddings, t. These embeddings are derived through linear projections of the output queries and f t , respectively. This loss is defined as:

where s+i and s-i denote the similarity scores between the object embedding o and the text feature vector t for matched and unmatched cases, respectively. F and N represent the positive features and the number of objects or text embeddings, respectively. The final semantic alignment loss is computed as Lsem = 1 2 (L{f v ,k} sem +L{f t ,l} sem ).

By minimizing these two losses, the model learns to align natural language descriptions with 3D scene features more effectively and encourages the visual and text representations of the same target to converge in feature space, thereby yielding improved accuracy on 3D visual grounding tasks.

4 Experiment

4.1 Datasets and EvaluationMetrics

We evaluated the effectiveness of DDPA-3DVG using three widely adopted and challenging datasets: ScanRefer [Chen et al., 2020], Sr3D and Nr3D [Achlioptas et al., 2020]. ScanRefer is a 3D visual grounding dataset constructed upon 800 scenes from ScanNet [Dai et al., 2017], encompassing 51,583 text descriptions. Within each scene, if a target object is the only instance of its class, it is labeled as unique; alternatively, if there are multiple instances of that class, it is labeled as multiple. For ScanRefer, the evaluation uses Acc@mIoU, which measures the proportion of descriptions for which the predicted bounding box's Intersection over Union (IoU) with the ground truth exceeds m, where m ∈ {0.25, 0.5}. Similarly, Sr3D and Nr3D dataset [Achlioptas et al., 2020], are also constructed from ScanNet. Nr3D contains 41,503 descriptions (annotated by humans) spanning 707 scenes, whereas Sr3D comprises 83,572 machine-generated descriptions. Both subsets use accuracy as the evaluation metric. Furthermore, each scene is categorized as easy or hard based on whether more than two instances of the same object class are present.

4.2 Comparison with State-of-the-Art Methods Quantility and Visual Results

As presented in Table 1, we benchmarked our proposed DDPA-3DVG on the ScanRefer dataset against leading approaches. Our method achieves state-of-the-art accuracy rates of 55.3% and 43.2% for IoU thresholds 0.25 and 0.5, respectively, indicating robust performance. Notably, many earlier methods have leveraged auxiliary 2D image data to enhance point-cloud features [Chen et al., 2020; Zhao et al., 2021; Cai et al., 2022], whereas our approach uses only sparse point clouds and still surpasses these image-dependent baselines. This result underscores the efficacy of the dual-decoupling mechanism and progressive alignment module we propose.

Following [Wu et al., 2023], we further compared our approach with other one-stage methods (see the lower part of Table 1). Even without relying on object-level visual features, DDPA-3DVG outperforms competing alternatives, reinforcing the strength of the proposed design. We visualized our grounding results in Figure 3 for qualitative comparison. In typical scenes with fewer distractions, both EDA and our proposed method achieve high-precision grounding. However, in more complex scenarios where target identification is hindered by multiple interfering objects, our approach more effectively mitigates these distractors, consistently delivering correct predictions, which underscore our method's aptitude for understanding spatial contexts and accurately localizing targets in cluttered scenarios.

To confirm the generalizability of DDPA-3DVG, we also evaluated it on the SR3D and NR3D subsets of the ReferIt3D dataset [Jain et al., 2022; Wu et al., 2023]. As reported in Table 2, our method again outperforms other approaches, with accuracies of 54.4% on SR3D and 69.0% on NR3D. These results affirm the robustness of our approach across diverse 3D point-cloud environments.

To provide a more comprehensive look into the inner workings of DDPA-3DVG, we illustrated the progression of both static and dynamic queries in Figure 4 at four points during training: steps 0, 25, 50, and 100. Initially, the dynamic queries lie scattered throughout the 3D scene, showing no discernible structure. With successive training iterations, these queries begin to converge toward the reference object, precisely guided by the linguistic clues provided in the text descriptions. As the model refines its understanding of the spatial relationships and semantic attributes, the dynamic queries gradually adjust their positions from the reference zone toward the actual target region in the scene.

This gradual realignment process is driven by our dualdecoupling mechanism, which aims to isolate and refine relevant visual and text features. By decoupling vision and language at multiple levels, our method yields more discriminative representations, allowing the dynamic queries to interpret the scene context more precisely. The shift of the dynamic queries over time provides tangible evidence of how decoupled semantics and visual cues are integrated step by step. Critically, our progressive fusion strategy-carried out through specialized reference and target decoders-ensures that the model systematically narrows down potential targets, eventually pinpointing the correct object with higher accuracy. The consistent adjustment of query positions confirms that combining vision-language dual decoupling with the progressive alignment module is instrumental in tackling challenging 3D grounding tasks, especially within complex environments.

Grounding without Object Name

To assess our model's ability to reason beyond direct linguistic cues such as explicit object names, we adopted the "Grounding without Object Name" task proposed by EDA [Wu et al., 2023], wherein all target names in the ScanRefer validation set are replaced with the token "object". EDA further divided the language descriptions into four subsets that focus on: (i) only attributes, (ii) only spatial relationships, and (iii) both attributes and relationships. As summarized in Table 3, our model, without additional training, achieves state-of-the-art results across all these subsets.

This superior performance indicates that the reference object and spatial relationship information extracted via our decoupling framework has a highly effective guiding impact on the grounding process, enabling accurate localization even when explicit object names are omitted.

Results about Convergence

Figure 5 presents the experimental results. In terms of convergence, our method achieves faster convergence during training. Specifically, the number of epochs required for our method to reach a performance of 52.7% is 37 fewer than that of EDA, representing a 1.97× reduction in training epochs. Additionally, after convergence, our method achieves an average accuracy improvement of 1.23% compared to EDA when IoU is set to 0.25.

These results highlight the effectiveness of our dual decoupling of vision and language features and the progressive alignment mechanism, which enables the model to align cross-modal features more efficiently and effectively. This not only accelerates the training process but also enhances the overall model performance. In summary, our method outperforms EDA in terms of both convergence efficiency and effectiveness for 3D Visual Grounding.

4.3 Ablation Studies

Effectiveness of Mutil-granularity Visual Representation

To evaluate the effectiveness of multi-granularity visual representation, we trained four model variants under different representations, as reported in Table 4. Incorporating the object-, voxel-, and point-level features in a unified manner consistently enhances performance. This improvement arises from the enriched, hierarchically distinct 3D spatial information provided by voxel-level embeddings, which supplement the coarse-to-fine granularity of the visual features.

Effectiveness of Language Decoupling

To validate the effectiveness of language decoupling, we removed the decoupling process and instead used sentencelevel features directly in the progressive alignment. When reference- and target-related linguistic cues are entangled, the decoder struggles to precisely align visual and linguistic features, resulting in noticeably inferior performance, as shown in Table 5. Specifically, accuracy decreases by 10.2% and 7.7%, respectively. This highlights the importance of separating target and reference-related elements, as the proposed feature decoupling provides fine-grained discriminative ability to guide precise visual grounding.

Effectiveness of Mixed Points

To investigate the benefits of mixed points in the decoder queries, we performed an ablation experiment with three configurations: (1) Qs only, (2) Qd only, and (3) the combination Qs +Qd. As detailed in Table 6, leveraging both static and dynamic point queries significantly improves localization accuracy. Dynamic points converge around the target object during training, whereas static points retain a broader receptive field and correct for instances that dynamic points might overlook. These two categories of points thus complement each other, resulting in more robust grounding performance.

5 Conclusion

In this paper, we present DDPA-3DVG, an effective approach for the 3D Visual Grounding task. We innovatively introduce a voxel-based, multi-granularity visual representation, endowing the model with hierarchical 3D spatial information. Furthermore, we decouple text and multi-granularity visual features into reference and target information using Language Decoupling and Text-guided Vision Decoupling modules. Our Progressive Alignment module then progressively grounds the target object, effectively mitigating interference from the 3D scene and notably improving localization precision. Though effective and demonstrating substantial improvements over existing methods, our approach could benefit from more efficient alignment modules and learnable candidates to eliminate dependence on pre-trained detectors, which remain valuable directions for future work.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (62376080), the Zhejiang Provincial Natural Science Foundation Key Fund of China (LZ23F030003), the National Natural Science Foundation of China (62476075), and the Zhejiang Key Laboratory of Optoelectronic Intelligent Imaging and Aerospace Sensing.