3D Vision and Language Pretraining with Large-Scale Synthetic Data

1 Introduction

Bridging 3D scenes with natural language represents a pivotal advancement in the pursuit of embodied artificial intelligence [Huang et al., 2023] including 3D vision-language (3D-VL) tasks such as 3D visual grounding [Chen et al., 2020; Achlioptas et al., 2020], dense captioning [Chen et al., 2021], and question answering [Azuma et al., 2022; Mo and Liu, 2024],etc.

Drawing inspiration from the impact of 2D vision-text pretraining models [Radford et al., 2021; Alayrac et al., 2022], recent researches [Zhu et al., 2023; Jin et al., 2023] demonstrate that 3D-VL pre-training can enhance the performance in subsequent tasks. However, the performance of these methods is limited by the size and diversity of 3D-text data, as 3D data collection largely dependents on the scanning device, introducing inherent complexities and increased costs in contrast to the simpler process of gathering 2D images. There are limitations to the existing 3D-VL datasets: (1) limited diversity of 3D vision, the largest-scale 3D-VL dataset ScanScribe [Zhu et al., 2023] combines multiple existing 3D datasets, but still only contains 1.2K indoor scenes, which causes limited vision diversity for pre-training; (2) insufficient fine-grained 3D-text associations, ScanScribe provides 278K descriptions of objects where only the sentence-level target objects is annotated and ignores the other fine-grained vision-language associations between the spatial region and other objects mentioned in the sentence. (3) labor-intensive data collection procedure, the 3D data collection of a real scene needs 30 minutes with advanced scanners [Yeshwanth et al., 2023] and annotating a scene with the help of the automatic segmentation still requires 1.93 minutes, makes largescale real-world 3D scene infeasible to collect.

∗Corresponding Author

To address the limitation of existing 3D-VL datasets, we propose the dataset SynVL3D, which contains the freecollected and easy-to-annotate synthetic 3D data generated by a 3D scene simulator [Deitke et al., 2022], instead of using realistic data. As shown in Figure 1, our dataset SynVL3D has four advantages: (1) Diverse scene data. Benefit from the capability of the 3D simulator [Deitke et al., 2022], we collected 10,000 indoor 3D scenes with 108 object categories. And SynVL3D provides semantic (category), visual (high-quality mesh), and fine-grained visual annotation (error-free position, orientation and segmentation mask) of each object . To acquire fine-grained understanding for scenes, we formed a scene graph for each house (scene) with the relationships between objects based on their spatial position, size, and shape. (2) Rich textual description. We utilize templates, off-theshelf image captioners [Dai et al., 2023], and GPT3 [Brown et al., 2020] to generate appearance and relationship descriptions of objects, and obtain more than 1M template-based or free-form descriptions. (3) Multi-grained association. Based on the above textual descriptions and scene graph, we retrain a distinct identifier for every noun phrase, creating a connection between each identifier and a spatial region. This establishes a robust phrase-region association, enhancing the grounded interpretation and facilitating a more meaningful evaluation. (4) Low collection cost. Based on simulators, we can obtain free 3D scene data, use off-the-shelf models to automatically describe scenes and obtain fine-grained annotations with few manual annotations. With virtual scenes, we also avoid privacy issues compared to real scenes.

Based on the diverse and fine-grained annotations of large scale of 3D scene provided by SynVL3D, we can construct more fine-grained pre-training tasks. The tasks include relationship prediction between objects, multi-level and viewaggregated region-word alignment, beyond scene-text matching and masked language/object modeling that previous pretraining methods [Zhu et al., 2023; Jin et al., 2023] use. The multi-grained 3D VLP tasks enable the model to capture rich 3D vision-language knowledge, which benefits downstream vision-language tasks. However, pre-training based on synthetic 3D data suffers from the domain shift between the synthetic data and the realistic data used in downstream tasks. Thus, we introduce synthetic-to-real adaptation with domain discriminator in the fine-tuning process of downstream tasks to reduce the representation distribution difference between the synthetic and realistic data of our pre-trained model. The efficacy of our model is evaluated across 3DVL tasks, including visual grounding (e.g., ScanRefer [Chen et al., 2020], Nr3D/Sr3D [Achlioptas et al., 2020]), dense captioning (e.g., Scan2Cap [Chen et al., 2021]), question answering (e.g., ScanQA [Azuma et al., 2022]).

Our main contributions can be summarized as follows: (1) We construct a large-scale synthetic dataset SynVL3D, featuring diverse 3D scenes, textual descriptions, and finegrained 3D region-text associations at a low manual cost, which is a significant improvement in terms of data diversity and scale compared to prior datasets. (2) We introduce a pretraining framework featuring new auxiliary tasks such as pairwise object relationship prediction and multi-level 3D regionphrase alignment to enhance comprehensive understanding. To address the domain shift between synthetic and real 3D data, we introduce a domain adaptation technique, improving the fine-tuning process for downstream tasks. (3) We demonstrate that with the data scale-up and model design, our model SynFormer3D achieves state-of-the-art performance on a variety of 3D-VL tasks, including visual grounding, dense captioning, and question answering.

2 RelatedWork

2.1 3DVision-Language Dataset

There has been a growing interest in 3D-VL datasets. [Chen et al., 2020] and [Achlioptas et al., 2020] introduced the ScanRefer and ReferIt3D datasets, which serve as benchmarks for anchoring sentence-level natural language to ground the 3D object. [Abdelreheem et al., 2024] and [Yuan et al., 2022a] extend to provide phrase-level grounding annotations. Scan2Cap [Chen et al., 2021] provides the textual descriptions about the scene in ScanRefer for captioning. ScanQA [Azuma et al., 2022] developed a 3D questionanswering dataset, which annotates the objects mentioned within the questions and answers. ScanScribe [Zhu et al., 2023] endeavors to build a large-scale dataset by combining the aforementioned datasets for pre-training purposes. However, these 3D-VL datasets are constrained by a lack of scenelevel diversity, fine-grained annotations (the largest existing dataset ScanScribe with only 1.2K scenes and 280K textual descriptions, and the laborious process involved in collecting and annotating 3D scenes. To overcome these obstacles, we constructed the dataset SynVL3D, which features freely collected and easily annotated 3D data generated by a 3D scene simulator [Deitke et al., 2022], as opposed to utilizing realistic data. It encompasses diverse 3D scene data (10K scenes), provides 1M rich textual descriptions with low cost. Moreover, compared to the sentence-region association in ScanScribe, we provide the phrase-region finegrained association, which means that most noun phrase in description has an association with the spatial region in the 3D scene.

2.2 Large-scale 3D-VL Pre-training

In recent years, large-scale pre-training has emerged as a fundamental paradigm in natural language processing (NLP) [Radford et al., 2018; Radford et al., 2019; Brown et al., 2020] and 2D vision-and-language (2D-VL) domains [Lu et al., 2019; Radford et al., 2021]. While the pre-training technique has become indispensable in these domains, its application in the 3D-VL field remains relatively uncharted. 3D-VISTA [Zhu et al., 2023] seeks to pioneer a pre-training model for 3D-VL tasks, employing strategies such as masked language modeling, masked object modeling, and scenesentence matching. Concurrently, 3D-VLP [Jin et al., 2023] introduces a context-aware spatial-semantic alignment to forge connections between point clouds and the textual descriptions of ground objects. However, the efficacy of these pre-training tasks is often compromised by the lack of detailed annotations in existing datasets. In this paper, we propose three fine-grained pre-training tasks, including object relations prediction, multi-level region word alignment and view-aggregated region word alignment, to enhance the model's perception capability of object relationships, scenes at different levels and from various views.

2.3 Simulation-to-Real Domain Adaptation

Simulation-to-real domain adaptation refers to the process of transferring knowledge acquired in a simulated environment to the real world, and is widely applied in 2D vision [Pei et al., 2024; Zhao et al., 2023; Pei et al., 2023; Xu et al., 2022; Chen and Liu, 2020; Chen et al., 2018] and 3D fields like autonomous vehicles [Zhao et al., 2021b], indoor scenes layout [Ding et al., 2022], and robotics [DeBortoli et al., 2021]. Existing methods primarily aim to design a simulation-to-real domain adaptation framework for uni-modal applications, such as the 3D semantic segmentation [Zhao et al., 2021b; Ding et al., 2022], point cloud object detection [DeBortoli et al., 2021] and point cloud registration [Chen et al., 2023]. However, there are few works exploring the simulation-toreal domain adaptation in 3D cross-modality applications, where the domain shift exists in both 3D vision and language. To address this dual and joint domain shift, we propose a taskspecific fine-tuning strategy with domain adaptation in 3D vision, language, and joint 3D-language domains.

3 Dataset Generation: SynVL3D

In this paper, we introduce SynVL3D, the first large-scale dataset of 3D synthetic scene-text pairs designed for 3D-VL pre-training. IAs indicated in Table 1, SynVL3D not only surpasses existing 3D-VL datasets in size but also offers a richer variety of 3D scenes, text, and fine-grained 3D-text annotations. Our dataset SynVL3D showcases key advantages in four aspects:

Diverse 3D Scene Data. We collected mesh data for 10,000 distinct scenes from an off-the-shelf 3D scene simulator [Deitke et al., 2022]. It encompasses 108 object categories, totaling over 1 million distinct instances. Besides the variety surpasses the existing datasets as shown in Table 1, we provide following annotation: (1) For each Single Instance, we automatically acquire its semantic category and precise bounding boxes and segmentation masks directly from the object and scene information provided by the simulator [Deitke et al., 2022] at no extra cost. (2) For Multiple Instance relationship, we construct a scene graph for each scene based on densely annotated objects and their attributes, where each node represents an object instance, and edges represent the relationships between an instance and its neighbors. Inspired by [Wald et al., 2020], the scene graph encompasses a rich array of relationships, which can be classified into three types: a) support relationships(e.g., standing, lying); b) spatial relationships (e.g., next to, in front of) that consider the relative positions of neighboring instances; and c) comparative relationships (e.g., bigger than, same shape as) that are based on the size and shape of an instance.

Rich Textual Description. We have generated more than 1 million textual descriptions, both templated and free-form, encapsulating detailed appearances and relationships between objects. (1) Template-based: For object appearances, we utilize an off-the-shelf image captioner [Dai et al., 2023] to generate captions for the front view of each object. We manually reviewed these descriptions and make corrections where necessary. For relationship triplets of the form <object1, relation, object2>, we craft descriptions using the template "the object1 is relation to object2". The description of an instance is structured as a paragraph by merging its appearance description with the relationship descriptions of its neighbors. (2) Free-form: We leverage GPT-3 [Brown et al., 2020] to rephrase the templated description to enhance the naturalness. These descriptions not only enrich the dataset with diverse visual perspectives but also enhance the semantic understanding of spatial relations and object attributes.

Phrase-Region Association. The grounding annotations in previous datasets [Achlioptas et al., 2020; Chen et al., 2020; Zhu et al., 2023] mainly consist sentence-region association for the single referred target object. In this paper, for 10K scene, we aim to curate grounding annotations of most objects in each description by providing phrase-region explicit association with an identifier of a phrase that refers to an object in a 3D scene. In practice, GPT-3 is tasked with rephrasing the descriptions while maintaining a critical requirement: the preservation of the identifier for each object, formatted as category(id of instance), throughout the rephrasing process. This ensures that the region-word association remains intact, maintaining the accuracy of object referencing in the dataset.

Low Collection Cost. Unlike other datasets in Table 1 that are manually collected and annotated from real-world scenes, our SynVL3D leverages freely synthetic 3D scene data. We utilize off-the-shelf models for automatic scene description and require minimal manual annotation cost. By using virtual scenes, our approach also circumvents privacy issues that can arise from scanning real-world environments.

4 Methodology

As shown in Figure 2, our SynFormer3D first uses multimodal encoder to extract and fuse 3D and text features. The pipeline mainly consists two steps: synthetic pre-training (Section 4.2) and task-specific fine-tuning (Section 4.3).

4.1 Multi-Modal Encoder

Following [Zhu et al., 2023], firstly we employ text encoder to encode words to text tokens {wCLS,w1:L}, where wCLS denotes the [cls] token and L denotes the length of a description. The 3D-object encoder is to encode M object proposals and extract features from the input point cloud, represented by o1:M. To achieve multi-modal fusion, we concatenate the text tokens and the 3D object tokens {wCLS, w1:L, o1:M} and forward them to a Transformer. The outputs of the crossmodal fusion module are denoted as {WCLS,W1:L,O1:M}, corresponding to [cls] token, word tokens and object features.

4.2 Synthetic Pre-Training

To align 3D scene and text in self-supervised manner, pretraining tasks Masked Language Modeling (MLM), Masked Object Modeling (MOM), and Scene-Sentence Matching (SSM) are adopted. Further, to achieve better alignment, we propose three fine-grained pre-training tasks based on the fine-grained annotation in SynVL3D: Object Relationship Prediction (ORP), Multi-level Region-Word Alignment (MRWA) and View-aggregated Region-Word Alignment (VRWA), details are as follows:

ORP. The previous methods [Zhu et al., 2023; Jin et al., 2023] neglect the importance of relationship understanding between object pairs, resulting in limitations for downstream tasks. And we introduce object relationship prediction task, which predicting pairwise relationship with prediction head r(·, ·), and the relationship annotations in SynVL3D are utilized to form label.

where i and j denote the indices of object features within the set of object proposal features O, M is the total number of object proposal and Ri,j is the label of relations between object i and j.

MRWA. 3D tasks and operations involvemulti-level targets with varying granularity, ranging from object, room, to entire scene. To achieve multi-level alignment, we align regions and words at object, room and scene levels respectively:

where L and M are the length of text(word) token sequence and the number of object features respectively. yi,j means the binary label of whether the i-th word xi and j-th region(object) zj is matching. l ∈ {object, room, scene} indicate the levels of a single object, room, and the entire scene, respectively. Wl represents the word feature sequence from descriptions at different levels , and Ol is the object feature set derived from the point cloud of the corresponding level. p(·, ·) is the region-word matching prediction head. The set of objects involved in different levels of textual description is different. At the object level, only objects near the description are mentioned, while room and house levels provide a larger range of objects. Therefore, region-word alignment at different levels considers the relationships between objects from the object set at different levels (the self-attention is conducted on different set of objects/words).

VRWA. The perspective changes when executing operations or object searching in 3D scenes, which makes viewaggregated object-text alignment critical. Following [Guo et al., 2023], we rotate the point cloud of a 3D scene into V to different views using the annotated camera parameters from SynVL3D: Pv = R(θv )P, where R(θ) is the rotation matrix [Guo et al., 2023], P is the scene point cloud and θv is the rotation angle adopted for v-th view: θv = 2π V (v-1), v ∈ {1, . . . , V } for a total of V views. R(θv ) ×P represents rotating the point cloud of whole scene clockwise by θv degrees along the Z-axis. Then we encode the v-th view features Ov , and apply multi-modal encoder and aggregate multi-view information for the i-th object: Gi = 1 V ∑Vv=1 Ovi . To achieve precise alignment between regions and words (as nouns in descriptions correspond to object identifiers in Figure 2), we perform region-phrase matching besides aligning scene and text descriptions:

where zi,j is the matching label of i-th word Wi and j-th region(object) Gj. s(·, ·) is the region-word matching prediction head.

Final Pre-training Loss. The final pre-training objective combines the losses from the proxy tasks mentioned earlier:

where α is a hyperparameter used to balance the basic auxiliary tasks(the losses LMLM,LMOM,LSSM is following [Zhu et al., 2023]) with our proposed fine-grained pre-training tasks. Note that the pre-training approach we propose is selfsupervised and downstream task-agnostic.

4.3 Task-Specific Fine-Tuning

By incorporating lightweight task-specific decoders, the pretrained model can be easily adapted to a variety of 3D-VL tasks. Due to the possible domain shift in our synthetic dataset and downstream task data (real scene scan data), we propose a task-specific fine-tuning strategy for synthetic-toreal domain adaptation to be better generalized to downstream tasks. In particular, we fine-tune the model on the following tasks: 3D Visual Grounding, 3D Dense Captioning, 3D Question Answering. We equip each task a specific lightweight decoder and corresponding fine-tuning losses.

Synthetic-to-Real Adaptation. Domain shift from multiple perspectives between synthetic data for pre-training and realistic data used in downstream tasks may hinder the performance on various 3D VL tasks. Vision differences can be observed between the synthesized data the scanned real data as shown in Figure 2. Besides, the descriptions in our dataset are generated by templates or GPT, which also has a gap with artificially annotated natural language descriptions in downstream datasets. Moreover, downstream tasks also involve the use of joint vision language, such as 3D question answering, where this type of domain shift requires a joint adaptation. Therefore, we introduce synthetic-to-real adaptation with a triple domain discriminator (shown in Figure2) in the fine-tuning process of downstream tasks to reduce the representation distribution difference between the synthetic and realistic data of our pre-trained model. The triple domain discriminator consists of three separate discriminator for vision, language and vision-language joint domains, addressing unimodal and cross-modal domain shifts.

A vision domain discriminator Dv is placed after multimodal encoder E to predict the domain label of the object features O:

where LBCE denotes the binary cross entropy loss. Gradient Reverse Layers(GRL) is adopted for min-max optimization. Similarly, a language domain discriminator Dl is adopted to predict the domain label of sentence tokens ˆW:

where the sentence token ˆW is the mean of the text token W1:L in a sentence. In order to reduce the gap of 3D-text alignment in different domains, we also introduce a joint discriminator Djoint. And the adaptation loss for shared-domain is:

where [ ˆWi;Oi] denote i-th data sample (setence and corresponding 3D object) which is the concatenation of the ˆW and O.

The final synthetic-to-real adaptation loss can be combined as: Lalign = Lvision + Llang + Ljoint.

Final Fine-tune Loss. The final fine-tuning objective combines the losses from the specific downstream task and synthetic-to-real domain adaptation:

where Ltask is the specific loss of the downstream task mentioned above and where β is a hyperparameter used to balance the downstream task loss and domain-adaptation loss.

5 Experiments

5.1 Implementation Details

The pre-training runs for 100 epochs with a batch size of 64 with NVIDIA A100 GPUs. We set the balance hyper-parameters α, β as 0.5 and 0.8. We use the AdamW [Loshchilov and Hutter, 2019] optimizer and learning rate is set to 1e-4 .

5.2 DownstreamTask

3D Visual Grounding. In Table 2, we evaluate our model on Nr3D, Sr3D [Achlioptas et al., 2020] and ScanRefer [Chen et al., 2020]. For Nr3D/Sr3D, we report the results as the grounding accuracy. For ScanRefer, we follow [Chen et al., 2020] to use detector-generated object proposals and report the results as Acc@IoU> k. As shown in Table 2, SynFormer3D obtains an overall accuracy of 67.5% and 78.6% on Nr3D and Sr3D, which has gains of +1.1% and +1.5% compared to the previous leading methods. It indicates that SynFormer3D is trained on our largescale SynVL3D with diverse 3D vision and text can understand complex 3D scenes better. Notably, our approach can be adaptive to different views better(+1.7% and +5.2% on "View-Dep" compared with 3D-VISTA), benefiting from the well-explored view-aggregated region-word alignment during the pre-training stage. Our SynFormer3D obtains an overall accuracy of 52.3% @0.25 and 46.2% @0.5, which outperforms all other methods on ScanRefer. Compared to the pretraining methods [Jin et al., 2023; Zhu et al., 2023], we introduce more fine-grained pre-training tasks benefiting from the scale-up data and rich and accurate annotations of our SynVL3D.

3D Dense Captioning. In Table 3, we evaluate our model on the Scan2cap. Our method outperforms the second best scores on CIDEr@0.25 (+1.1%), METEOR@0.25 (+0.5%), BLEU-4@0.5 (+2.6%), METEOR@0.5 (+0.9%) and ROUGE@0.5 (+0.5%). While our method achieves stateof-the-art in most metrics, it is only comparable with previous methods on BLEU-4@0.25, ROUGE@0.25 and CIDEr@0.5. The cause could be attributed to the fact that the textual descriptions utilized in pretraining are automatically generated with minimal manual annotations. This may result in gaps and biases about fluency and free-form expression.

3D Question Answering. We evaluate our model on the ScanQA dataset [Azuma et al., 2022]. The quantitative comparison (Table 4) shows our method surpasses existing models on both EM@1 (+0.6%/+1.1%), EM@10(+0.4%/+0.4%) and captioning metric ROUGE(+0.6%/+0.5%). While the textual input of 3D-QA is a question rather than referring expression , our method still benefits the 3D QA task with the multi-modal features frommulti-grained alignment learning.

5.3 Ablation Studies

We conduct the ablation studies on three tasks in Tables 5 to 7. 'VG' stands for overall accuracy of 3D visual grounding on Nr3D, 'Cap' for the CIDEr@0.25 result on Scan2Cap and 'QA' for the EM@1 results(w/ objects) on ScanQA.

Effectiveness of the fine-grained pre-training tasks. We evaluate the effectiveness of fine-grained alignment tasks in enhancing 3D visual grounding, dense captioning, and question-answering tasks (Table 5): (1) ORP (Row 2): Compared to the baseline (Row 1), ORP achieves improvements of +2.4%, +2.8%, and +0.8% for 3D visual grounding, dense captioning, and question-answering tasks, respectively. This suggests ORP's effectiveness in understanding objectenvironment relations in 3D scenes. (2) VRWA (Row 3): VRWA leads to gains of +2.8%, +4.1%, and +1.1%. Its focus on viewpoint-based region-word alignment seems to enhance adaptability from various perspectives. (3) MRWA (Row 4): Incorporating MRWA results in +2.6%, +3.8%, and +1.4% improvements. MRWA's multi-granularity alignment (object/room/scene) appears to enhance model perception in 3D environments. (4) VRWA and MRWA v.s. ORP: Both VRWA and MRWA show more significant improvements in visual grounding and question-answering than ORP, likely due to the enhanced perception across multiple perspectives and granularities from VRWA and MRWA, that is required by VG and QA tasks. (5) Integrating All Tasks: The combination of ORP, VRWA, and MRWA yields the highest gains: +7.0%, +9.1%, and +2.5%, demonstrating their capabilities.

Effectiveness of synthetic-to-realistic domain adaptation. We evaluated the effectiveness of utilising three specific domain adaptation discriminators that address the domain gap between synthetic and real-world down-stream data (Table 6). (1) 3D Vision Domain Adaptation (Row 2): By applying adaptation to the 3D vision domain, we observe performance increases of 9.2%, 6.7%, and 0.8% for 3D visual grounding, dense captioning, and question-answering tasks, respectively. This highlights the significance of 3D domain adaptation. (2) Language Domain Adaptation (Row 3): Extending adaptation to the language domain further enhances results, with gains of 2.8%, 2.7%, and 1.0%. This step demonstrates the value of refining the language domain. (3) Joint VisionLanguage Domain Adaptation (Row 4): Finally, incorporating vision-language joint domain adaptation leads to additional improvements of 2.4%, 4.0%, and 2.1%. This confirms the effectiveness of our comprehensive approach in reducing domain discrepancies and improving overall performance.

Ablation of MRWA levels. We analyze the impact of different MRWA levels (object, room and house) across three tasks (Table 7), by comparing with baseline object-only alignment. (1) Room-Level Alignment (Row 2): Adding roomlevel alignment to the baseline object-only alignment (Row 1) yields improvements of 0.4%, 1.2%, and 0.4%. This suggests that room-level associations between words and objects enhance the model's understanding of object relationships within rooms. (2) House-Level Alignment (Row 3): Extending alignment to the house level (whole scene) leads to gains of 0.2%, 0.8%, and 0.2%. This level of alignment assists the model in comprehending the entire scene by providing associations across the house. (3) Combining All Levels (Row 4): Combining alignments at both room and house levels results in the most substantial improvements of 0.8%, 2.3%, and 0.5%. It indicates the effectiveness of multi-level alignment in improving the model's 3D scene perception.

6 Conclusion

To tackle the scarcity of data and concerns regarding data collection cost for the 3D vision-language pretraining, we employ synthetic data generation using 3D simulators and offthe-shelf models and create a substantial and diverse synthetic dataset SynVL3D. With the rich annotations in SynVL3D, we pre-train a model SynFormer3D aligning 3D and language with diverse auxiliary tasks, including predicting object relationships, multi-level and view-aggregated regionword alignment. A domain adaptation strategy in fine-tuning is adapted to downstream tasks. Extensive comparisons on various benchmarks show the effectiveness of our method.

Acknowledgements

This work was supported by the grants from the National Natural Science Foundation of China 62372014.