3DBench: A Scalable 3D Benchmark and Instruction-Tuning Dataset

1 Introduction

Recently, there have been significant advancements in multimodal large language models (MLLMs) [Li et al., 2023; Zhu et al., 2023; Radford et al., 2021], catalyzing a profound revolution across various tasks. Large-scale instructiontuning data is essential to harness the capabilities of MLLMs. To facilitate the integration of large models into the 3D domain, our goal is to establish a scalable evaluation benchmark specifically designed for assessing 3D-LLMs. Additionally, we elaborate on the detailed development of a large-scale dataset to address the scarcity of instruction-tuning datasets in the 3D domain.

∗Corresponding author

Existing instruction-tuning datasets for 3D-LLMs originate from publicly open datasets. PointLLM [Xu et al., 2023] utilizes Cap3D [Luo et al., 2023], a 3D object captioning dataset derived from Objaverse [Deitke et al., 2023]. PointBind & Point-LLM [Guo et al., 2023] is trained using Ulip [Xue et al., 2023], constructed based on ShapeNet [Chang et al., 2015]. LlaVa [Wang et al., 2023] efficiently incorporates the image modality into large models by prompting GPT-4 [Fu et al., 2022] with captions from the COCO dataset [Lin et al., 2014]. They also conduct experiments to validate the effectiveness of using GPT-generated [OpenAI, 2023] imagetext pairs as training data.

However, existing benchmarking efforts primarily focus on single-object classification and captioning, neglecting an effective evaluation of the spatial understanding capabilities of large models for complex and scene-oriented point clouds. Benchmarks for point cloud scenes, such as Visual Grounding (VG), Detection, and VQA tasks outlined by LAMM [Yin et al., 2023], can assess a model's interpretation of object positions in the given scene and its general knowledge inferred from language model. However, these tasks still do not sufficiently address the comprehension of multi-object relationships and the room-area perception capabilities, commonly referred as the scene reasoning ability of large models.

Additionally, high-quality instruction-tuning datasets in the 3D domain remain limited [Wei et al., 2021]. Current datasets are mainly collected to introduce large models and are generated based on open 3D datasets. This approach introduces a potential risk of data leakage in pre-training models [Dai et al., 2024]. Significant distinctiveness in task categories exists among datasets, with few benchmarks for each one and a lack of uniformity, making it challenging to accurately and comprehensively assess the capabilities of 3D large models.

To address above issues, We propose a scalable benchmark comprising ten multi-modal tasks and three evaluation metrics. As shown in Fig. 2, the tasks include common ones such as classification, VG, detection, and counting. Building on this foundation, we extend VG to include a room detection task. Additionally, we expand object understanding from 2D to 3D, requiring reasoning about the relative positions and attribute relationships of multiple objects. To evaluate the quality of large model dialogues and long-text generation capabilities, we include QA and caption tasks. Finally, we design a navigation benchmark to evaluate the spatial planning capabilities of 3D-LLMs. Evaluation metrics consist of three types: (1) For tasks with textual outputs, like relationship reasoning, we adopt a heuristic approach to instruct ChatGPT to score the predictions. (2) Traditional metrics such as precision and mAP are employed to evaluate detection and VG tasks. Furthermore, we introduce a path loss to assess navigation task. (3) For questions with simple-structured answers like classification and counting, we generate 300 to 2000 multiple-choice questions [Liu et al., 2023] and calculate the accuracy of output options.

The process of acquiring a large-scale instruction-tuning dataset consists of two main steps. During the initial step, we extract comprehensive metadata from the Procthor simulation framework [Deitke et al., 2022]. This metadata comprises depth maps and corresponding diverse types of ground-truth for both objects and scenes. Utilizing the depth maps, we manage to reconstruct a variety of 3D objects and scenes. In the second step, we utilize the ground-truth to inspire GPT to generate knowledge for textual tasks. By incorporating the ground-truth into diverse dialogue templates, we obtain a instruction-tuning dataset for various fine-grained tasks. In summary, our contributions can be summarized in three main aspects:

• Evaluation Benchmark: We develop a benchmark that spans from the object to the scene scale, covering dimensions of perceptual and reasoning abilities. It includes ten diverse multi-modal tasks, assessed with three types of customized evaluation metrics. The benchmark is enhanced and extended beyond existing standards through the incorporation of our proposed tasks and metrics.
• Instruction-tuningDataset: We design an approach for the automatic acquisition of a large-scale 3D instructiontuning dataset, resulting in 34,000 point clouds of everyday objects and 30,000 indoor scenes containing them. The dataset comprises over 0.23 million QA pairs, encompassing ten fine-grained tasks.
• Experiments and Observations: Experimental validation substantiates the efficacy of our proposed dataset. As shown in Fig. 1, we conduct a thorough quantitative evaluation of existing 3D-LLMs using 3DBench. Observations cover the quality of the generated dataset and variations in training protocols, offering insights for future explorations.

2 RelatedWork

The proposed 3DBench comprises two main contributions: benchmark and dataset. To assess the distinctiveness of existing work and our contributions, we review related works from two perspectives.

2.1 Multi-modal Evaluation Benchmark

Several existing benchmarks evaluate MLLMs across a range of tasks [Hao et al., 2022]. MMBench [Liu et al., 2023], with a focus on perception and reasoning, designs 20 finegrained tasks that cover the three-tier capacity dimensions of MLLMs. MME [Fu et al., 2023] curates all instructionanswer pairs manually to prevent data leakage and utilizes 14 sub-tasks to assess the perceptual and cognitive capabilities of VLLMs. LAMM [Yin et al., 2023] extends research to incorporate point clouds, creating three visual task benchmarks for 3D tasks to facilitate the assessment of scene-level perceptual abilities. To assess the text quality of large model outputs, AlpacaFarm [Dubois et al., 2023] introduces a swift and reliable GPT-4-based automated benchmark. Additionally, traditional metrics such as Bleu [Papineni et al., 2001], ROUGE [Lin, 2004], and METEOR [Banerjee and Lavie, 2005] produce ineffective evaluation results due to answer bias, unlike GPT. Although current benchmarks have made noteworthy contributions to evaluating MLLMs, there remains a deficiency in comprehensive quantitative assessment for 3D-LLMs.

2.2 Instruction-tuning Dataset

Instruction-tuning datasets consist of paired samples with instructions and their corresponding answers. Public multimodal datasets such as 2D COCO Caption and 3D ShapeNet offer objects and respective captions. LlaVa prompts GPT to generate detailed image descriptions with concise subtitles and object positions. Subsequently, it introduces new modalities into LLMs through textual information associated with the images. LAMM goes a step further by embedding ground truth from traditional computer vision tasks into dialogue templates, incorporating rich metadata from the dataset into instruction fine-tuning data to enhance the MLLM's ability to handle diverse tasks. In the 3D domain, ULIP-2[Xue et al., 2023] and Cap3D prompt MLLMs to automatically generate captions for images rendered from point clouds. Point-LLM[Guo et al., 2023] uses prompts from ChatCaptioner to generate high-quality Objaverse descriptions, while PointLLM[Xu et al., 2023] leverages captions generated by Cap3D to prompt GPT for more detailed point cloud descriptions in a conversational format. However, existing 3D instruction-tuning datasets are relatively scarce, and their sources are often limited to publicly available datasets. The availability of high-quality instruction-tuning datasets for 3D tasks is indeed a critical need.

3 3DBench

3.1 Overview

3DBench, as illustrated in Fig. 2 and 3, stands out from existing 3D benchmarks for two key reasons. Firstly, 3DBench includes ten diverse tasks to thoroughly assess the capabilities of 3D models in addressing aforementioned challenges. Secondly, 3DBench is built upon a unified instruction-tuning dataset that is extensive and hierarchically rich. It is grounded in human cognition of real-world tasks, illustrating a threetiered assessment. The top-level dimension, denoted as L-1, signifies the most apparent object-scene scale perception dimension. Beyond this, we consider the characteristics of large models and the logical reasoning process, allowing us to categorize tasks into three distinct categories: perception, reasoning, and expression. This introduces the L-2 competency dimension, covering three dimensions: 1) single or multipleinstance perception; 2) multi-instance relational reasoning; 3) conversational and descriptive abilities. Additionally, we derive the L-3 tasks dimension from the L-2 competency dimension. The benchmark includes ten evaluation tasks and three types of evaluation metrics.

3.2 Evaluation Tasks

Expansion Tasks. We systematically compile tasks from the 3D domain and language models. After excluding tasks beyond the capability of existing large models, we choose four tasks for our evaluation: classification, VG, detection, and counting. Building on prior works, we extend the scope of these existing tasks. Additionally, we expand the detection task to encompass the scene scale, instructing the model to predict bounding boxes for all rooms to assess the large model's overall perception capability in complex 3D scenes. Recognizing the expressive capacity of 3D-LLMs, we also introduce two textual generation tasks: generating detailed descriptions for long texts and generating dialogues for chat.

Novel Tasks. Drawing inspiration from 2D multi-instance tasks, we introduce tasks that require considering relationships and positional reasoning among multiple objects in realworld scenes, specifically, the multi-instance task within a given scene. Additionally, anticipating the capabilities of future large models in scene navigation and planning, we establish a navigation benchmark to assess the intricate scene perception and path planning abilities of 3D models.

3.3 EvaluationMetrics

GPT Prompting-based Metrics. As depicted in Fig. 3, we present a heuristic prompt tailored to address text generation tasks spanning different scales. When utilizing LLMs as the text evaluator, we provide GPT with information about all actual objects in the scene to avoid inflated scores caused by illusions. Moreover, this approach enables GPT to assess the authenticity of subtitles without depending solely on manual annotations. Furthermore, manual annotations may not necessarily represent the only correct solution for text generation. For evaluating text generation quality aligned with human habits, we recommend employing GPT for increased robustness.

Localization Metrics. We replace the conventional IOU metric, known for yielding low scores in detection and visual grounding (VG) tasks, with an innovative approach. We evaluate whether the predicted object's center falls within the bounding box, resulting in the 'in box' metric. Furthermore, we relax the criteria, permitting the predicted center to be within a one-meter radius of the ground-truth center, leading to the 'around box' metric. Additionally, we propose a novel path loss metric for the navigation task as shown in Fig. 5. It selects the longer trajectory between the prediction and ground-truth, measuring the distance between each endpoint of this trajectory and its nearest neighbor from the other one. The accumulated distance is compared against a pre-defined threshold to determine the success of the navigation.

Multiple-choice Metrics. We generate numerous multiple-choice questions for tasks with straightforward ground truth, encompassing classification, counting, and spatial relationships. To improve the ability of weaker 3D-LLMs to choose the correct answer, we direct all models to consider the correct answer as the only option. If a question is difficult to answer, we recommend all models to generate a random answer to avoid nonsensical responses.

3.4 Dataset Construction

Pipeline. The objective is to create large-scale 3D instruction-tuning datasets. To accomplish this, we design a scalable data construction pipeline. The pipeline comprises two main steps, as shown in Figure 4. In the initial step, we collect data for 30,000 houses using Procthor[Deitke et al., 2022] and extract depth images and metadata ground-truth from the embodied AI simulation framework Ai2thor[Kolve et al., 2017]. In the second step, we reconstruct point clouds and derive instruction-tuning datasets for all tasks based on the ground-truth.

Specifically, we reconstruct scene point clouds from color and depth images of the scene and objects with instance segmentation. Prothor can automatically generate random complete indoor scenes and is built on a simulation robot training framework, enabling us to easily obtain metadata from diverse perspectives. Depending on the scene's size, we teleport the robot to different positions to capture complete scene images. Through this automated process, we acquire 30,000 completely random indoor scene point clouds and over 34,000 object point clouds representing 93 categories of everyday items. We use ground-truth to prompt GPT to acquire rich world knowledge, serving as our training data for tasks such as object relation, QA, and caption. To enhance the ability to handle diverse visual tasks, we process the original ground-truth to obtain results for tasks like counting and room detection. Additionally, we generate diverse dialogue templates tailored for different tasks, embedding the results into conversations to create an augmented instruction-tuning dataset.

Data Statistics. We construct a dataset of more than 0.23 million instruction-tuning samples, utilizing 224,000 for training and 8,000 for evaluation. For the training sets related to visual tasks, we extract over 20 samples from each scene, ensuring that the fine-tuning sample size for each task exceeds 10,000. For text generation tasks, we provide approximately 1,000 training samples for fine-tuning models. The distribution of our instruction-tuning dataset is depicted in Table 1.

4 Experiments

4.1 Experiment Settings

The experimental settings are detailed in Table 2. Five groups of validation experiments are conducted to assess the effectiveness of our benchmark and dataset. As LAMM establishes benchmarks for detection, VQ, and VQA tasks, we use it as a robust baseline. It is originally trained on ShapeNet and 3RScan datasets, and then evaluated on the ScanNet dataset. Our experiments consist of five groups:

• E1 & E2: We initially assess the zero-shot performance of the LAMMmodel on the 3DBench test split (referred to as E1). Subsequently, employing the LAMM training framework, we exclusively train a model on our training split and evaluate its zero-shot performance on LAMMBench (noted as E2). The cross-validation results of three LAMM tasks (detection, VG, and VQA) for E1 & E2 are compared to analyze differences between our dataset and publicly available ones.
• E3: We systematically categorize the 3DBench dataset into various scales and employ the same framework to train multiple LAMM models. The objective is to investigate how the expanded dataset scale influences the enhancement of LLM performance across three tasks: detection, VG, and classification.
• E4: We conduct a re-training of the LAMMmodel using the complete version of our 3DBench. Subsequently, we compare the performance of the re-trained model with its original version (assessing zero-shot ability) on our test split, encompassing ten tasks.
• E5: We assess the zero-shot performance of two additional 3D-LLMs using the complete version of 3DBench to evaluate their capabilities across ten tasks.

LAMM Settings: To ensure a fair comparison of the point cloud understanding abilities among three models, we conduct tests using the 7B versions for all models. During the retraining experiment with LAMM, we identify biases in various evaluation metrics related to output text lengths. As a result, we adjust the target length for different tasks, aiming to reveal the optimal performance of each model on the respective dataset.

PointLLM & Point-LLM Settings: We maintain the default model parameter settings for both models. Following their guidelines, we uniformly sample point clouds to a fixed quantity. The evaluation encompass all ten tasks in 3DBench for PointLLM and Point-LLM. Due to the limitation of training data obtained from public sources, which is confined to the object level, and the potential vulnerability of inference results from untrained tasks to illusions, we focus on the evaluation of scene-level tasks. This aims to scrutinize the performance of large models when confronted with unfamiliar tasks and data.

4.2 Cross-set Validation (E1 &E2)

Considering LAMM's questionable performance under conventional IOU evaluation, and subsequent corrections revealing a success rate below 1%, we introduce "in box" and "around box" metrics as alternatives to the IOU metric, detailed in Section 3.3. In Fig. 6, it is evident that models trained on our dataset outperformed those trained on ShapeNet and 3RScan in zero-shot results on the LAMM benchmark within the same training framework. This enhancement is attributed to the richer content and broader numerical distribution inherent in our dataset.

4.3 Comparisons of Training Scales (E3)

In Fig. 7, we present the performance of models re-trained with datasets of varying scales on 3DBench. This analysis serves to validate the effectiveness of dataset expansion and discern the maximum limit of performance enhancement with increased data volume. Initially, we assess the impact of dataset size by selecting 33k objects from the first 500 scenes, forming an object-level instruction-following dataset. Subsequently, we extract multiple objects from 30k scenes to create a scene-level dataset, re-training the large model with datasets of different sizes. We further explore the upper limit of performance improvement by expanding the object dataset, as illustrated by the solid line. The results reveal that, as the training dataset size increases, the performance of the LAMM model reaches a plateau around 20k. This suggests that, owing to the incapacity to learn additional features, the simple structure of an encoder plus LLM is insufficient for addressing multi-modal tasks in the 3D domain seamlessly.

4.4 Results after Re-training (E4)

Table 3 displays the performance results of the LAMM on the 3DBench dataset before (to evaluate zero-shot ability) and after re-training, revealing a significant overall improvement in almost all tasks. Particularly notable is the approximately 20% enhancement in classification and counting tasks, suggesting that the features of object point clouds and scene point clouds in the 3DBench dataset are readily learned by the large model. However, there is a discernible decline in performance for partial text generation and position relation tasks. This might be attributed to the use of GPT-3.5 for acquiring world knowledge, potentially causing a disparity in text quality compared to GPT-4, which is employed by LAMM. The experiments, considering comparisons with public datasets and the effectiveness of intrinsic features, collectively affirm the efficacy of the large-scale dataset obtained from our pipeline.

4.5 Comparisons of 3D-LLMs (E5)

In Fig. 9, PointLLM and Point-LLM exhibit significantly better performance in object-level tasks compared to LAMM. PointLLM benefits from its point cloud encoder utilizing point cloud color, and Point-LLM possesses a powerful multimodal feature extractor, allowing them to gain more information compared to LAMM. However, in scene-level tasks, both models achieve inference results close to failure due to the training data only including objects. We plan to further assess their spatial understanding capabilities after re-training both 3D large models using the 3DBench dataset.

5 Observations &Analysis

We summarize a series of observations to assess the performance of 3D-LLMs across diverse tasks, aiming to provide valuable insights to the academic community.

Challenges in Incorporating Additional Features. LAMM demonstrates limited zero-shot classification capabilities, yet a notable improvement is evident when incorporating color information from point clouds as in PointLLM and Point-LLM. Furthermore, we observe a general improvement in classification performance with an increased number of input point clouds, though it tends to reach a plateau rapidly.

We hypothesize that integrating more efficient feature extraction structures can enhance the performance of traditional vision tasks for 3D-LLMs.

Limitations in Spatial Understanding Capability. 3DLLMs originally demonstrate subpar performance in tasks involving positional relationships, VG, and object detection. This suggests that existing large models struggle to adeptly capture positional information within scene point clouds. By expanding the training set size and adopting more reasonable evaluation metrics, as depicted in Fig. 8(a), reveals a gradual alignment of inference results with the ground-truth, achieving accuracy several times higher than completely random outcomes. The spatial understanding capability of 3D-LLMs holds considerable potential for enhancement.

Structure of Task Templates. Training 3D-LLMs directly with ground-truth as responses may yield excessively brief outputs, resulting in empty inference results for certain tasks, as demonstrated in the case of classification in Fig. 8(b). Meanwhile, monotonous conversation templates could compromise the diversity and richness of outputs from 3D-LLMs. Upon examining the responses to Point-LLM counting tasks (zero-shot inference on 3DBench) in Fig. 8(c), we notice a repetitive pattern such as "There are N objects" and "The image shows N objects." Therefore, it is advisable to employ GPT for generating diverse predefined conversation templates and integrating ground-truth into these templates to construct an instruction-tuning dataset.

Navigation Benchmark. Fig. 8(d) illustrates results for the navigation task. Navigation places heightened demands on the spatial perception and planning capabilities of 3DLLMs, particularly in localization tasks. Our evaluation strategy yields improved scores for suboptimal output results, yet still leaves rooms for further improvements.

6 Conclusion

In this paper, we introduce 3DBench, a scalable benchmark designed for evaluating 3D-LLMs covering ten diverse multimodal tasks with three types of evaluation metrics. Furthermore, we present a pipeline for automatically acquiring highquality instruction-tuning datasets. Through extensive experiments, we validate the effectiveness of our dataset by crossvalidate 3D-LLMs trained with various protocols. Our findings suggest that existing 3D-LLMs have considerable potential for further improvements in point cloud understanding and reasoning. We anticipate that our research will aid the research community in optimizing their models, and inspire the development of more efficient large models and high-quality instruction-tuning datasets.

Acknowledgments

This work was supported by National Natural Science Foundation of China (Grant No. 62202283).