UniM-OV3D: Uni-Modality Open-Vocabulary 3D Scene Understanding with Fine-Grained Feature Representation

1 Introduction

Accurate and resilient comprehension of 3D scenes plays a vital role in various practical applications, including autonomous driving [Mao et al., 2022], virtual reality [Park et al., 2008] and robot navigation [Shafiullah et al., 2022]. Despite significant progress in recognizing closed-set categories on standard datasets [Graham et al., 2018; Vu et al., 2022; Misra et al., 2021], existing models frequently struggle to recognize novel categories that are not present in the training data label space. The limited scalability of existing models in open-set scenarios hinders their practical applicability in the real world and motivates us to explore the open-vocabulary 3D scene understanding capability.

Benefit from the 2D foundation model, such as CLIP [Radford et al., 2021] and DINO [Caron et al., 2021], previous methods [Peng et al., 2023; Chen et al., 2023; Liu et al., 2023a; Liu et al., 2023b] attempt to distill knowledge from 2D pixels to 3D features, aligning point clouds with 2D images. Another line of works [Takmaz et al., 2023; Huang et al., 2023c] focuses on points-only mechanism. The existing methods inadequately investigate data from multiple modalities, such as point clouds, images, linguistic input, and depth information, which are crucial for effective open-vocabulary 3D scene understanding. Despite their effectiveness, the absence of alignment and learning for other modalities makes them miss a lot of semantic information and hinders their ability to effectively handle fine-grained point cloud object instances, as shown in Figure 1 (a).

Considering the properties of 3D scene, we should not overlook the inherent characteristics of 3D data itself. Firstly, depth information serves as a crucial modality for depthinvariant feature aggregation, which is often neglected. Early works [Zhang et al., 2022; Zhu et al., 2023; Huang et al., 2023b] in 3D recognition have attempted to transfer depth modality to 3D understanding. However, mere projection or rendering, and merely aligning with a single modality such as image or text, cannot fully exploit all the characteristics of depth information, nor can it perform well in the task of open-vocabulary 3D scene understanding. The second aspect to consider is the exploration of point cloud information itself. Aiming at the point clouds caption learning, previous approaches either solely rely on simplistic templates derived from CLIP [Peng et al., 2023; Liu et al., 2023a; Zhang et al., 2023] or utilize images as a bridge to generate corresponding textual descriptions from 2D images [Ding et al., 2023b; Yang et al., 2023]. However, point clouds captioning can make alignment easier and more comprehensive, particularly when multiple modalities are incorporated during the alignment training. And the point clouds encoder almost comes from mature 3D extractors [Choy et al., 2019; Graham et al., 2018], which often keep frozen during training. The frozen backbone and bridging alignment impede the generalization in complex 3D open vocabulary scenarios.

To this end, in order to fully leverage the synergistic advantages of various modalities, we propose a comprehensive multimodal alignment approach that co-embeds 3D points, image pixels, depth, and text strings into a unified latent space for open-vocabulary 3D scene understanding, namely UniMOV3D. We thoroughly explore and utilize the characteristics of point clouds from two perspectives. First, to extract finegrained geometric features of different levels from the original irregular 3D point clouds, we design a hierarchical point cloud feature extraction module. Second, in terms of generating point-caption pairs, we make the first attempt to generate corresponding text directly from point clouds instead of using images as a bridge. And we build hierarchical point-semantic caption pairs, including global-, eye- and sector-view captions, which can offer fine-grained language supervisions.

Furthermore, we introduce depth information into the entire network and the modified depth encoder inspired by CLIP2Point [Huang et al., 2023b] keeps learning during training . As for image modality, we employ the pre-trained model PointBIND [Guo et al., 2023] to extract image feature. Given the multi-modalities and their dense representations, we finally conduct multimodal contrastive learning for robust 3D understanding. In this way, we are capable of optimizing their reciprocal benefits by adeptly amalgamating multiple modalities.

Extensive experiments on ScanNet [Dai et al., 2017], ScanNet200 [Rozenberszki et al., 2022], S3DIS [Armeni et al., 2016] and nuScenes [Caesar et al., 2020] show the effectiveness of our method on 3D open-vocabulary tasks, surpassing previous state-of-the-art methods by 3.2%-7.8% hIoU on semantic segmentation and 3.8%-10.8% hAP50 on instance segmentation. More interestingly, UniM-OV3D demonstrates a robust ability to understand complex language queries, even those that involve intricate reasoning or necessitate extensive world knowledge, as illustrated in Figure 1 (b).

Our main contributions are summarized as follows:

• Within a joint embedding space, UniM-OV3D firstly aligns 3D point clouds with multi-modalities, including 2D images, language, depth, for robust 3D openvocabulary scene understanding.
• In order to acquire more comprehensive and intricate fine-grained geometric features from point clouds, we propose a hierarchical point cloud extractor that effectively captures both local and global features.
• We innovatively build hierarchical point-semantic caption pairs that offer coarse-to-fine supervision signals, facilitating learning adequate point-caption representations from various 3D viewpoints directly.
• UniM-OV3D outperforms previous state-of-the-art methods on 3D open-vocabulary semantic and instance segmentation tasks by a large margin, covering both indoor and outdoor scenarios.

2 RelatedWork

2.1 3DRecognition

To process point clouds directly, PointNet [Qi et al., 2017a] and PointNet++ [Qi et al., 2017b] are the two pioneering studies in this field that focus on developing parallel MLPs for feature extraction from unstructured data which have significantly enhanced accuracy [Huang et al., 2023a; Huang et al., 2022]. Further, while the VLM [Radford et al., 2021; Caron et al., 2021] has achieved remarkable results in zero-shot or few-shot learning of 2D images by utilizing extensive image data, the availability of such internetscale data for 3D point clouds is limited. Consequently, extending the open-vocabulary mechanism to 3D perception is a nontrivial task due to the difficulty in gathering sufficient data for point-language contrastive training, similar to the approach employed by CLIP [Radford et al., 2021] in 2D perception. Therefore, PointCLIPs [Zhang et al., 2022; Zhu et al., 2023] take the first step to convert point clouds into CLIP-recognizable images and align the point cloud feature with the language feature by projecting 3D point cloud into 2D space. Similarly, CLIP2Point [Huang et al., 2023b] attempts to align the projected depth map with the images by designing a trainable depth encoder and cross-modality learning losses. Different from point projecting, a series of works [Xue et al., 2023a; Xue et al., 2023b; Zeng et al., 2023; Liu et al., 2023b] try to collect multi-modalities triplets to train the 3D backbone. To enable zero-shot capability and improve standard 3D recognition, they align the 3D feature with the CLIP-aligned visual and textual features. This alignment facilitates the integration of 3D features with CLIP, enhancing both the zero-shot capability and the standard 3D recognition capability. However, existing methods cannot integrate the four modal information types, namely point cloud, image, depth, and text, which is the focus of this paper. We propose a unified multimodal network that incorporates fine-grained feature representation from these four modalities, effectively leveraging their strengths.

2.2 Open-Vocabulary 3D Scene Understanding

Facing the challenge of the lack of diverse 3D openvocabulary datasets for training generalizable models, one solution is to distill knowledge between pre-trained 2D openvocabulary models and 3D models [Wu et al., 2023a]. OpenScene [Peng et al., 2023] and CLIP-FO3D [Zhang et al., 2023] propose to distill the knowledge of 2D CLIP into 3D feature without any extra annotation. OpenMask3D [Takmaz et al., 2023] first generates class-agnostic instance mask proposals with the 3D point cloud, which is used to choose the 2D views and masks. Meanwhile, 3D-OVS [Liu et al., 2023a] distills the open-vocabulary multimodal knowledge and object reasoning capability of CLIP into a neural radiance field. OpenIns3D [Huang et al., 2023c] employs a "Mask-SnapLookup" scheme to learn class-agnostic mask proposals in 3D point clouds. Another line focuses on point-language contrastive training. To obtain 3D backbone-language alignment, PLA-family [Ding et al., 2023b; Yang et al., 2023; Ding et al., 2023a] uses 2D images as a bridge to generate descriptions of point cloud data. Although carefully designed, describing point clouds through images may not fully capture the details and point-semantic information due to differences between point clouds and images. Therefore, methods that describe point cloud data more directly and accurately need to be further explored. So we establish a point-semantic caption learning mechanism, in which captions can be obtained directly from point clouds with more accurate coarse-to-fine languages supervisions.

3 Method

3.1 Preliminary

The objective of open-vocabulary 3D scene understanding is to localize and recognize unseen categories without the need for corresponding manual annotations. To put it formally, annotations on semantic and instance levels, denoted as Y = {yB , yN }, are split into base categories CB and novel categories CN . During the training phase, the 3D model has access to all point clouds P = {p}, but only to annotations for base classes CB . It remains unaware of both annotations yN and category names related to novel classes CN . However, during the inference process, the 3D model is required to localize objects and classify points belonging to both basic and novel categories CB ∪ CN . This inference step is crucial for achieving comprehensive scene understanding and enabling the model to generalize to unseen classes. During inference, the model can predict any desired category by computing the similarity between point-wise features and the embedding of queried categories, enabling open-world inference.

3.2 Hierarchical Feature Extractor with Local and Global Fusion

Taking the sparse point clouds as input, we propose a trainable hierarchical point cloud extractor to capture fine-grained local and global features, instead of just utilizing the frozen 3D extractors. As illustrated in Figure 2, the input is channeled into a transformer network, which employs attentionbased layers to regress a 4×4 transformation matrix. This matrix comprises the elements that represent the learned affine transformation values, which are used for aligning point clouds. Following alignment, the points are introduced into multiple stacked spatial-aware layers, which serve to produce a permutation-invariant embedding of these points. Within this structure, the residual attention module functions as a linking bridge between two adjacent layers, facilitating the transfer of information. After the information has been processed through the attention-based layers, the outputs from all these N-dimensional layers are concatenated. Finally, the segmentation head can be added to output the global information aggregation for the point clouds, providing a comprehensive summary of the data.

Spatial-aware Layers. The architecture of the spatialaware layers is shown in Figure 2, where the current i layer acts as an intermediate layer and the features are not only transmitted by the mainstream information flow but the residual attention linking from i - 1 layer. Consider a RD dimensional embedding point clouds set with n points in the i layer p = {pi,1, pi,2, ..., pi,n}. In each attention-based flow, the local PointBERT [Yu et al., 2022] layer and the global PointMAE [Pang et al., 2022] layer process these points in parallel. In the branch of the local PointBERT layer, a transformation is employed on the points to get the feature HiL:

where B : RD → RM and g is the local patch. After that, the attention map from i - 1 layer by the residual linking is added to the output HiL in a point-wise manner which can be written as:

where ⊗ denotes element-wise multiplication. ω1 functions as a feature filter, dampening the impact of noisy features while amplifying the beneficial ones. Similarly, global PointMAE layer branch applies global transformation E on the points p and the output from decoder denotes HiG. And the final output after the linking of attention block is:

where ω2 is similar to ω1. The outputs f iL and f iG from the two branches are of the same dimension and are transferred to the attention block of the next layer. Ultimately, they are concatenated together, and these combined embedding are used as the input for the subsequent layer.

The attention block consists of a series of residual units [He et al., 2016], inserted with interpolation to form a symmetrical top-down architecture. Two consecutive 1×1 convolution layers are connected at the end to balance the dimensions. More details are presented in the supplementary.

3.3 Point-semantic Caption Learning

Recent success of open-vocabulary works in 2D [Bangalath et al., 2022; Wu et al., 2023b] and 3D [Ding et al., 2023b; Yang et al., 2023] vision has shown the effectiveness of introducing language supervision to guide the model to access abundant semantic concepts with a large vocabulary size. However, utilizing images as a bridge to associate the points

Input Point Cloud

Point-Cloud Based Caption

The bathroom features a toilet, a sink, and a bathtub. The toilet is located on the right side of the bathroom, while the sink is positioned on the left side .……

There are several chairs, some of which are placed near the center of the room. A couch can be seen in the middle of the room, and a bed with a trash can is located towards the right side ……

Figure 3: Point-caption pairs comparison. The middle column represents the captions generated by the point-captioning model, PointLLM [Xu et al., 2023], and the third column represents the corresponding image captions.

with language is the core idea in these 3D works. And the final textual descriptions in the so-called point-caption pairs are all sourced from image descriptions and do not possess semantic information specific to the point cloud. To address this problem, we propose to provide direct language supervision and a coarse-to-fine point-semantic caption generation mechanism is designed.

Global-view Point-semantic Caption. Due to the similarity of point-captioning and image captioning [Mokady et al., 2021; Wang et al., 2022] as fundamental tasks, various base models [Xu et al., 2023; Guo et al., 2023] trained on a large number of 3D point samples are already available for perceiving 3D scenes. Given the ith 3D point cloud scene pi, the simplest and most rudimentary manner is to link language supervision to all points. Specifically, the corresponding global language description tglobal i can be generated by the pre-trained point-captioning model G as:

Surprisingly, with the semantic understanding on the overall points, the entities covered in the generated captions have already encompassed the entire 3D semantic label space. As illustrated in Figure 3, the middle column represents the captions generated by the point-captioning model, and the third column represents the corresponding image captions. It is observable that the global point-captions not only offer a more precise and holistic depiction of the scene, but they also represent the orientation information of objects more accurately within the scene and the interrelationships among them. Despite the simplicity of global-view caption, we find that it can significantly improve the whole open vocabulary capabilities of understanding the scene and locating objects.

Hierarchical Point-semantic Caption. Diving into the exploration of more fine-grained point-semantic caption learning, global-view point cloud captions are still too coarse to capture fine-grained details between objects in point clouds, making them suboptimal for 3D scene understanding tasks. Hierarchical point captions can further refine the semantic information of point clouds without ignoring all object features in the scene. Therefore, we propose two other

Image-Based Caption

A white toilet sitting to a toilet paper dispenser. a white towel hanging on the side of a bathroom door .……

A black suitcase sitting on a wooden table. a black backpack with a black handle on a wooden floor. a small room with a bed and a trash can .……

association patterns on point sets at different 3D viewpoints. For each 3D point cloud scene pi, we initiate from the x-axis and segment the entire point cloud space into sectorshaped regions with an angular magnitude of θ degree, where the intersecting angle between adjacent sectors is ϕ degree. The segmented 3D scene psi can be denoted as:

For the partitioned sector-shaped regions pθ ϕ ij , we categorize them into two distinct types of viewpoints based on the magnitude of the angle, named as sector-view and eye-view. The corresponding angle size of the two views is θ1 and θ2, where 0 < θ1 ≤ 90 and 90 < θ2 ≤ 180. This implies that the sector-view focuses on the minutiae of local point cloud information, while the eye-view concentrates on a broader point cloud area, yet the extent of this area is within an obtuse angle range, akin to human eyes.

Contrastive Point-semantic Caption Training. With the obtained global-, eye- and sector-view pointsemantic captions from different views, we can conduct point-language feature contrastive learning to guide the 3D network to learn from vocabulary-rich language supervisions.

Specifically, we extract the text embeddings f t i based on the pre-aligned and fixed text encoder in PointBIND [Guo et al., 2023]. And the 3D feature f p i can be acquired by the trainable 3D point cloud encoder. We formulate the point-totext alignment using the contrastive loss as:

where m ∈ {global, eye, sector}, N is the total number of the point-semantic caption pairs and τ is a learnable temperature parameter. With Equation 6, we can easily compute different caption losses on global-view Lglobal capt , eye-view Leye capt and sector-view Lsector capt . The final caption loss is a weighted combination between different views as follows:

where α, β and γ are used to balance the relative importance of different parts and are set to 1, 0.8, 0.8 by default. The effect of different views and the comparison of different combinations are shown in ablation studies.

3.4 AligningMultimodal Representations

With the construction of the four modalities, the subsequent objective of UniM-OV3D is to conduct dense alignment between the 3D points with their corresponding image, depth and text.

For the paired image-text (I, t) data, we utilize their corresponding encoders from PointBIND [Guo et al., 2023] for feature extraction, which are kept frozen during training. For each depth map D, we set the gated aggregator weights to 0 in CLIP2Point [Huang et al., 2023b] to extract the depth feature, formulated as

where f t , f img , f d denote the text, image and depth embeddings. And the depth encoder Ed keeps learnable during training. For the 3D point clouds, we adopt the designed hierarchical point cloud feature extractor module to extract 3D features for the point cloud. In order to transform the encoded 3D feature into multi-modal embedding space, we append a projection network after the 3D encoder Ep. So the final 3D features f p can be formulate as:

Dense Associations across Modalities. As shown in Figure 2, with a 3D scene and its corresponding features f p , f t , f img , f d , we employ a contrastive loss between point clouds and other modalities, which effectively compels the 3D embeddings to align with the joint representation space, formulated as:

where M1 and M2 represent two modalities in (point, image, depth) and (i, j) indicates a positive pair in each training batch. And ϵ is a learnable temperature parameter, similar to CLIP [Radford et al., 2021].

Finally, the overall objective function to perform unified multimodal alignment is defined by:

where the language modality delivers comprehensive and scalable textual descriptions, while the image modality provides accurate guidance on object edges and contextual data. Moreover, the depth and 3D modality exposes vital structural details of objects. By aligning these modalities jointly in a common space, our approach can maximize the synergistic advantages among them, resulting in superior openvocabulary scene understanding performance.

4 Experiments

4.1 Setup

Datasets and Metrics. To validate the effectiveness of our proposed UniM-OV3D, we conduct extensive experiments on four popular public 3D benchmarks: ScanNet [Dai et al., 2017], ScanNet200 [Rozenberszki et al., 2022], S3DIS [Armeni et al., 2016] and nuScenes [Caesar et al., 2020]. The first three provide RGBD images and 3D meshes of indoor scenes, while the last one supplies Lidar scans of outdoor scenes. We use all four datasets to compare to alternative methods, covering both semantic and instance segmentation. For semantic segmentation, we employ the commonly used 3D segmentation metric mean intersection over union (mIoUB , mIoUN ) and harmonic mean IoU (hIoU) for evaluating base, novel categories and their harmonic mean separately. For instance segmentation, We apply mean average precision under 50% IoU threshold (mAPB50 , mAPN50 , hAP50) as major indicators.

Category Partition. ScanNet densely annotates 20 classes, ScanNet200 owns 200 classes, S3DIS contains 13 classes and nuScenes involves 16 classes. Following [Ding et al., 2023b; Yang et al., 2023], we disregard the "otherfurniture" class in ScanNet and randomly partition the rest 19 classes into 3 base/novel partitions, i.e. B15/N4 (15 base and 4 novel categories), B12/N7 and B10/N9, for semantic segmentation. Following SoftGroup [Vu et al., 2022] to exclude two background classes, we acquire B13/N4, B10/N7, and B8/N9 partitions for instance segmentation on ScanNet. Similarly, we ignore the "clutter" class in S3DIS and get B8/N4, B6/N6 for both semantic and instance segmentation. For ScanNet200, we split 200 classes to B170/N30 and B150/N50. For nuScenes, we drop the "otherflat" class and obtain B12/N3 and B10/N5.

4.2 Main Results

To validate the effectiveness of our proposed UniM-OV3D, we compare it with previous state-of-the-art 3D openvocabulary methods on semantic and instance segmentation tasks.

3D Semantic Segmentation. As shown in Table 1, compared to previous captioning-learning based method RegionPLC [Yang et al., 2023], we obtain 5.1%-13% mIoU performance gains among different partitions on ScanNet. Compared with previous zero-shot state-of-the-art method OpenIns3D [Huang et al., 2023c], our method still obtains 3.2%-5.4% and 5.5%-7.8% improvements in terms of hIoU across various datasets and partitions. It is noteworthy that our approach consistently improves performance across different datasets and partitions. Moreover, for partitions with more novel categories, our method shows even greater enhancement. This further demonstrates the robustness and effectiveness of our approach in open vocabulary scene understanding tasks.

When facing the long-tail problems in ScanNet200, as shown in Table 2, our method still surpasses corresponding zero-shot method by 5.1%-6.9% hIoU on both splits and 3.5%-5.2% mIoU on novel categories. Furthermore, when evaluating the performance of our UniM-OV3D on outdoor setting, our method lifts the hIoU and mIoU of novel categories by 4.8%-6.4% and 5%-5.6% as shown in Table 3. In this regard, UniM-OV3D demonstrates superior generalizability to complex 3D open-world scenarios.

3D Instance Segmentation. Our pipeline not only offers point-semantic language descriptions for fine-grained point clouds, but also encourages points to learn discriminative features, which in turn benefits the instance-level localization task. As shown in Figure 4, by learning a uni-modalities feature representations, our method outperforms RegionPLC [Yang et al., 2023] by 4.1% hAP50, 5.3% mAPB50 , 3% mAPN50 on B13/N4 on the ScanNet. On the other datasets, our UniM-OV3D surpasses OpenIns3D [Huang et al., 2023c] by 5%-10.6% hAP50, 5%-5.3% mAPB50 and 3.1%-13.2% mAPN50 on their respective B8/N4, B12/N3 and B170/N30 partitions (RegionPLC only reports their instance segmentation results on ScanNet without open-source code). This illustrates the efficacy of our UniM-OV3D in empowering the model to identify unseen instances without human annotations.

4.3 Ablation Studies

In this section, we change components and variants of our proposed UniM-OV3D by conducting extensive ablation studies on ScanNet dataset for semantic segmentation by default. And the partitions are mainly B15/N4 and B12/N7.

Analysis of Different Components. We illustrate the importance of different components of our network by removing some parts and keeping all the others unchanged. The baseline setting is to only use three modalities of image, text and point cloud. The 3D point cloud encoder uses sparse convolutional UNet [Choy et al., 2019]. The point captions come from corresponding image captions, and we use CLIP as the image and text encoder. As shown in Table 4, using the depth map to construct the uni-modality learning contributes much to the whole performance which proves the necessity of establishing a unified modal architecture. Without the point-semantic captions from different view, the performance drops dramatically which shows that coarse-to-fine supervision signal from fine-grained point-semantic caption learning is indeed a crucial factor. And the performance degradation caused by the absence of hierarchical point cloud extractor proves that only the combination of local and global features can produce the best results.

Comparison of Different Fusion Methods in Hierarchical Feature Extractor. To validate the effect of different fusion methods in the hierarchical point cloud feature extractor, we design six fusion patterns as shown in Table 5. With the attention block to transfer features between connected layers, the fused models override the original ones by 0.8%-2.3% IoU. And our final fusion method of global and local with attention block outperforms the alternative by 0.5%-1.4% mIoU.

Effect of Different Point-semantic Caption Views. In order to explore the effect of dividing the point cloud into different angles in Section 3.3 to obtain the corresponding finegrained point-semantic description, we split the point cloud into six angles for experiments. As shown in Table 6, the 120 degree in eye-view and 60 degree in sector-view perform relatively well, but they are both slightly inferior to global-view (360 degree). So, in order to further obtain the coarse-to-fine supervision, we combine 60 degree and 120 degree with 360 degree respectively, and finally adopt the method of combining the three in the last row.

5 Conclusion

In this paper, we propose UniM-OV3D, a unified muitimodal network for open-vocabulary 3D scene understanding. We jointly establish dense alignment between four modalities, i.e., point clouds, images, depth and text, to leverage their respective advantages. To fully capture local and global features of point clouds, we stack spatial-aware layers to construct a hierarchical point cloud extractor. Furthermore, we deeply explore the possibility of directly using point clouds to generate corresponding captions, and the established pointsemantic learning mechanism can provide language supervision signals more effectively. Extensive experiments on both indoor and outdoor datasets demonstrate the superiority of our model in 3D open-vocabulary scene understanding task.

Contribution Statement

All authors conceive the paper's methodology and experiments. Qingdong He and Jinlong Peng contribute equally to this work and should be considered co-first authors. Yabiao Wang and Jiangning Zhang are the co-corresponding authors of this paper.