Point Cloud Mixture-of-Domain-Experts Model for 3D Self-supervised Learning

1 Introduction

Recently, with the rapid development of 3D scanning technology, 3D point clouds have become the mainstream representation for 3D objects due to their ease of acquisition, explicit representation, and efficient storage. Point clouds can be categorized into scene domain point clouds [Dai et al., 2017; Song et al., 2015; Armeni et al., 2016] and object domain point clouds [Wu et al., 2015; Chang et al., 2015; Uy et al., 2019] based on the modeling object. As shown in Figure 2 (a), object domain point clouds describe specific objects or entities, such as an airplane, with relatively fewer points. Scene domain point clouds represent the entire environment or scene, such as indoor scenes, including multiple objects, structures, and background elements, with a larger number of points. Due to the significant disparity in point count and the elements being described, a substantial domain gap exists in these two types of point clouds.

∗Corresponding author.

Point cloud self-supervised learning (SSL) [Yu et al., 2022; Pang et al., 2022; Zhang et al., 2022; Dong et al., 2023], pretrained on massive point cloud data, have become the mainstream paradigm for learning 3D representations and have been widely transferred to various point cloud tasks [Zhou et al., 2024; Zha et al., 2025]. Despite the significant success, most of these methods are domain-specific due to the notable domain gap between object-domain and scene-domain point clouds. As shown in Figure 2 (a), these methods [Yu et al., 2022; He et al., 2022; Zhang et al., 2022] use scene-level data to learn scene model and object-level data for object model. This domain-independent learning approach not only fails to effectively integrate knowledge across different domains, but also struggles to perform well on data from multiple domains simultaneously. For example, Point-MAE [Pang et al., 2022], which is pre-trained on ShapeNet [Chang et al., 2015], primarily performs object point cloud tasks. For scene point clouds, it requires re-pretraining on scene-level datasets like ScanNet [Dai et al., 2017] to adapt to the scene domain. As shown in Figure 2 (c), directly transferring an object domain Point-MAE to scene tasks results in a significant performance drop. Similarly, models pre-trained on the scene domain also exhibit performance declines when transferred to object task.

8 6

8 4

8 2

8 0

Figure 2: Transferability of the domain-specific Point-MAE model. Transferring a Point-MAE pre-trained on the scene to an object task results in a clear performance drop, and the same occurs for models in the object domain..

Learning a more comprehensive 3D representation by simultaneously integrating knowledge from both the object domain and the scene domain is a reasonable solution. However, it is highly challenging for two main reasons. Firstly, the input data is inconsistent. Scene-level point clouds, such as ScanNet [Dai et al., 2017], typically consist of 50k points, while object-level point clouds like ModelNet [Wu et al., 2015] typically consist of 1k points. The disparity in point count between the two types is significant, making it difficult to process both types of data simultaneously using a single model. Secondly, there is inconsistency in task emphasis. Scene point clouds typically involve object detection or segmentation, which often prioritizes understanding fine-grained local point clouds. On the contrary, object point clouds generally involve classification tasks, which tend to prioritize understanding global geometry.

To address the aforementioned challenges, we propose a block-to-scene pretraining strategy to pre-train a Point cloud Mixture-of-Domain-Experts model (Point-MoDE). We address the challenge of inconsistent input data by using domain-specific experts to process data from their respective domains. Additionally, we finetune the pre-trained model to address the inconsistency of task emphasis. Specifically, as shown in Figure 2 (b), we first design a point cloud mixtureof-domain-experts model that consists of a scene domain expert and multiple shared object domain experts. In the finetuning phase, for object domain data, our model selectively activates the object-domain expert for analysis. However, in the case of scene point clouds, we activate multiple shared object expert encoders to assist the scene expert in analyzing scene domain data collaboratively.

Furthermore, we propose a block-to-scene pre-training strategy that couples masked reconstruction and position regression tasks of random object blocks within a scene for selfsupervised training, enabling us to train experts for different domains simultaneously. Specifically, we first randomly select point blocks within a scene and apply a set of transformations to convert each point block coordinates from the scene space to the object space. Then, within the object expert, we use a mask and reconstruction pipeline to recover the masked points of each block, enabling it to learn universal object representations. Finally, we introduce a scene-expertbased block position regression pipeline, which utilizes the blocks' features in the object space to regress these blocks' initial positions within the scene space, enabling the scene expert to learn scene representations with the assistance of the object experts. By block-to-scene pretraining, our PointMoDE can simultaneously learn powerful object-level and scene-level representations and exhibit superior transferability. Our model can be fine-tuned directly on downstream tasks such as object point cloud classification, segmentation, completion, and scene point cloud detection without the need for any additional domain adaptation training.

The main contributions can be summarized as follows:

• We propose a mixture-of-domain-experts model, integrating both scene domain and object domain experts to effectively process point clouds from different domains.
• We introduce a novel block-to-scene pretraining strategy that combines masked reconstruction and position regression tasks, enabling the simultaneous learning of object-level and scene-level representations.
• Extensive experiments across different datasets and tasks demonstrate the superiority and transferability of our model.

2 RelatedWork

2.1 Self-supervised Learning for Point Cloud

Self-supervised learning, which enables the learning of general representations from large amounts of unlabeled data, has been widely applied in fields such as language [Semnani et al., 2019; Brown et al., 2020; Achiam et al., 2023] and image [Bao et al., 2021; Chen et al., 2020b; Chen et al., 2020a; He et al., 2022]. Inspired by the success of visual pretraining, numerous point cloud pretraining methods have also been proposed. Based on the pretext tasks, they can be categorized into contrastive learning paradigms [Oord et al., 2018; Tian et al., 2020] and masked reconstruction paradigms [Bao et al., 2021; He et al., 2022]. PointContrast [Xie et al., 2020], CrossPoint [Afham et al., 2022], and DepthContrast [Zhang et al., 2021] construct positive and negative sample pairs using various methods and employ contrastive learning techniques to learn 3D representations. Point-BERT [Yu et al., 2022] was the first to propose learning universal 3D representations using the paradigm of masked reconstruction. Subsequently, numerous explorations have improved masked reconstruction from various perspectives. Point-MAE [Pang et al., 2022] and Point-M2AE [Zhang et al., 2022] introduced the masked autoencoder for reconstruction, and PointGPT [Chen et al., 2024] proposed pretraining using an autoregressive approach. To address the limited amount of point cloud during pretraining, many approaches integrate multimodal knowledge to aid in learning point cloud features. ACT [Dong et al., 2023] leverages a pre-trained image model to assist in point cloud reconstruction, while I2PMAE [Zhang et al., 2023] employs an image-guided masking strategy. Despite the significant success, the above methods are domain-specific. In this paper, we propose training a mixture-of-domain-experts model to integrate knowledge from different domains for learning comprehensive 3D representations.

3 Methodology

In this section, we provide a detailed explanation of how to use our block-to-scene pretraining strategy to train our point cloud mixture-of-domain-experts model.

3.1 Point CloudMixture-of-Domain-Experts

The overall architecture of our Point-MoDE is shown in Figure 3, it is composed of four main components: a scene expert encoder, a scene expert decoder, a shared object expert encoder, and a shared object expert decoder. It is primarily used for two main task pipelines: object point cloud processing and scene point cloud processing.

Our Point-MoDE is a hybrid model. Due to significant differences across domains of point clouds and tasks, we selectively activate different sub-modules for various downstream data and tasks. For instance, in tasks such as object point cloud classification and object part segmentation, we selectively activate our object expert encoder according to specific tasks. For scene point cloud detection tasks, we activate all encoders. This collaborative approach is primarily adopted because utilizing the object expert encoder for analyzing local point blocks within the scene contributes to the scene encoder's comprehension of scene intricacies.

In the scene expert model, we adopt a standard Transformer as our scene expert, comprising 3 layers of Transformer-based encoders and 8 layers of Transformerbased decoders with a PointNet-based [Qi et al., 2017a] token embedding layer. In our object expert model, we directly leverage existing pre-trained object models based on maskbased point modeling, such as Point-BERT [Yu et al., 2022], Point-MAE [Pang et al., 2022], PointGPT [Chen et al., 2024], etc. Using these models pre-trained on the object domain allows us to incorporate object-level priors into our pipeline while saving training resources and enabling better extension of existing models. This characteristic makes our object experts flexible, allowing us to replace them with different backbones.

3.2 Block-to-Scene Pretraining

Our block-to-scene pretraining primarily consists of the following three key components: random point block generation, object-level block masked reconstruction, and scenelevel block position regression. Below, we provide a detailed illustration of the specific implementation of each component.

Random Point Block Generation

Random point block selection. To leverage scene local details for scene understanding in an unsupervised manner, we randomly select Ko local point blocks from the entire scene. We first random select Ko points from the entire scene point cloud as the center points for each point block. For each center point, we then use the K-nearest neighbors algorithm to select the nearest No points around it, forming the initial point block objects B = {B1 , ...,BKo }, where the i-th point block is Bi ∈ RNo×3 .

Ground-truth block position generation. We generate the ground truth position of each random point block in the scene, which will be used to constrain the predicted position regressed by the final scene decoder. Inspired by the detection [Carion et al., 2020; Misra et al., 2021; Dai et al., 2021] task, we use the 3D bounding box of each random point block as its ground truth position. By computing the mean of all points in each dimension of the entire point block, the coordinates of the center point are obtained. The half-lengths of the bounding box in each dimension are calculated by subtracting the minimum value from the maximum value in each dimension and dividing by 2. Subsequently, the center point coordinates and half-lengths in each dimension (x, y, and z) are concatenated to form the bounding box. Finally, standard procedures [Misra et al., 2021] are applied to compute bounding box parameters Bb such as size and corners for each bounding box.

Object-level Block Masked Reconstruction

We first initialize our object expert model using the PointMAE (or Point-BERT, PointGPT, etc.) model pre-trained on ShapeNet [Chang et al., 2015]. Since these models are opensource, we directly use the pre-trained weights provided by the official repository. This initialization allows us to incorporate object-level knowledge priors into our pipeline without the need for additional object data or specialized training.

Object point block generation. We treat each randomly selected point block in B as an object point block and transform its coordinates from the scene space to the object space for processing by the object expert. Specifically, we apply a simple set of transformation functions to each point block. First, we subtract the coordinates of the center point from the No local points to obtain the relative coordinates of each point. Then, we normalize these coordinates to the range [-1, 1]. Finally, after applying a random rotation transformation to each point block, we obtain all point blocks Bo = {B1 o , ..., BKo o } as input to the object expert encoder. Through these transformation functions, we convert the coordinates of each point block from the scene space to the object space, decoupling the point block object coordinates from the original scene coordinates. This enables the object expert to learn the universal shape features of the point block objects.

Object point block masked and reconstruction. We use a shared object autoencoder to perform mask-based reconstruction self-supervised learning on all generated object point blocks Bo, enabling our object expert to learn a general representation of objects. We illustrate the entire mask and reconstruction process for the object point blocks using PointMAE [Pang et al., 2022] as an example. For the i-th object point block Bio ∈ RNo×3 , we use farthest point sampling and the K-nearest neighbors algorithm to divide it into Mo point patches. Then, after randomly masking most of the patches, we generate initial tokens and positional encodings for each unmasked patch using MLP-based token encoding and positional encoding. By adding them, we obtain the token E0 o ∈ RrMo×Co for each unmasked patch, where r represents the unmasked ratio, and Co denotes the object feature dimension. Finally, we use a shared object expert encoder to extract object features Eno o ∈ RrMo×Co , where no is the number of layers in the scene expert.

In the decoding phase, we concatenate Eno o with randomly initialized masked tokens to obtain D0 o ∈ RMo×Co . Then, we use a standard Transformer-based decoder to decode, getting Dmo o ∈ RMo×Co . Finally, we use an MLP-based reconstruction head to reconstruct the coordinates of the masked point patches Rio ∈ RNo×3 .

Scene-level Block Position Regression

Scene encoding. Given an input point cloud Ps ∈ RNs×3 with Ns points, we first use farthest point sampling and the Knearest neighbors algorithm to partition it into blocks. Then, using an MLP-based token encoding layer and a positional encoding layer, we generate the semantic token and positional encoding for each patch. By adding them, we obtain the token E0 s ∈ RMs×Cs for each patch, where Ms represents the number of scene patches, and Cs denotes the scene feature dimension. Finally, we use a scene encoder based on the standard Transformer [Vaswani et al., 2017] architecture to extract scene features Ens s ∈ RMs×Cs , where ns is the number of Transformer layers in the scene encoder.

Scene decoding and position regression. We apply max pooling to the features of all blocks output by the object encoder to obtain the global feature for each block. After passing through the projection layer, these point block features are transformed into features Bg ∈ RKo×Cs for scene decoding input. However, we stop the gradients of Bg from propagating backward into the mask reconstruction pipeline during the backpropagation process, thereby mitigating the multi-task interference caused by the scene regression task on the object encoder. We will provide a detailed explanation of this issue in Section 4.4. We then add the transformed point block features to randomly initialized queries to obtain enhanced queries Q0 ∈ Rq×Cs , where q is the number of queries. Since the number of point blocks and queries often differ, we replicate Bg to match all queries.

We use a Transformer decoder based on self-attention and cross-attention as our scene expert decoder. The input queries Q0 , with the assistance of the encoded features Ens s , pass through our decoder to obtain the decoded query features Qms , where ms is the number of scene expert decoder. Finally, we use different MLP-based reconstruction heads to predict the 3D bounding box Bp b of each random point block.

Loss Function

We use a combination of mask reconstruction loss and point block regression loss to jointly constrain our pre-training process. For the reconstruction loss calculation, we follow previous work [Pang et al., 2022; Zhang et al., 2022] and use Chamfer Distance [Fan et al., 2017] (CD) as the loss function. For the regression loss calculation, we use Generalized Intersection over Union [Rezatofighi et al., 2019] (GIoU) as the loss function. Therefore, our loss function is defined as

where λ1 and λ2 is a weighted combination of reconstruction loss and regression loss.

4 Experiments

First, we pre-train the Point-MoDE model using our blockto-scene pretraining strategy based on point cloud data from the scene domain. After pre-training, we directly transfer the pre-trained model to various downstream tasks in different point cloud domains for fine-tuning. During fine-tuning, we selectively activate different sub-modules depending on the domain of the point cloud; for instance, we activate the object expert encoder for object point clouds, while utilizing all encoders for scene point clouds. This strategy enables PointMoDE model, pre-trained with the block-to-scene approach, to outperform existing domain-specific models in most cases without any additional domain adaptation training, demonstrating the superiority and transferability of our model.

4.1 Dataset and Pre-training

We initialize our object expert model using the Point-MAE or other models pre-trained on ShapeNet [Chang et al., 2015], which can be directly obtained from these official repositories. Then, We combined all training data from the two most commonly used indoor scene datasets, SUNRGB-D [Song et al., 2015] and ScanNetV2 [Dai et al., 2017], to construct our block-to-scene pretraining dataset. Specifically, SUNRGBD includes 5K single-view RGB-D training samples with oriented bounding box annotations for 37 object categories. ScanNetV2 contains 1.2K training samples, each with axisaligned bounding box labels belonging to 18 object categories. We extracted 50K points for each of the 6.2K samples, using only the xyz coordinates of each point to construct the pretraining dataset.

During the pretraining phase, we input all 50K×3 point clouds into the scene encoder of Point-MoDE to extract scene-level features. Simultaneously, we randomly select 32 point blocks from the scene point cloud, with each block containing 2K local points, and input these into the 32 object encoders with shared parameters. We use the AdamW optimizer with a base learning rate of 5e-4 and a weight decay of 0.1. Simple rotation is applied as data augmentation to both the scene point cloud and each selected object point cloud. For the object point cloud masks, we set the mask ratio to 60% following previous work [Pang et al., 2022; Dong et al., 2023]. We train the model for 200 epochs using 8 A100 GPUs.

4.2 Fine-tuning on DownstreamTasks Object Point Cloud Classification

We first evaluate the performance of our model on object point cloud classification tasks using the object expert encoder of Point-MoDE. We conduct point cloud classification on two of the most commonly used object point cloud datasets: ScanObjectNN [Uy et al., 2019] and ModelNet40 [Wu et al., 2015]. ScanObjectNN contains 15K real scanned point clouds, each with various backgrounds, occlusions, and noise, which effectively assesses the model's robustness. ModelNet40 includes 12K synthetic point clouds belonging to 40 different categories, with each point cloud being complete and clean, providing a better representation of 3D object shapes.

Following previous work [Dong et al., 2023; Liang et al., 2024], we use 2K points as input for ScanObjectNN, apply simple rotation for data augmentation, and report classification accuracy without using voting. For ModelNet40, we use 1K points as input, apply scale and translate data augmentation, and report classification accuracy with the standard voting mechanism.

As presented in Table 1, firstly, compared to previous single-domain baselines (Point-MAE, Point-BERT, and PointGPT), our Point-MoDE method achieves consistent and stable improvements, indicating that leveraging scene domain knowledge to assist object domain learning is an effective strategy. This approach not only preserves object representations but also enables the learning of more comprehensive knowledge. Secondly, our Point-MoDE shows a significant improvement on the real-world scanned point cloud dataset ScanObjectNN, while the improvement on the synthetic point cloud dataset ModelNet40 is smaller. This is because we use randomly sampled scene data during pretraining, which lacks explicit semantics. Such non-semantic data is more similar to real scanned point clouds, thereby aiding the model in learning more realistic point cloud representations.

Object Point Cloud Part Segmentation

We assess the performance of our Point-MoDE in part segmentation using the ShapeNetPart dataset [Chang et al., 2015], comprising 16,881 samples across 16 categories. We utilize the same segmentation setting after the pre-trained encoder as in previous works [Pang et al., 2022; Zhang et al., 2022] for fair comparison. The experimental results displayed in Table 2. Our method also achieves consistent improvements compared to single-domain approaches, demonstrating the superiority of leveraging object knowledge to assist in scene tasks. This improvement appears less pronounced compared to detection tasks, which can be attributed to the inherently challenging nature of fine-grained tasks like segmentation. Therefore, even relatively small improvements are sufficient to highlight the effectiveness of our approach.

Scene Point Cloud Detection

We further fine-tune the pre-trained Point-MoDE on scenelevel object detection tasks. At this stage, we primarily rely on the scene expert encoder to process scene-level inputs. Simultaneously, we randomly select 32 point blocks from the scene point cloud and use the 32 shared object encoders to handle these local point blocks. During the decoding phase, we integrate the results from the scene expert encoder with random queries from the scene, helping the scene expert to better understand scene details. We use ScanNetV2 [Dai et al., 2017], to evaluate our model's scene understanding capabilities.

Table 3 shows our detection results, our Point-MoDE model has shown significant improvements compared with other single domain models. In particular, there is an improvement of 6-9% in the AP50 metric. This substantial improvement is mainly attributed to our Point-MoDE using multiple object encoders to assist the scene encoder in analyzing the overall scene. This approach enables the scene model to focus on more local details, thereby enhancing scene understanding.

4.3 Scene Semantic Segmentation

We have conducted experiments on scene-level semantic segmentation tasks to assess the performance of Point-MoDE in classifying each point in a scene into semantic categories. We validated our model using the indoor S3DIS [Armeni et al., 2016] dataset for semantic segmentation tasks. Specifically, we tested the model on Area 5 while training on other areas and report the mean IoU (mIoU) and mean Accuracy (mAcc). To ensure a fair comparison, we used the same codebase based on the PointNext [Qian et al., 2022a] baseline and employed identical decoders and semantic segmentation heads.

The experimental results are shown in the table 4. Compared to training the PointNeXt model from scratch, our method improves the mIoU score by 2.3%. It also shows significant improvements over other pretraining models, such as Point-MAE [Pang et al., 2022] and PointDif [Zheng et al., 2024]. This enhancement is largely due to our block-to-scene pretraining, which equips the model with strong scene understanding capabilities and further demonstrates the generalizability of our approach.

4.4 Ablation Study

The Impact of Different Pretraining Strategies

We first analyze the impact of each key component in the block-to-scene pretraining strategy. Our block-to-scene pretraining strategy consists of four key parts: scene-level block position regression (Scene B. P.), object-level block mask reconstruction (Object M. R.), coordinate space transformation (Coordinate S. T.) of object blocks (from scene space to object space), and joint pertaining (Joint T.) of objects and scenes (connecting objects and scenes via the Projection module in Figure 3). A detailed analysis of these components is provided in Table 5.

When using only Scene B. P. or Object M. R., the pretraining strategy enhances only the subsequent scene tasks or object tasks, failing to achieve simultaneous improvements. Even when both strategies are employed without Joint T., while simultaneous enhancement is possible, the inability of the two pipelines to interact prevents the complementary knowledge from being effectively learned, thereby limiting the representation capability. Only by jointly employing both regression and reconstruction can the most comprehensive representation be achieved.

Secondly, Coordinate S. T. is crucial for learning object representations. This is because the coordinates of blocks directly cropped from the scene belong to the scene space and are often not normalized within the range of [-1, 1]. In contrast, most object-level point cloud analysis assumes coordinates within the range of [-1, 1]. Thus, transforming the coordinates is essential. As shown in the table, omitting this transformation results in performance degradation on object tasks. This is because the scene-space coordinates disrupt the learning of object representations, highlighting the importance of this normalization step for effective object-level analysis.

The Impact of Stop Gradients

In our implementation, gradient stopping plays a crucial role. This mechanism allows different encoders within the model to jointly learn domain-specific representations during block-to-scene pretraining without interfering with each other. Thus, ensuring the accuracy and independence of each encoder's learning objectives during training is of paramount importance.

Since the position regression objective and the masked reconstruction objective are two distinct tasks in our pretraining process, failing to decouple the learning processes of the different encoders could lead to catastrophic forgetting. For instance, without gradient stopping, gradients from the object-level reconstruction tasks could backpropagate into the scene's encoder, interfering with its ability to learn scenelevel knowledge. By applying gradient stopping, we effectively prevent this interference, ensuring that each encoder remains focused on its specific task and thereby avoiding catastrophic forgetting. Table 6 reports the impact of applying gradient stopping. When the gradient stopping mechanism is removed, the AP50 on scene-level detection tasks drops by 7.3%, clearly indicating that the learning process of the scene encoder is disrupted. Similarly, significant performance degradation is observed in classification tasks. These results collectively demonstrate that decoupling the representation learning of different encoders is essential for preventing catastrophic forgetting.

The Impact of Object Expert Initialization

We further analyze the impact of initializing the object expert model. In our implementation, we initialize the model using an object model pre-trained on ShapeNet, after which no object data is used. This initialization ensures that we can learn representations from the object domain prior. We further examine the effect of not using object priors (Scratch), where no object domain data is used during training. The results, as shown in Table 7, indicate that the absence of object priors has a minimal impact on scene representation learning, resulting in only slight performance degradation. However, it significantly affects the learning of object point cloud representations. For the real point cloud ScanObjectNN, even when trained from scratch, the model eventually surpasses the one trained only on ShapeNet55 (88.27). However, for the synthetic point cloud ModelNet40, there is a larger gap compared to the model trained only on ShapeNet55 (93.8). This is because the point blocks cut from the scene lack specific semantics, making their distribution more similar to real point clouds but significantly different from the distribution of synthetic point clouds, leading to poorer performance on synthetic point clouds.

5 Conclusion

In this paper, we propose a block-to-scene pretraining strategy to train a mixture-of-domain-experts model that seamlessly integrates scene domain and object domain experts. This strategy enables the simultaneous acquisition of objectlevel and scene-level representations, enabling the learning of comprehensive 3D representations. Extensive experiments conducted across various datasets and tasks validate the superiority and transferability of our proposed approach. Our findings highlight the potential of leveraging cross-domain knowledge to achieve robust and versatile 3D representation learning, paving the way for future study.

Acknowledgements

• This work is supported in part by the National Natural Science Foundation of China, under Grant (62302309,62171248), Shenzhen Science and Technology Program (JCYJ20220818101014030,JCYJ20220818101012025), and the PCNL KEY project (PCL2023AS6-1).