DyG-Seg: Unsupervised 3D Point Cloud Segmentation via Geometric Manifold Rectification

1 Introduction

Semantic segmentation of 3D point clouds serves as a cornerstone capability for autonomous driving, embodied AI, and robotic navigation [Dai et al., 2017; Armeni et al., 2017]. While fully supervised paradigms have yielded remarkable progress on specific benchmarks [Qi et al., 2017a; Qi et al., 2017b; Graham et al., 2018], their heavy reliance on extensive point-wise human annotations acts as a severe bottleneck to scalability in open-world scenarios. To alleviate the dependency on expensive labeling, unsupervised 3D segmentation has emerged as a compelling research direction [Zhang et al., 2023; Chen et al., 2023b; Zhang et al., 2025]. However, prevalent methodologies typically operate under a strong assumption: the number of semantic categories K in a scene is a priori known and static. In realistic open environments, scene complexity and object variety are inherently dynamic. Prescriptively fixing the cluster number is not only impractical but often induces semantic ambiguity, leading to erroneous merging or over-segmentation. Consequently, this paper addresses a more challenging yet practical task: achieving automatic semantic discovery and segmentation of 3D scenes without pre-specifying the category count K and devoid of human supervision.

∗Corresponding authors.

Despite recent strides in leveraging self-supervised features or multi-modal distillation [Caron et al., 2021; Oquab et al., 2023; Kirillov et al., 2023; Peng et al., 2023], substantial hurdles remain when applying existing unsupervised methods to complex indoor and outdoor environments. The foremost bottleneck stems from the rigidity of a fixed K. Most approaches depend on standard clustering algorithms (e.g., K-Means) that force data into a predetermined number of partitions [Caron et al., 2018; Ji et al., 2019; Zhang et al., 2023]. When the preset K mismatches the true semantic structure, the model fails to adapt, causing distinct objects to fuse or single entities to fragment. The second critical limitation is the geometric incompleteness inherent in raw point clouds. Unlike dense and regular 2D images, 3D scans often suffer from occlusion, sensor noise, and varying density, resulting in fragmented object structures. Conventional feature extractors struggle to infer the integral geometric manifold from these incomplete observations, leading to indistinct feature representations that are easily overwhelmed by dominant background patterns. Fundamentally, the inability to dynamically adapt to unknown cluster counts and the failure to rectify incomplete geometric structures represent two pivotal constraints restricting current unsupervised performance.

To bridge these gaps, we propose DyG-Seg, a novel unsupervised framework designed to achieve robust scene segmentation via geometric-semantic alignment. Unlike traditional methods that perform static clustering directly on raw features, we reformulate the task as a progressive process of geometric recovery, structural inference, and semantic learning. First, to address the issue of geometric incompleteness, we introduce the Geometric Manifold Rectification (GMR) module. This module leverages a generative reconstruction task to recover latent manifold structures from incomplete inputs, thereby enhancing the integrity of geometric feature representations. Second, to tackle the unknown K problem, we propose the Dynamic Prototype Contrastive Clustering (DPCC) strategy. By integrating non-parametric Bayesian inference [Ferguson, 1973] with contrastive learning, DPCC dynamically infers the optimal number of clusters and learns compact class prototypes from the data distribution. Finally, we implement an Iterative Optimization with Dynamic Constraints mechanism, which cyclically refines pseudo-labels and feature embeddings to ensure robust convergence in complex scenes.

Our main contributions are summarized as follows:

• We propose DyG-Seg, a unified unsupervised framework capable of automatically discovering the number of categories and performing accurate segmentation without human labels.
• We design the Geometric Manifold Rectification (GMR) module, which effectively recovers latent manifolds from incomplete point clouds, significantly improving the geometric completeness of learned features.
• We introduce the Dynamic Prototype Contrastive Clustering (DPCC) strategy integrated with an Iterative Optimization with Dynamic Constraints mechanism. This synergy allows the model to adaptively discover semantic categories while enforcing spatial consistency and class balance.
• Extensive experiments on the challenging ScanNet and S3DIS datasets demonstrate that our method significantly outperforms existing state-of-the-art unsupervised approaches.

2 RelatedWork

2.1 Unsupervised 3D Semantic Segmentation

Early approaches primarily relied on fully supervised architectures like point-based MLPs [Hu et al., 2020; Qi et al., 2017a; Qi et al., 2017b] or sparse convolutions [Graham et al., 2018; Zhu et al., 2021]. Despite their effectiveness, prohibitive annotation costs accelerated the shift towards weakly supervised paradigms. While methods utilizing sparse annotations [Hu et al., 2022; Liu et al., 2021; Unal et al., 2022] or 2D tags [Chibane et al., 2022; Wei et al., 2020] mitigate labeling efforts, they still necessitate human intervention. Recently, fully unsupervised segmentation has advanced significantly. For instance, GrowSP [Zhang et al., 2023] proposes progressive superpoint growing, PointDC [Chen et al., 2023b] leverages self-supervised distillation with soft clustering, and LogoSP [Zhang et al., 2025] utilizes global graph grouping in the frequency domain. However, these methods typically operate under the assumption of a fixed category count. Moreover, they cluster raw sparse features directly, neglecting the geometric manifold discontinuity inherent in point clouds, which degrades performance on fine-grained or long-tail categories.

2.2 Leveraging 2D Priors for 3D Segmentation

Distilling knowledge from 2D foundation models, such as SAM [Kirillov et al., 2023; Ravi et al., 2024] and CLIP [Radford et al., 2021], has emerged as a dominant paradigm. By projecting rich semantic priors onto 3D points, numerous approaches have achieved impressive results in openvocabulary scene understanding [Chen et al., 2023a; Guo et al., 2024; Kim et al., 2024; Li et al., 2024; Liu et al., 2023; Peng et al., 2024; Xiao et al., 2024; Yin et al., 2024]. For instance, OpenScene [Peng et al., 2023] and PLA [Ding et al., 2023] demonstrate effective zero-shot transfer by aligning 3D features with 2D embeddings. However, their reliance on external human supervision (text or image labels) violates the strictly unsupervised premise. Consequently, recent research has pivoted towards distilling self-supervised features, notably from DINO [Caron et al., 2021] and DINOv2 [Oquab et al., 2023]. While providing valuable semantic initialization, direct distillation often overlooks the domain gap between dense images and sparse point clouds. Existing works like PointDC utilize these features but struggle to maintain geometric consistency. We argue that 2D priors are beneficial but must be complemented by robust 3D geometric regularization to align semantic features with the underlying manifold.

2.3 Clustering and Category Discovery

Traditional deep clustering methods, such as DeepCluster [Caron et al., 2018] and IIC [Ji et al., 2019], typically rely on K-Means with a pre-specified cluster count K. This rigidity remains a critical bottleneck in 3D segmentation. While recent contrastive approaches [Cho et al., 2021; Xie et al., 2020] enhance feature discriminability, they generally suffer from the same limitation due to the use of static prototypes. Attempts using over-clustering strategies often require manual upper bounds and struggle to converge to true semantic entities. In contrast, Non-parametric Bayesian methods [Ferguson, 1973], particularly the Dirichlet Process Mixture Model (DPMM), offer a theoretical framework for dynamic cluster inference. However, the efficient integration of DPMM into end-to-end deep learning for large-scale 3D semantic discovery remains an underexplored frontier.

2.4 DiffusionModels for Geometric Representation Learning

While diffusion models have achieved remarkable success in 3D shape generation and completion [Ho et al., 2020; Zhou et al., 2021], most approaches prioritize generation fidelity over feature discriminability. Conversely, recent 2D studies [Baranchuk et al., 2021] reveal that the intermediate representations within the denoising process encapsulate rich semantic information. Inspired by this, we repurpose diffusion from a generative tool to a self-supervised pretext task for geometric rectification. Instead of synthesizing new objects, we leverage the denoising objective to force the backbone to recover latent manifolds from sparse inputs. This strategy explicitly enhances the model's sensitivity to local geometric structures, effectively mitigating the geometric incompleteness inherent in raw point clouds.

We present DyG-Seg, a framework designed to address the challenges of unsupervised point cloud segmentation by seamlessly integrating geometry, structure, and semantics. The pipeline begins with a 2D-to-3D distillation module that transfers semantic knowledge from foundation models [Caron et al., 2021; Oquab et al., 2023; Kirillov et al., 2023] to initialize superpoint features [Chen et al., 2023b; Zhang et al., 2025]. To mitigate geometric sparsity, the Geometric Manifold Rectification (GMR) module serves as a structural prior, refining the latent manifold of the input data. Subsequently, the core Dynamic Prototype Contrastive Clustering (DPCC) module adaptively infers the optimal cluster number and learns discriminative prototypes. Finally, the segmentation network is optimized through dynamic class balancing and spatial consistency constraints to ensure robust predictions.

3.1 Cross-Modal Semantic Transfer

To leverage the rich semantic priors encapsulated in largescale vision foundation models, we implement a multiview 2D-to-3D distillation strategy. Following established paradigms [Chen et al., 2023b; Peng et al., 2023], this process distills pixel-level semantics to the 3D domain through a projection-then-regression pipeline.

Feature Projection. Given a point cloud P = {pi}Ni=1 and its associated multi-view images I = {Ik}Kk=1 , we first extract pixel-level features using a frozen 2D encoder Φ2D (e.g., DINOv2 [Oquab et al., 2023]). By leveraging camera parameters and depth maps, these 2D features are unprojected into 3D space. For points visible across multiple views, we aver- Contrastive Pull bedding f target i . Points lacking valid projections are excluded from the supervision signal.

3D Distillation. We employ a learnable 3D backbone Φ3D (e.g., SparseConv [Graham et al., 2018]), which takes the raw point cloud as input to regress these projected 2D features. The network predicts 3D features fi ∈ RC supervised by maximizing the cosine similarity with the target embeddings:

where Nv denotes the number of points with valid 2D projections.

Superpoint Partitioning. Upon convergence, the frozen distillation network Φ3D generates geometry-aware semantic features for the entire dataset. To enforce local consistency and reduce computational complexity for the subsequent stage, we partition the point cloud into disjoint superpoints S = {sj}Mj=1 using an unsupervised region growing algorithm based on geometric curvature and color similarity. Finally, we compute the unified embedding zj for each superpoint sj by average pooling the distilled features of its constituent points:

where Nj represents the index set of points in sj. The resulting matrix Z ∈ RM×C provides the initialized geometricsemantic representation for the subsequent stage.

Weig

3.2 Geometric Manifold Rectification

In unsupervised settings, sparse and noisy 3D acquisitions often degrade the extraction of consistent semantic features [Chen et al., 2021; Wang et al., 2021]. To address this, we introduce the Geometric Manifold Rectification (GMR) module. Unlike standard denoising approaches [Pang et al., 2023] that merely maximize the likelihood of observed coordinates, GMR operates as a structure-guided generative process. We formulate the rectification as learning a DensityRectified Vector Field, where the denoising trajectory is actively modulated by a repulsive topological prior to ensuring both geometric fidelity and structural uniformity.

Unified Density-Aware Generative Objective. We treat the geometric recovery not as two separate tasks, but as optimizing a unified target potential. Let pt be the noisy coordinates at timestep t, and Ψθ(pt, t) be the denoising network predicting the clean manifold geometry ˆp0. Standard diffusion models [Ho et al., 2020] approximate the score function ∇p log q(p) to map noise back to data. However, this solely captures spatial location, ignoring the density distribution on the manifold. To unify geometry and topology, we define a Target Potential E(ˆp0) that integrates geometric reconstruction with structural constraints:

where pGT denotes the underlying clean surface (approximated by the reconstruction target). The second term is derived from the Determinantal Point Process (DPP) [Kulesza and Taskar, 2012], where Lˆp0 is the Gaussian similarity kernel of the predicted points. Geometrically, minimizing this potential induces a Repulsive Gradient Field ∇ˆpLDPP that counteracts the tendency of diffusion models to collapse points into high-density modes.

Gradient-Guided Optimization. During training, we compel the network to minimize the expected energy of the rectified coordinates. The unified optimization objective is formulated as:

Remark: This formulation implicitly performs structural guidance during training. Through back-propagation, the gradients from the Repulsive Potential flow through the denoising network Ψθ, updating the backbone parameters. This compels the network to learn a rectified vector field that pushes points towards the manifold surface while simultaneously spreading them uniformly, effectively creating a "cleaner" geometric support for the subsequent segmentation task.

3.3 Dynamic Prototype Contrastive Clustering

Following the geometric rectification, the framework proceeds to semantic discovery. A critical limitation in existing unsupervised segmentation methods is the reliance on a pre-specified number of clusters K [Caron et al., 2018; Ji et al., 2019; Zhang et al., 2023]. In open-world scenarios, the semantic structure is typically unknown and varies across scenes. To overcome this, we propose Dynamic Prototype Contrastive Clustering (DPCC), a unified non-parametric framework that orchestrates semantic structure discovery and representation learning via an iterative Expectation-Maximization (EM) paradigm. DPCC treats the superpoint embeddings Z as observations and models the semantic discovery as a Bayesian inference process. We maintain a dynamic set of prototypes whose existence probability is governed by a Dirichlet Process, while their feature distribution is optimized to guide the backbone training.

E-Step: Bayesian Structural Inference. In the inference step, we formulate the discovery of semantic categories as a Variational Inference (VI) problem within a Dirichlet Process Mixture Model (DPMM) [Ferguson, 1973]. To model the potentially infinite number of clusters, we adopt the Stick-Breaking construction [Sethuraman, 1994]. Let V = {vk}Kmax k=1 be a set of latent variables drawn from a Beta distribution vk ∼ Beta(1, α), where α controls the sparsity. The existence probability πk for the k-th prototype is defined as:

To infer the optimal structure (pseudo-labels ˆy), we approximate the posterior by maximizing the Evidence Lower Bound (ELBO) [Blei and Jordan, 2006]:

This objective creates a dynamic competition: the Data Fitting Term encourages activating more prototypes to capture fine-grained details, while the Complexity Penalty Term imposes a sparsity constraint. Through this optimization, the model automatically eliminates redundant prototypes, converging to the optimal cluster number Kopt and generating high-quality pseudo-labels ˆy for the subsequent training phase.

M-Step: Discriminative RepresentationLearning. In the learning step, guided by the structurally inferred priors, we employ Prototype Contrastive Clustering [Caron et al., 2021; Xie et al., 2020] to shape the semantic space. We define the set of active prototypes as C = {ck}Kopt k=1 . These prototypes serve as the anchors for metric learning. For each superpoint feature zj , the assignment probability to the k-th prototype is modeled as a softmax distribution:

where τ is a temperature hyperparameter. The backbone network is then supervised using the generated pseudo-labels ˆ yj from the E-step. By minimizing the cross-entropy loss between the predicted distribution and the pseudo-labels, this contrastive objective enforces intra-class compactness and inter-class separability, ensuring that the dynamically discovered clusters are semantically distinct.

In summary, DPCC bridges structural inference and representation learning: the Bayesian prior determines the cardinality of the prototypes, while the contrastive objective governs the semantic content these prototypes represent, yielding Figure 2. Qualitative results on ScanNet dataset. Red circles highlight the differences.

robust pseudo-labels to supervise the iterative optimization in

3.4 Iterative Optimization with Dynamic Constraints

In the M-Step of our framework, the backbone network is updated using the pseudo-labels ˆy generated by DPCC. To address the inherent challenges of unsupervised point cloud segmentation-specifically long-tail class distributions and spatial discontinuity-we incorporate dynamic constraints into the optimization landscape.

Dynamic Class Balancing. Real-world 3D scenes exhibit severe class imbalance, where dominant categories (e.g., walls, floors) can overshadow rare objects during training. Directly supervising with raw pseudo-labels risks model collapse into trivial solutions [Caron et al., 2018; Ji et al., 2019]. To mitigate this, we implement an online frequency estimation mechanism (illustrated as "Dynamic Balancing" in Fig. 1). We maintain a global class counter v ∈ RKopt updated via Exponential Moving Average (EMA) at each iteration t:

where m is the momentum coefficient. We then derive an inverted re-weighting vector w to modulate the loss contribution of each class k:

This ensures that tail classes with lower frequencies receive larger gradients, effectively flattening the optimization landscape.

Spatial Geometric Consistency. To enforce the spatial smoothness of predictions, we construct a spatial affinity graph where edges E connect adjacent superpoints. Crucially, this adjacency is defined on the rectified manifold recovered by GMR (Sec. 3.2), ensuring that topological constraints are applied to geometrically valid neighbors rather than noise. We minimize the Kullback-Leibler (KL) divergence between the predictive distributions of neighboring superpoints:

Unified Objective and Iterative Refinement. The DyG-Seg framework unifies geometric recovery and semantic learning into a single optimization objective. The total loss Ltotal combines the segmentation-specific task loss with the geometric rectification objective defined in Eq. (4):

where Lce denotes the weighted cross-entropy loss against pseudo-labels. This joint optimization creates a positive feedback loop (depicted as "Feature Update" in Fig. 1): Lgeo provides a cleaner geometric support for feature extraction [Chen et al., 2021], which in turn improves the clustering quality in DPCC; simultaneously, Lseg injects semantic discriminability back into the backbone. Through this iterative self-training, the framework progressively aligns geometric structures with semantic categories, converging to robust segmentation performance.

4 Experiments

4.1 Experimental Setup

Datasets. To validate the effectiveness of our proposed framework, we conduct extensive experiments on two largescale public indoor 3D benchmarks. ScanNetV2 consists of 1,201 scenes for training and 312 scenes for validation, where points are annotated into 20 semantic categories [Dai et al., 2017]. S3DIS contains 271 rooms distributed across 6 areas [Armeni et al., 2017],where points are annotated into 13 semantic categories. Following standard protocols [Zhang et al., 2023; Chen et al., 2023b; Zhang et al., 2025], we evaluate our method using 6-fold cross-validation, reporting the average performance across all folds to ensure a comprehensive assessment.

Baselines. We compare our proposedmethod with comprehensive state-of-the-art unsupervised 3D semantic segmentation methods, including: 1) K-means: a classical clustering algorithm applied directly to per-point xyzrgb features. 2) IIC [Ji et al., 2019] and 3) PiCIE [Cho et al., 2021]:

methods adapted from the unsupervised 2D segmentation domain, modified to utilize the same 3D backbone as ours. 4) GrowSP [Zhang et al., 2023]: a pioneering work that employs a superpoint growing strategy for semantic segmentation. 5) PointDC [Chen et al., 2023b]: a feature distillation method utilizing the 2D pre-trained STEGO [Hamilton et al., 2022] model. For a fair comparison, we also implement 6) PointDC-DINOv2, which replaces the teacher model with DINOv2 [Oquab et al., 2023], consistent with our setting. 7) LogoSP [Zhang et al., 2025]: a recent framework that incorporates local-to-global graph learning for unsupervised segmentation.

EvaluationMetrics. The standard metrics, including mean Intersection-over-Union (mIoU), Overall Accuracy (OA), and mean Accuracy (mAcc), are reported across all classes.

4.2 Implementation Details

Implemented in PyTorch, our framework utilizes SparseConv [Graham et al., 2018] as the backbone, with initial priors distilled from pre-trained DINOv2 (ViT-S/14) [Oquab et al., 2023]. The training pipeline comprises initialization clustering followed by online self-training. The GMR module is trained end-to-end with 50 diffusion timesteps, incorporating a DPP regularization term (Gaussian kernel, σ = 0.05, weight 0.01) to maintain geometric uniformity. We employ the Adam optimizer with a learning rate of 10-3 and train for 200 epochs. For DPCC, the concentration parameter α is set to 0.4. Key hyperparameters include a dynamic class balancing momentum of 0.99, a smoothing term of 1.2, a spatial consistency loss weight of 0.15, a geometric loss weight of 0.8, and a contrastive temperature of 0.1.

4.3 Evaluation on ScanNet

Settings and Protocol. We report results on both the offline validation and online hidden test splits. Consistent with [Zhang et al., 2023; Chen et al., 2023b; Zhang et al., 2025], we employ Hungarian matching to align predicted labels for evaluation. The initial superpoint count is set to 40, with the final category count dynamically inferred (Table 3). Undefined points are excluded from loss computation.

Results and Analysis. As shown in Tables 1 and 2, DyGSeg significantly surpasses baselines on both splits. A key strength is its long-tail robustness; notably, bathtub performance approaches supervised levels. This improvement stems from our dynamic class balancing and diffusion-based manifold recovery, which effectively capture the geometric features of sparse objects. Qualitative results are provided in Figure 2 and the Appendix.

4.4 Evaluation on S3DIS

Settings and Protocol. Following standard practice [Zhang et al., 2023; Zhang et al., 2025], we exclude the clutter category during training and evaluation due to its semantic ambiguity. We adopt standard 6-fold cross-validation, maintaining hyperparameters consistent with the ScanNet experiments.

Results and Analysis. As reported in Tables 4 to 6, DyG-Seg consistently outperforms baselines. Crucially, our method achieves state-of-the-art mIoU performance. Its distinct advantage lies in its robustness towards long-tail distributions, securing the best results on tail categories. This confirms that our dynamic graph structure and manifold recovery provide substantial benefits beyond the quality of initial feature priors. Additional results are in the Appendix.

4.5 Ablation Study

To validate the contribution of each component, we conduct ablation studies on the ScanNet validation set, investigating the impact of Geometric Manifold Rectification (GMR), Dynamic Prototype Contrastive Clustering (DPCC), and Iterative Optimization. Results in Table 7 show that removing GMR significantly degrades performance, confirming the criticality of geometric manifold recovery for sparse 3D data. Furthermore, Figure 4 demonstrates that the model achieves optimal performance with 40 initial superpoints and 50 diffusion timesteps, effectively balancing semantic granularity the performance metric.

with geometric smoothing. Comprehensive ablation analysis is provided in the Appendix.

5 Conclusion

In this paper, we presented DyG-Seg, a novel unsupervised framework for 3D point cloud segmentation. By introducing a diffusion-based Geometric Manifold Rectification (GMR) mechanism, our method goes beyond simple denoising to actively recover latent topology from geometrically incomplete observations, providing a robust geometric foundation. We further proposed the Dynamic Prototype Contrastive Clustering (DPCC) and Iterative Optimization to enforce spatial consistency and feature distinctiveness. Extensive experiments on ScanNet and S3DIS benchmarks demonstrate that DyGSeg significantly outperforms existing state-of-the-art unsupervised methods.

Acknowledgements

This work was supported by the National Natural Science Foundation of China under Grant 62471266, 62006131, 62071260; the National Natural Science Foundation of Zhejiang Province under Grant LQ24F020001; and the Ningbo Major Research and Development Plan Project (No.2023Z224).