IE-PMMA:Point Cloud Completion Through Inverse Edge-aware Upsampling and Precise Multi-Modal Feature Alignment

1 Introduction

The development of scanner technology has led to the widespread use of 3D point clouds in various applications. However, the point clouds from scanners are often sparse and incomplete[Geiger et al., 2013] due to limited perspectives[Jia et al., 2024] and scanner resolution. This makes it challenging to use them directly in applications like reconstruction of archaeological[Jakliˇc et al., 2015], autonomous vehicle detection[Ma et al., 2022], virtual reality scan modeling[Kim et al., 2019], and other point-based downstream tasks[Tesema et al., 2023]. As a result, completing point clouds has become an increasingly important task in 3D computer vision.

Currently, mainstream point cloud completion methods are typically based on a two-stage framework. In the first stage, a coarse point cloud with a complete shape (referred to as seed points) is generated from the input partial points, known as the seed points generation step. In the second stage, based on the extracted point cloud features, the seed points are upsampled to the target resolution in order to generate a complete and dense point cloud. This stage is referred to as the seed points upsampling step.

With the widespread application of multi-modal ideas, 2D images have become common as supplementary data in point cloud completion tasks. ViPC[Zhang et al., 2021] is the pioneer in the use of 2D data for completion tasks, taking the missing crucial global structure information from an extra single-view image. CSDN[Zhu et al., 2023c] estimates the global and local geometric information of point clouds from 2D images through a dual-refinement network. Nevertheless, these methods usually use projection relations or attention mechanisms to fuse multimodal data. Such fusion methods can be limited and imprecise, not fully taking advantage of the correlation between 2D and 3D data.

Therefore, inspired by the feature fusion method in BEV[Chen et al., 2022] tasks, we propose a two-stage precise multi-modal data alignment method, which is divided into two steps: local feature alignment and global feature fusion. Firstly, the projection depth maps under different viewpoints are obtained through the partial points, and the 2D image features are extracted. At the local scale, the image features corresponding to each point in the point cloud are obtained through projection relations and bilinear interpolation. Then, the scale factor adjusts the feature's magnitude to get accurate 2D features. The 3D features are extracted directly via PointNet++[Qi et al., 2017b]. At the global scale, we continue to use the attention mechanism-based fusion method in previous work to further fuse features.

In addition, we notice that in many works related to point cloud, edge information has been introduced as a way to augment metadata, such as point cloud generation[Liu et al., 2022], classification[Wu et al., 2023], and other tasks. Many works suggest that the edge points have rich geometric details, which can provide more information about the local structure of the point cloud. Therefore, we hope to introduce the idea of edge-aware to the point cloud completion task.

To achieve edge-aware completion, we first design a new spatial representation method for extracting edge points called Edge Probability Volume (EPV). This method divides the space containing the point cloud into grids. Each voxel contains two values: the coordinates of the point within that voxel and the probability of the voxel being an edge voxel.

We applied the edge points extracted by EPV during the seed points upsampling process. Calculate the chamfer distance between edge points and seed points, as well as between input points and seed points, to represent edge importance and missing importance, respectively. The selfattention module is employed to achieve edge-aware upsampling. However, in practice, we discover that inverse edgeaware upsampling is more effective. Specifically, the closer a point is to the edge points, the less important it is as a seed point. By utilizing the cross-attention operation between the extracted aware features and the input partial point features, we obtain the displacement of point cloud coordinates through two decoders and MLP layers.

We experiment with our method on three common datasets: PCN, ShapeNet34/55, and KITTI. The results show that our method achieves SOTA in metrics of chamfer distance, density-aware chamfer distance, and F1 score. Our main contributions can be summarized as:

1. We propose a local-to-global feature extraction module based on precise multi-modal data alignment. This module fully integrates 2D and 3D features and effectively leverages the information contained in the depth map.
2. We introduce an edge point extraction method that can extract a fixed number of edge points, effectively reducing the impact of point cloud noise and adapting to the point cloud completion task.
3. We apply edge point information to the point cloud completion task for the first time. We utilize edge information during the upsampling process of seed points to achieve inverse edge-aware upsampling.

2 RelatedWork

2.1 Two-Stage Point Cloud Completion

The existing point cloud completion methods based on deep learning are generally two-stage completion frameworks. Firstly, a complete seed point cloud is generated from the input partial point cloud, and then the final completion result is obtained through the upsampling process. PCN[Yuan et al., 2018] is an outstanding contribution to the two-stage point cloud completion framework, which uses the PointNet[Qi et al., 2017a] layer to extract global features and generate seed points with MLPs, then uses Folding-based[Yang et al., 2018] upsampling block to upsample seed points. PoinTr[Yu et al., 2021] uses a lightweight DGCNN[Wang et al., 2019] to extract point cloud features, a geometry-aware Transformer to predict missing point proxies, and upsamples seed points using Folding block as in PCN. Snowflakenet[Xiang et al., 2021] uses PointNet++[Qi et al., 2017b] to extract features and generate seed points with point-wise splitting operations and MLPs. SeedFormer[Zhou et al., 2022], AnchorFormer[Chen et al., 2023], and ProxyFormer[Li et al., 2023] design unique point cloud representations to improve the ability to extract local geometric information around seed points. ODGNet[Cai et al., 2024] designed a shape prior dictionary-guided seed point generation U-Net, and used the Upsampling Transformer in SeedFormer for seed points upsampling. SVDFormer[Zhu et al., 2023b] applies the depth projection of the partial points to the point cloud completion and uses the two-dimensional depth map to complete. On this basis, GeoFormer[Yu et al., 2024] applies the rich features contained in the three-channel canonical coordinate maps to improve the multi-modal point cloud completion. TCorresNet[Duan et al., 2025] generates point proxies with the help of the Gaussian spherical template and upsamples them with the Folding block.

These methods can effectively complete the missing parts of the point cloud, but the current completion works have relatively few studies that incorporate multi-modal data, especially those that focus on further aligning data from different modalities. Furthermore, current research lacks exploration of the usage of the edge points.

2.2 Point Cloud Edge Extraction

The method of point cloud edge extraction can be divided into geometry-based methods and learning-based methods. Geometry-based methods are mainly based on local geometric features, such as the eigenstructure of the covariance matrix[Gumhold et al., 2001], normals[Weber et al., 2010], curvatures[Hackel et al., 2016], or other statistical metrics[Zhu et al., 2023a]. There are also some point cloud edge feature extraction methods based on graph Fourier transform[Chen et al., 2017; Deng et al., 2022]. These geometry-based methods are able to extract approximate edge points, but they are less accurate.

With the development of deep learning, learning-based point cloud edge extraction methods have emerged. The earliest methods are based on point cloud classification. PIENet[Wang et al., 2020], EDC-Net[Bazazian and Par´es, 2021], and PCEDNet[Himeur et al., 2021] classify point clouds into edge points, non-edge points, and corner points based on the sharpness labels of points in the dataset to extract edge points and parametric lines. EC-Net[Yu et al., 2018] and DEF[Matveev et al., 2022] regress the distance between the sampled points and the feature line to extract the identified edge points. These methods are explicit methods that are easily affected by point cloud noise[Zhu et al., 2023a]. NerVE[Zhu et al., 2023a] divides the point cloud space into grids and binary classifies whether each voxel is an edge voxel or not. NEF[Ye et al., 2023] used the image of the object model to extract edge information and designed an edge neural radiance field. PBWR[Huang et al., 2024] extracts the edge of LiDAR point clouds through a transformer to obtain edge regression results.

These methods can extract edge points better than geometry-based methods, but the number of edge points is not fixed, making them unsuitable as a data augmentation scheme for downstream point cloud tasks.

3 Method

3.1 Overview

We follow the commonly used two-stage point cloud completion paradigm, first generating the seed points with a complete shape from the input partial points and then upsampling the seed points to the target resolution, as shown in Figure 1. Specifically, the global feature is extracted from input partial points through the Precise Multi-Modal Alignment feature extractor (called PMMA module). The 3D coordinates of the seed points are regressed by a group of MLP layers. In order to take advantage of the information on the edge points, we first obtain the edge seed points through the pre-trained Edge Probability Volume (called EPV module). The partial points, seed points, edge seed points, and global feature are taken as input of the Inverse Edge-aware upsampler (called IEup module), for the upsampling of seed points. In line with other coarse-to-fine upsampling methods, our IEup module is also performed twice to achieve the coarse-to-fine upsampling of seed points.

3.2 Seed Points GenerationWith Precise Multi-Modal Feature Alignment

The first stage of the two-stage completion framework is the generation of seed points with a completed shape. Currently, multi-modal data is used in many point cloud completion works. Among them, the works represented by [Zhu et al., 2023b] project the partial point cloud, and the obtained depth maps are used as the image modal data. The alignment between the 3D point cloud features and the 2D image features is then achieved through the cross-attention mechanism. This approach only aligns features on a global scale and can't achieve the local accurate point-to-pixel feature alignment. Inspired by AutoAlignv2[Chen et al., 2022], we design a local-to-global feature fusion module, which enables precise multi-modal feature alignment.

At the local scale, as show in Figure 2, firstly, the input partial point cloud is projected according to the coordinates of viewpoints as the camera position to obtain the depth map Di in three directions, i indicates the number of viewpoints. Afterward, use 2D Backbone to extract features from the depth map and obtain a low-resolution coarse feature map Ci. The 2D Backbone here is a simplified ResNet18[He et al., 2016]. Then, the low-resolution basic feature map is upsampled to the fine feature map Fi through bilinear interpolation, which is consistent with the resolution of the depth map Di. At the same time, the point cloud is projected according to three viewpoints to calculate the sub-pixel coordinates corresponding to each point. The image feature P ji corresponding to each point is extracted on the fine feature map through bilinear interpolation, while j represents the index of the point.

The extracted features are scaled through scale factor γ, and the processed features are fed into the max pooling layer for aggregation and obtaining the image features Ai that are accurately aligned with the coordinates of the 3D points. The formula of this process can be expressed as:

where N represents the number of points. Then cascading Ai from all viewpoints to get the final 2D feature. The corresponding 3D features are directly extracted through PointNet++[Qi et al., 2017b]. Finally, these two different dimensional features and the viewpoints are inputted into the Global Fusion Module to get the global feature G.

At the global scale, as shown in Figure 3, we use the attention mechanism[Zhu et al., 2023b] to achieve the re-fusion of multi-view 2D image features and 3D point cloud features. First, the 3D features are duplicated and concatenated with the 2D features. The view features are added to the 2D & 3D concatenation features to serve as the Key and Query of the multi-head self-attention, while the concatenation features as the Value. The alignment features obtained by the max pooling layer are then concatenated with the 3D features to obtain the global feature G, which is fed into MLP layers to regress the coordinates of seed points.

3.3 Edge Points Extraction Based on Edge Probability Volume

In the fields of point cloud generation [Liu et al., 2022], classification, retrieval [Wu et al., 2023; Wen et al., 2023], and reconstruction [Wen et al., 2023], the edge information of the point cloud can enhance the metadata. This is due to the fact that the edge points contain a wealth of sharp details. It can be inferred that the edge information of the point cloud has a positive effect on a variety of point cloud tasks. Therefore, we hope to use the edge information of the point cloud as a new exploration in the point cloud completion task.

To adapt to the point cloud completion task, we propose a new representation called Edge Probability Volume (EPV) to predict and rank the probability of whether one point is an edge point. This section mainly introduces the definition of EPV and the method of extracting edge points.

To solve the problem that the classification-based point cloud edge extraction methods [Wang et al., 2020; Bazazian and Par´es, 2021; Himeur et al., 2021] are susceptible to noise, NerVE [Zhu et al., 2023a] follows the voxel representation of a 3D shape [Choy et al., 2016] to discretize continuous 3D curves into a volumetric grid. On this basis, we improve the binary classification of edge points to probabilistic regression, which realizes the orderliness of point cloud probability so that a fixed number of edge points can be obtained.

As shown in Figure 4, a simplified PointNet++[Qi et al., 2017b] module and a 3D CNN module are used to obtain the feature grid of the point cloud, which has the same resolution as EPV output. After that, two individual decoders are used to predict the attributes of each voxel. In the first step, one MLP regresses the probability pi,j,k of whether a voxel is an edge voxel. In the second step, the other MLP calculates the coordinates xi,j,k of the point in each voxel. The process can be expressed as:

where Ei,j,k is the voxel feature and ξ is the threshold. Only voxels with a probability greater than the threshold will proceed to the coordinate prediction step.

Since the ABC dataset[Koch et al., 2019] we used for training only labels the sharpness of each point, without the probability of whether a point is an edge point. Therefore, we have to define the calculation method of the probability to get the ground truth of the probability.

According to the first law of geography, the closer a point cloud is to an edge point, the higher the probability that it is an edge point. As the distance from the edge point increases, the probability of the point cloud being an edge point decreases. Therefore, for each voxel vi,j,k, first retrieve the nearest edge voxel ea,b,c to it, then define the distance between them as:

In order to convert the distance into a probability in the range of (0, 1], we can use the function f(x) = e-x . In this way, the converted distance f(di,j,k) is the ground truth of probability pgt i,j,k . In the training process, we chose MSE and L1 loss as the loss function to train the probability and coordinates prediction model, respectively , which can be expressed as:

where N represents the resolution of the EPV in one direction, Lprob represents the probability prediction loss, and Lpos represents the coordinate prediction loss.

With the pre-trained EPV model, we can predict the probability of whether a voxel is an edge voxel. After sorting the probability values, we can extract the first M voxels with the highest probability value. The points in those voxels are taken as the edge points of the point cloud. M can be modified according to the requirements of the point cloud completion task. In this paper, we choose 32 × 32 × 32 as the resolution to pre-train the EPV model.

3.4 Seed Points Upsampling Based on Inverse Edge-Aware Upsampler

In the previously mentioned point cloud tasks, the premise of introducing point cloud edge information to enhance the point cloud data is the integrity of the point cloud. The edges of the point cloud are meaningful only if the point cloud is complete. In the two-stage point cloud completion method, the input of the seed point generation stage is an incomplete point cloud, and the input of the upsampling stage is the seed points with a relatively complete overall shape. Therefore, we chose to introduce point cloud edge information in the upsampling process, as shown in Figure 5.

Specifically, for the generated seed points, we first apply the EPV described in 3.3 to extract the edge points. The first step of upsampling is the Missing & Inverse Edge-aware Self-Attention Module. We define the missing importance as the chamfer distance between the seed points and the partial points, and the greater the chamfer distance from the input point cloud, the greater the missing importance. Similarly, we define the inverse edge importance as the chamfer distance between the seed points and the edge points, and the greater the chamfer distance from the edge points, the greater the inverse edge importance.

It is speculated that point clouds near edge points should be given greater importance based on their impact on other applications. However, our practical findings indicate that using inverse edge importance can actually enhance point cloud completion performance. In other words, the farther a point cloud is from the edge point, the greater importance it is assigned. The missing importance and inverse edge importance can be expressed as:

where the ps represents the point in the set of seed points Pseed, pin represents the point in the set of input points Pinput, pe represents the point in the set of edge points Pedge. MI represents the Missing Importance of a point, while IEI represents the Inverse Edge Importance. The Aware Importance can be expressed as:

where µ is a scaling coefficient set as 0.2, Sin is the sinusoidal function[Vaswani, 2017]. The aware feature is obtained through multi-head attention layers, which can be expressed as:

The aware feature FA and partial feature Fin, which is extracted from the input point cloud, are fed in the crossattention layer to obtain the local feature FL. FA and FL are fed in two decoders respectively to obtain the upsampled features F ′A and F ′L. The coarse feature Fl of the l-th upsampling process is obtained by cascading F ′A and F ′L. The coarse features are fed into the MLP layers to obtain the displacement of point cloud coordinates at the l-th step. Such a process will be performed twice to achieve the coarse-to-fine upsampling.

3.5 Loss Function

As with other methods, we use the chamfer distance as the loss function. CD loss is defined as follows:

where O is the predicted point cloud with N1 points. GT is the ground truth point cloud with N2 points. As in the previous work, we also use CD-L1 and CD-L2 as the loss functions for different datasets, respectively, where CD-L2 is the same as CD, and CD-L1 takes the square root of L2-Norm and divides it by 2. The total loss is defined as:

The FPS represents the farthest point sampling function, the first parameter represents the point cloud to be sampled, and the second parameter represents the number of sampling points. Pseed represents the seed point cloud, Pgt represents the ground truth, and Pl=1 and Pl=2 represent the corresponding point clouds obtained through two upsampling stages.

4 Experiments

4.1 Dataset and EvaluationMetrics

Dataset. In order to verify the point cloud completion effect of our IE-PMMA in different scenarios, we tested it on common datasets PCN, ShapeNet34/55 and KITTI and evaluated them with a number of evaluation metrics. PCN[Yuan et al., 2018] dataset is a subset of the ShapeNet[Chang et al., 2015] dataset, which contains 8 types of objects and 30,974 point clouds. ShapeNet34/55 is proposed by PointTr[Yu et al., 2021]. The ShpeNet55 dataset contains 55 categories of point cloud data, and the ShapeNet34 dataset contains 34 visible category point clouds and 21 invisible category point clouds. We further tested our results on the KITTI[Geiger et al., 2013] dataset. Following the previous method, we only focus on the points in the bounding box of the object marked as a car, with a total of 2483 incomplete point clouds and no ground truth.

Metrics. In order to comprehensively evaluate our method, we used Chamfer Distance (CD), Density Chamfer Distance(DCD), and F1 Score as evaluation metrics on the PCN and ShapeNet34/55 datasets. For the KITTI dataset, we used Fidelity and Minimal Matching Distance (MMD) as the metrics since KITTI is without ground truth.

4.2 ComparisonWith SOTAMethods

Evaluation on PCN dataset. We conduct experiments on the PCN dataset and compare them with 8 recent point cloud completion methods as shown in Table 1. The results show that IE-PMMA exhibits SOTA results on all metrics. Especially on the DCD, our method is 1 × 10-2 higher than ODGNet. Figure 6 shows the visual comparison results of our method and other methods on the PCN dataset. Our IEPMMA works best on table legs, aircraft engines, chair backs, and car tails. This is due to the fact that the IEup module makes the model pay more attention to the global structure. At the same time, the PMMA module promotes the generation of the local detailed structure.

Evaluation on ShapeNet34/55 dataset. To investigate the generalization of the method, we test IE-PMMA on ShapeNet34/55. The first is the ShapeNet55 dataset with 55 categories of objects, as shown in Table 2. Our method outperforms the SOTA methods on three representative metrics. As for the ShapeNet34-21 dataset, we surpass the SOTA approach on CD metrics as shown in Table 3. Figure 7 shows the visualization of our method on the Shapenet55 dataset. In addition to visualizing the point cloud results, we also used the ball pivoting algorithm to reconstruct the surface to reflect the representation power of the point cloud. Compared with the SOTA methods, our method is able to accurately recover the local structure of the point cloud, such as the armrests of the chair and the detailed wings of the aircraft. From the results of surface reconstruction, the point clouds completed by our method also have a stronger geometric representation ability.

Evaluation on KITTI dataset. To verify the effectiveness of our method on the real-world scanned dataset, we conduct experiments on the KITTI dataset. We train on the car category of the PCN dataset and recover the full shape from the scanned incomplete vehicle point cloud. Our method surpasses the SOTA methods on the Fidelity metric, as shown in Table 4. Figure 8 shows that our point clouds have more complete shapes and less noise.

4.3 Ablation Experiments

Ablation on IEup Module. In order to verify the role of the Inverse Edge-aware upsampler in the point cloud completion process, we perform ablation experiments corresponding to model A and model B in Table 5. After adding the IEup module, all of the metrics have been significantly improved. In order to show the execution of the IEup module more concretely, we visualize the importance of the point cloud during the upsampling process and the edge points extracted by EPV in Figure 9. The orange points are the edge points, while magenta indicates the greater importance of point clouds, and cyan is the opposite. It can be found that the Inverse Edgeaware upsampler gives more importance to the point cloud far away from the sharp edge of the point cloud, which makes the upsampling process pay more attention to the global shape of the object, rather than over-fitting the local details during the training process. In addition, the EPV spatial representation module can also extract the edge points effectively.

Ablation on PMMA Module. The Model B, C, and D in Table 5 verify the effectiveness of the PMMA module. Model C only adds a point-to-pixel feature projection module to Model B, which leads to worse results. This is because Model B directly extracts a set of image features from ResNet, while the image features in Model C are composed of N sets of features. The magnitude of the image features in Model C is much larger than that of Model B, resulting in the features not being able to function normally in the subsequent steps. For Model D, the scaling factor is added, which can shrink the groups of features extracted from Model C and restore the feature vectors to the normal magnitude. In Figure 10, we visualize the density of the point clouds generated by the four sets of models. Compared with Model D, Model C has a higher density of complete point clouds around the input points, such as the edge of the hull and the back of the chair, which indicates that unscaled features will affect the distribution of global features and lead to an imbalanced density of point clouds.

4.4 Model Complexity and Resource Costs

The detailed complexity and resource cost analyses are shown in Table 6. While the two-step alignment does incur additional computational costs and the EVP module requires more parameters, these design choices were carefully considered to achieve the demonstrated performance improvements, which reduce CD by 0.06 × 10-3 over the baseline.

5 Conclusion and Discussion

In this article, we propose IE-PMMA, which is the first point cloud completion method that takes into account the edge information of the point cloud. The PMMA module achieves precise point-to-pixel alignment, significantly improving upon previous coarse feature matching methods. The novel inverse weighting approach in the IEup module prioritizes global shape coherence to prevent local detail overfitting. The EPV module's probabilistic volume representation offers enhanced noise robustness and broader applicability. Various experiments show that IE-PMMA achieves the most advanced performance in the point cloud completion task.

Acknowledgments

This work was supported by the Sichuan Science and Technology Program under Grant 2025YFHZ0034. The authors would like to thank Zi Guo from the Nuclear Power Institute of China for the helpful and valuable discussions.