Invertible Residual Neural Networks with Conditional Injector and Interpolator for Point Cloud Upsampling

1 Introduction

In recent years, with the development of depth data sensors, point clouds have become one of the most popular data types representing the 3D world. However, the collected point cloud data is usually sparse and irregular due to the complexity of real-world illumination and materials. Lowquality point clouds have a direct impact on the performance of downstream tasks, e.g., self-driving cars, robotics, surface reconstruction, and medical analysis, etc. Therefore, upsampling sparse and irregular point clouds into dense and uniform data has attracted increasing attention in computer vision and computer graphics.

Currently, existing upsampling methods, e.g., PU-Net [Yu et al., 2018], PU-GAN [Li et al., 2019], PU-GCN [Qian et al., 2021a], DIS-PU [Li et al., 2021] and PC2-PU [Long et al., 2022], usually perform feature extraction to encode the feature of the point cloud into a latent code. Then, these methods duplicate the latent code with some copies to increase the number of points, and finally reconstruct the 3D coordinates of points in Euclidean space by decoding from the duplicated latent code. Specifically, the coordinate reconstruction is constrained by evaluation metrics, such as uniformity and smoothness. The feature extraction and the coordinate reconstruction need to learn the parameters separately, and guarantee the relevance between the encoding and decoding processes. Different from these methods, a normalizing flow network architecture called PU-Flow [Mao et al., 2022] has been proposed for point cloud upsampling recently. The normalizing flow network parameterizes a bijective mapping between a complex distribution and a simple distribution. Thanks to the invertibility, PU-Flow achieves no information loss in the encoding and decoding processes. However, this method relies on designing structured Jacobians and specialized architectures, which have a significant impact on the model performance [Dinh et al., 2016; Miyato et al., 2018].

To achieve unconstrained Jacobians and architectures, in this paper we construct an invertible network with residual blocks by introducing lipschitz constraints [Chen et al., 2019]. To further improve the expressiveness of the network, we design a conditional injector, which includes a feature extractor module and a fusion module. The feature extractor module performs adaptive encoding of global and local features of sparse point clouds. The feature fusion module uses the attention mechanism [Vaswani et al., 2017] to aggregate global features with local features and outputs enhanced features. The enhanced features are injected into the main architecture, to enhance the nonlinear variability.

For upsampling, we also design a lightweight interpolator to scale the points, based on semantic similarity distance in the latent space modeled by the invertible network. The interpolation in the latent space can be reflected in the Euclidean space through the reverse mapping of the invertible network. Furthermore, we selected 25 complex models from Sketchfab, called the PU25 dataset, to quantitatively evaluate the upsampling point set. We compare our results with state-of-theart (SOTA) optimization and learning-based methods. Benefiting from our proposed invertible residual network with conditional injector and interpolator, our upsampling results have better uniformity and are closer to the underlying surface, and can better preserve the fine-grained information of the object.

The main contributions of this paper can be summarized as follows:

• We propose an easily embedded point cloud upsampling algorithm, called PU-INN, based on an invertible residual neural network, which has an unconstrained and expressive architecture by enforcing a Lipschitz constraint.
• We propose a conditional injector that includes a feature extractor and a feature fusion module, to enhance the feature transformation capability of the invertible neural network. The feature extractor module extracts fine local and coarse-grained global features, and the fusion module is designed to control the flow of information from local semantic features to global features based on an efficient channel attention.
• We propose a lightweight interpolator based on the semantic similarity distance. Considering that PU-INN learns a bidirectional mapping between the point cloud and the latent space, the interpolation in the latent space can immediately reflect the changes in Euclidean space.
• Experimental results show that PU-INN achieves better performance in qualitative and quantitative evaluations compared with SOTA methods.

2 RelatedWorks

Optimization-Based Upsampling. Traditional methods have tried various optimization strategies to generate upsampled point clouds. Early point upsampling methods usually adopted shape priors [Alexa et al., 2003; Lipman et al., 2007], and depended on the hand-crafted prior quality. Moreover, some methods relied on local geometric fitting, such as normal estimation [Huang et al., 2009] and smooth surface assumptions [Huang et al., 2013; Wu et al., 2015]. In general, the above metioned methods are not data-driven, thus leading to insufficient assumptions or requiring geometric attributes.

Learning-Based Upsampling. Deep learning methods introduce promising advances to optimization-based algorithms, owing to their data-driven characteristics and neural network learning capabilities. Deep neural networks [Qi et al., 2017a; Qi et al., 2017b; Wang et al., 2019; Thomas et al., 2019; Li et al., 2018] can learn features directly from point clouds. PU-Net [Yu et al., 2018], as a pioneer, proposed the first deep neural network for point cloud upsampling. MPU [Yifan et al., 2019] and PU-GAN [Li et al., 2019] proposed a patch-based progressive upsampling method and a GANbased framework separately. More recently, PU-GEO [Qian et al., 2020] considered the local geometric metric to upsample the point cloud. PU-GCN [Qian et al., 2021a] improved the performance of point cloud upsampling by using a graphbased module. DIS-PU [Li et al., 2021] applied a disentangled refinement framework to adjust the coarse upsampled points into fine scale points. Flexible-PU [Qian et al., 2021b] formulated the point cloud upsampling problem in an explicit manner by using the linear approximation theorem. PC2-PU [Long et al., 2022] paid more attention to patch correlation and position correction.

The above mentioned methods extract latent features by an encoding process and reconstruct coordinates by a decoding process. On the contrary, in this paper we proposed a method based on normalizing flows to unify the encoding and decoding processes.

Normalizing Flow in Point Cloud. Some inverible neural networks based on normalizing flows have been proposed to achieve efficient inverse computation by adopting the dimensional decomposition architecture, such as NICE [Dinh et al., 2014], Real-NVP [Dinh et al., 2016], i-RevNet [Jacobsen et al., 2018] and Glow [Kingma and Dhariwal, 2018]. However, these methods suffered from restricted transformations with sparse or structured Jacobians, thus leading to weak nonlinear transformation capability. Transformations that scale to high-dimensional data relied on specialized architectures. Recently, an invertible residual network [Behrmann et al., 2019] has been proposed based on invertible residual blocks, which have forward and backward Lipschitz boundaries, to avoid strict architectural constraints. To reduce the complexity and boost the adaptability, the invertible residual network [Behrmann et al., 2019] has been further improved by generalizing the Lipschitz constraint to induced mixed norms [Chen et al., 2019].

PointFlow [Yang et al., 2019] applied continuous normalizing flows [Chen et al., 2018] to 3D point clouds, but continuous flows caused slow convergence. Both DPF-Net [Klokov et al., 2020] and PU-Flow [Mao et al., 2022] used affine coupling layers [Dinh et al., 2016] to model discrete normalizing flow. These networks had a strong inductive bias that may hinder their application in supervised tasks. To improve the transformation capability, our PU-INN is inspired by an invertible residual network [Chen et al., 2019] and proposed with free-form Jacobian. Moreover, we impose a flexible conditional injector and a uniformity loss to further enhance the performance of the invertible residual network. Furthermore, the upsampling is performed with a point interpolator based on semantic similarity distance.

3 Method

Given a 3D point cloud X = {xi ∈ R3 }Ni=1 with N points and a user-specified upsampling factor r, we aim to generate a dense and homogeneous point cloud set P = {pi ∈ R3 }r×N i=1 with r × N points.

In this study, we put X into the injector, which fuses the local and global features extracted from the input, to get the conditional information C. Then, we feed X and C together into an invertible residual neural network, to get the highdimensional output in the latent Riemannian space. Finally, we interpolate in this latent space to get ˜ Z, which is transformed to the point cloud P in the Euclidean space by the inverse mapping of the invertible neural network.

3.1 Invertible Residual Neural Network

Our invertible neural network is constructed with some invertible residual layers. An invertible residual layer maps the original point cloud distribution Dx to the specified simple distribution Dz through a transformation function F, which consists of M invertible residual blocks,

where fθl (hl ) = hl +gθl (hl ), and gθl (hl ) computes the transformation of the input features hl at the l-th invertible residual block fθl . x is the original point cloud coordinates, and z obeys the specified simple distribution (like Gaussian distribution).

To ensure that each residual block is invertible, we enforce a Lipschitz constraint [Behrmann et al., 2019] as follows:

where the Lip(gθl ) is the Lipschitz norm of the function gθl . According to Banach's fixed-point theorem, the fixed point of the inverse function hl = hl+1 - gθl (hl ) is calculated by iterating a certain number of steps to converge to hl , which satisfies the accuracy requirement.

The training objective of the invertible neural network is as follows:

where log px(x) = log pz(z) + log |det JF )|. px(x) and pz(z) are the probability distribution of Dx and Dz, respectively. log |det JF | is the Jacobian matrix log-determinant of the transformation function.

3.2 Conditional Injector

We note that the constraint - the dimensionality of the output must be the same as the input in each invertible residual block (in Section 3.1) - limits the capacity of nonlinear transformation. Thus, we propose a conditional injector for injecting the necessary feature information into the original network to enhance its transformation capability. Then Equation (1) becomes:

where C0 is the output of the injector INJ(h0 ), which encodes h0 as a conditional information, Cv+1 = INJ(Cv)(v > 0). The injector consists of a feature extractor and a feature fusion module.

Point Feature Extractor

As shown in Figure 2, we propose a local and a global feature extractor to extract the fine and coarse-grained features of the point cloud, respectively.

Local Feature. To avoid the overall transformation from affecting the local information, we obtain relative point positions by subtracting the center point from the the k-nearest neighbor points. Then, we explicitly encode the relative point positions to enhance the neighboring point feature. We extract local features as follows:

where the T represents the T-Net [Qi et al., 2017a] which solves the rotation invariance problem, and ⊕ is the concatenation operation. ||xi -xki || computes the euclidean distance between the neighboring points xki and the centroid point xi, where xki is the k-th nearst neighbor of xi. The detailed architecture of MLP (Multi Layer Perceptron) can be found in Appendix A. Finally, we use the attentive pooling [Hu et al., 2020] instead of the traditional maximum pooling to preserve details during upsampling.

Global Feature. We propose the global feature extractor based on dense connections [Liu et al., 2019a]. Specifically, each dense connection learns from its previous layer and the raw input (on the same local region) simultaneously. Without explicit local information encoding, this global extractor can focus on the shape of point clouds, contour and other global information. We extract densely contextual representation HG as:

This extractor can use the dense connectivity pattern to repeatedly aggregate multi-level and multi-scale features. Instead of using the attentive pooling in the local feature extractor, a normal max pooling is used to extract global features.

Point Feature Fusion

We develop the point feature fusion module to incorporate local information HL with global features HG for aggregation. The fusion module also controls how much information from the global features should be merged with the local features based on the efficient channel attention [Vaswani et al., 2017]. The fused feature map is computed as follows:

where Norm(·) denotes a sigmoid function. Q = op(σ(HL)) ∈ RN×1 denotes queries of local features, and op(·) is a channel average operation. K = WKσ(HG) ∈ RN×Cg denotes keys of global features, and σ denotes a Softmax operation. V = WVHG ∈ RN×Cg denotes values of global features. WK,WV ∈ RN×N are linear transformation parameters. HC denotes output fusion features.

3.3 Point Interpolator

We design an interpolator for points expansion based on the semantic similarity distance. Unlike images with fixed topological relations, it is impossible to apply the continuous weight function [Liu et al., 2019b; Wu et al., 2019] to directly interpolate the feature map.

For each point, we perform interpolation in its k-nearest neighbors to generate new points, and the distance is calculated by similarity Si,j. Specifically, we set the cosine similarity score between the features of centroid and candidate points as a measure of the semantic similarity in the latent space. Given the fusion latent code z ∈ RNC=3 , which is the last residual layer's NC-dimensional output. The similarity Si,j of points zi and zj is defined as follows:

where ∥ · ∥2 is the L2 Normalization. Cosine distance can force the network to consider the spatial relationships between points, as two neighboring points naturally produce similar features and generate high cosine similarity.

By obtaining the neighborhood of each point, we can interpolate in it and learn the weight vector using a learning-based approach with the following formula:

where zi,u is the u-th nearest neighbor to zi, and ψ(zi, zi,u) ∈ RNC is the embedding in the feature space, which encodes the local structure and global information. ˜zi,l is the l-th interpolated feature from the original feature zi, l = 1, 2, · · · ,R (upsampling ratio). k is the number of nearest neighbors. The interpolation in the latent space can be reflected in the Euclidean space through the reverse mapping of the invertible neural network.

3.4 Loss Function

Prior Loss. Equation (3) allows us to train the flow layer by minimizing the negative log-likelihood by entering X :

where C = INJ(X). Optimizing the prior likelihood of X encourages encoding the shape representation to obtain high probability under a predefined prior pz(z), which is modeled by the flowmodule Fθ. In our experiments, the prior pz(z) is simply set to the standard Gaussian distribution N(0, I).

Similarity Loss. We use the Earth Mover's distance (EMD)to measure the dissimilarity between two multidimensional distributions in a given feature space, and EMD is calculated as follows:

where ϕ : P → X is the bijective mapping. EMD is fast, differential and robust to outliers[Fan et al., 2017].

Uniform Loss. We also take the uniformity of the point cloud into consideration. To evaluate the uniformity, we make each point to be as far as possible from the rest of the points in the same point set as follows:

where k denotes the size of the considered neighborhood.

4 Experiments

Dataset. The data we use for training and testing comes from two datasets, PU1K [Qian et al., 2021a] and Sketchfab

[Sketchfab, 2011]. Both datasets include point clouds of different resolutions and their surface models for evaluating the reconstruction metrics. Some mesh models have complex geometry and high-frequency details. To fairly compare different methods, we perform the same enhancement operations on the point cloud data, including random scaling, rotation and point perturbation.

Implementation Details. For each training object, we randomly crop it into M = 100 overlayable patches, resulting in 102000 training surface patches. On each surface patch, we extract rN points as the groundtruth Q by Poisson-disk sampling. Then, we randomly sample N = 256 points from Q as training input X .

We implement our network using the PyTorch platform and train 200 epochs using the Adam algorithm with an initial learning rate ϕ = 1e-4 (decayed by 0.7 every 40 epochs) by NVIDIA 3,090 GPU. The batch size is 16, and we empirically set the values of α, β, and γ in the compound loss function to 1000, 0.01, and 0.01, respectively. PU-INN has 10 residual layers, and each residual layer consists of 27 invertible residual blocks.

PU-GEO

PU-GCN

DIS-PU

Ours

Figure 4: Surface Comparisons. Comparisons of reconstructed surfaces from upsampled points of various methods, (EAR [Huang et al., 2013], PU-Geo [Qian et al., 2020], PU-GCN [Qian et al., 2021a], Dis-PU [Li et al., 2021] in 4× upsampling experiments using 5000 input points from PU25 dataset.

4.1 Comparisons with SOTAMethods

Quantitative Comparison. To quantitatively evaluate the quality of the output point cloud, we use the Earth Mover's distance (EMD), Chamfer distance (CD) [Butt, 1998], Hausdorff distance (HD) [Huttenlocher, 1993], Uniform Metric(UNI), and Point-to-surface (P2F) distance. A lower evaluation metric value indicates better performance. Table 1 shows the comparison results under the dataset PU1K [Qian et al., 2021a]. We observe that our method outperforms the previous SOTA methods on all the metrics. Specifically, the improvement of our method on EMD shows that it tends to encourage points to remain the underlying surface. Moreover, our method achieves the best results on CD and P2F metrics. The results show that PU-INN is more robust to noise compared with the other methods. Benefiting from the uniform loss constraint, our method outperforms the other methods especially on the uniformity metric, even for the challenging 16× upsampling.

Qualitative Comparison. The qualitative results of different point cloud upsampling models are presented in Figure 3. It shows that baseline methods tend to generate more noise (e.g., PU-Net), or destroy tiny-part structures (e.g., PUGAN). In high-curvature regions, PU-GCN cannot produce high-quality results. Dis-PU is able to well preserve tiny local structures by disentangled refinement module while losing the global information for guiding. Our output can better display the points' origin shape contour while preserving more structural details. As shown in Figure 4, we also reconstruct meshes from the generated upsampling points by SAP Algorithm [Peng et al., 2021], to show that our results help to reconstruct complex meshes with high-frequency details and fewer artifacts.

We have also conducted experiments to verify our model's robustness with respect to different input sizes and noise, and PU-INN achieves the best performance in all the tested cases. We give more details in Appendix B, which also shows more qualitative comparisons (real-scanned data, non-uniform data and other datasets), and quantitative comparisons.

4.2 Ablation Study

To evaluate the effectiveness of the major modules in the conditional injector of our proposed model, we conduct an ablation study in four cases: (1) removing the global feature extractor (w/o Injector(Global)); (2) removing the local feature extractor (w/o Injector(Local)); (3) removing the feature fusion module (w/o Injector(Fusion)), and the global and local features are concatenated after a transformation with MLP; and (4) removing the injector (w/o Injector). The time metric is the cost of PU-INN's inference with only one object, which has 5000 points, and the upsampling ratio is 4x. Table 2 shows that our conditional injector can effectively enhance the performance of our model on the EMD, CD, HD, UNI and P2F metrics. Although the computational cost and reference time are higher by introducing the injector, the computaion efficiency of our full method still outperforms the SOTA methods.

4.3 Lightweight Network Design

The existing point cloud upsampling methods generally construct networks with a large number of parameters, resulting in a complicated training procedure and high computation cost. The amounts of parameters typically over 100K, and some methods even contain over 1000K parameters. As discussed in Section 1, we design a lightweight interpolator to reduce the model complexity for the development of point cloud upsampling methods.

Benefiting from the architecture of invertible residual neural network, the resblock can share weights in the forward and reverse phases, and the elimination of gradient calculation in the reverse phase saves the calculations in the training phase, and thus reducing the model's parameters and saving the inference time. To reduce the computation cost, we only set the first residual block of each residual learning layer as a conditional residual block. As shown in Table 3, our model has 6.8x fewer parameters and 2.8x fewer FLOPs than PUFlow. PU-INN's inference speed performs up to 2.3x faster than PU-Flow.

5 Conclusions

In this paper, we propose a point cloud upsampling method based on invertible residual neural networks. Considering that bi-directional mapping limits the dimensionality of the input and output, we propose a conditional injector, which can inject the necessary feature information into the original network without affecting the invertibility and improve its nonlinear transformation capability. Furthermore, we design an interpolator based on semantic similarity distance, by which interpolating in latent space immediately reflects changes in Euclidean space. Extensive qualitative and quantitative experiments show that our network outperforms SOTA methods. With the good performance of PU-INN in solving upsampling problems, we could extend its adaptability to advanced 3D vision tasks, such as semantic segmentation in future work.

Acknowledgments

This work was supported by the NSF of Guangdong Province under 2022A1515011573, the Key-Area Research and Development Program of Guangzhou City (202206030009), the National Natural Science Foundation of China (U21A20520 and No.62202257), Beijing Natural Science Foundation (L222008).

Contribution Statement

Aihua Mao:Conceptualization, Funding acquisition, Investigation, Project administration, Resources, Supervision, Validation, Validation of analysis, Writing-review & editing. Yaqi Duan: Conceptualization, Formal analysis, Investigation, Methodology, Software, Validation, Writing - original draft, Writing - review & editing. Yu-Hui Wen: Writingreview & editing. Zihui Du: Conceptualization, Investigation, Validation. Hongmin Cai: Writing-review & editing. Yong-Jin Liu: Funding acquisition, Writing-review & editing. ∗ Corresponding author. † Equal first authorship.