Pointsoup: High-Performance and Extremely Low-Decoding-Latency Learned Geometry Codec for Large-Scale Point Cloud Scenes

1 Introduction

Large-scale point clouds are widely used in numerous 3D applications, such as Augmented Reality/Virtual Reality (AR/VR), autonomous driving, robotics, etc., owing to their capacity to realistically represent objects and scenes [Quach et al., 2022; Abbasi et al., 2023]. A large-scale point cloud typically consists of millions of sparse points, making it challenging to store and transmit, which urges the development of Point Cloud Compression (PCC) techniques.

Background. Two international PCC standards, i.e., Video-based PCC (V-PCC) and Geometry-based PCC (GPCC), were developed under the Moving Picture Experts G-PCCv23 (Octree)

∗Corresponding author

G-PCCv23 (Trisoup)

0.228 / 60.899dB 5.191s

Figure 1: Quantitative compression results of proposed Pointsoup and G-PCCv23. Colors are rendered by nearest mapping. The "conferenceRoom 1" in S3DIS Area 6 is used as an example, which has 1,067,709 points. Our method allows for the decoding of a millionscale point cloud geometry in 39 ms with only one RTX 2080Ti GPU while guaranteeing superior visual quality.

Group (MPEG) and released as part of ISO/IEC 23090-5 and 23090-9 [Chen et al., 2023; WG 7, 2020; WG 7 and Coding, 2023]. The octree representation [Schnabel and Klein, 2006] is adopted in G-PCC to efficiently encode the geometry information. Alternatively, the Trisoup geometry codec is a viable option in G-PCC to perform lossy geometry compression more effectively by estimating local point cloud surface as triangle meshes.

Recently, owing to the significant gains obtained by learning-based approaches, both MPEG and Joint Photographic Experts Group (JPEG) committees have launched explorations on Artificial Intelligence (AI) based PCC solutions. Despite the powerful performance demonstrated in the compression of dense point cloud objects, learningbased methods [Wiesmann et al., 2021; Fu et al., 2022; You et al., 2022; Huang et al., 2022; Wang et al., 2022; Zhang et al., 2023a; Song et al., 2023] still face the following two major challenges in compressing sparse point cloud scenes: 1) Unsatisfactory compression performance. The variability of sparse surfaces can make the neural model easily collapse; 2) High decoding power demands that restrict the application scope. We argue that the computational complexity for decoding should be rather low, in order to adapt to different computing-power client devices as well as livestreaming applications.

Our Approach. To address the above issues, we propose Pointsoup, an efficient learning-based geometry codec that attains high-performance and extremely low-decoding-latency simultaneously. Specifically, we first design the Aligned Window-based Down-Sampling (AWDS) module, which allows for the learned embedding of local surfaces, leveraging an effective attention-based aggregation. We then devise the Dilated Window-based Entropy Modeling (DWEM) module to aggregate dilated windows, which are built upon downsampled bones, to estimate the distribution of quantized features in parallel. Finally, a fast feature refinement block is intergrated with an efficient folding-based point generator, in the Dilated Window-based Up-Sampling (DWUS) module, to reconstruct the local surface with fast speed. Moreover, the Pointsoup provides variable-rate control with a single neural network model, by fully exploiting the flexibility of the pointbased pipeline.

Contribution. Main contributions can be summarized as:

• Leveraging a point model that harnesses an attentionbased encoder and the dilated window-based entropy modeling, our method achieves state-of-the-art efficiency on multiple large-scale benchmarks.
• By designing a fast feature refinement block followed by an efficient folding-based point generator, our method achieves extremely low-decoding latency. It enables nearly real-time decoding for million-scale point clouds, with up to 90∼160× faster than the G-PCCv23 Trisoup decoder on only one RTX 2080Ti GPU.
• Our method shows strong generalization capability and offers flexible bitrate control with a lightweight neural model, which is beneficial for practical applications.

2 RelatedWork

Numerous works have contributed to the Point Cloud Geometry Compression (PCGC) task, which can be roughly divided into two categories: voxel models and point models.

Voxel Model. Considering the sparsity of the point cloud geometry, original Point Cloud Geometry (PCG) can be reorganized to octree structure [Schnabel and Klein, 2006; Fu et al., 2022; Song et al., 2023] or multi-scale sparse representation [Wang et al., 2021a; Wang et al., 2022; Zhang et al., 2023a]. The octree iteratively divides the occupied space to produce an efficient tree-structured format, which is adopted by the well-known MPEG G-PCC standard [WG 7 and Coding, 2023] for its effectiveness and scalability. As another optional codec, Trisoup models the surface of the point cloud as a series of triangle meshes, which yields superior compression performance, but at the expense of high computational cost. The sparse tensor-based approach [Wang et al., 2022;

ÿ

ℱ௦௞௜௡

ÿ௦௞௜௡

Zhang et al., 2023a; Fan et al., 2023] utilizes multi-scale sparse representation and delivers significant compression gains. However, stacked convolutional layers still impose substantial computational demands, limiting their application scenarios.

Point Model. The past several years have witnessed the emergence of point-based techniques [Guo et al., 2021; Wu et al., 2022; Vinodkumar et al., 2023], which promoted the development of point models for learned PCGC tasks. Some works explored small-scale PCC techniques, but generally lack the applicability to large-scale point clouds [You and Gao, 2021; Zhang et al., 2022]. Other point models for largescale point clouds, on the other hand, struggle to achieve a competitive trade-off between compression efficiency and computing overhead [He et al., 2022; You et al., 2022; Huang et al., 2022; Huang and Wang, 2023]. For instance, 3QNet [Huang et al., 2022] devised the Model Breaking Strategy (MBS) to divide point cloud into blocks first, but the MBS can lead to significant gaps between blocks, which significantly diminishes visual quality. As a real-time codec designed for dense point maps, Depoco [Wiesmann et al., 2021] reports a low coding complexity, but it suffers from severe quality degradation.

3 Methodology

3.1 Framework

The overall workflow of our proposed Pointsoup is shown in Fig. 2. Specifically, the surface of the input point cloud P ∈ RN×3 is embedded to skin features Fskin ∈ RM×C by the Aligned Window-based Down-Sampling (AWDS) module, and the down-sampled bones are then instantly encoded and decoded, through the predictive tree of G-PCC. The Dilated Window-based Entropy Modeling (DWEM) module is Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI-24)

for the local details, based on a pervasive graph convolution (GraphConv) operation:

where N (pj , P iAW , km) represents finding km nearest neighbors of the point pj on the aligned window P iAW ; GraphConv is defined as GraphConv(·)=MaxPool(MLP(·)); F i (0) [j] ∈ R1×C refers to the jth feature vector (corresponding to the point pij ) in the feature matrix F i (0) ∈ RK×C .

Then, self-attention blocks are stacked following the subtraction vector attention [Wu et al., 2022]. It should be noted that since the local window is generated by KNN query, the first row of the Key matrix always represents the feature that attached to the center point of the window. Therefore, the Query matrix is constructed by repeating the first row of the Key, producing the subtraction relation anchored at the window center. Then, the process of the lth attention block can be described as follows:

where ϱ means the Softmax operation; Pem and Peb refer to the positional encoding multiplier and bias; Query, Key, and V alue are abbreviated as Q, K, and V, for a concise explanation.

Finally, the feature F i (L) ∈ RK×C output by the final selfattention block is aggregated to the skin feature F i skin ∈ R1×C , by a max-pooling operation:

We set km=16 and L=2 in our experiment.

3.3 DilatedWindow-based EntropyModeling (DWEM)

Considering the significant dependency among the target local window and the nearby area of the down-sampled skeleton, dilated window is introduced as the cross-scale prior to approximate the distribution of the skin feature. Meanwhile, the skin feature is further squeezed, yielding a compact representation for fast arithmetic coding.

ysis tasks [Zhao et al., 2021; Zhang et al., 2023b; Li et al.,

2023a], which first use the Farthest Point Sampling (FPS) to derive the skeleton Pbone ∈ RM×3 and then build a KNearest Neighbor (KNN) graph to formulate overlapping local windows. Particularly, due to the high computational cost of FPS, we first use the Random Point Sampling (RPS) to obtain a subset with no more than M×16 points, and then apply FPS on the subset to derive the result Pbone with M points.

Adaptive Aligning

We align the obtained overlapping local windows to facilitate network learning and enhance density adaptability. Each window is first shifted to the coordinate origin and then rescaled according to the skeleton density. Mathematically,

where P iAW refers to the aligned local window, and N(pi,P,K) represents finding K nearest neighbors of the point pi on the input point cloud P.

Attention-based Aggregation

Given an aligned window P iAW ∈ RK×3 , we embed the local surface into a high dimensional feature vector F i skin ∈ R1×C , by an effective attention-based neural network, as illustrated in Fig. 4.

Specifically, we first perform a mini embedding on each point within the window to produce feature F i (0) ∈ RK×C

DWEM: Dilated Window-based Entropy Modeling.

Dilated Window Construction DWUS: Dilated Window-based Up-Sampling.

ℱ෠ ௦௞௜௡

As shown in Fig. 5, a local window is dilated by employing the KNN graph on the down-sampled bones. Recall that the bones are handled by G-PCC at the base level, which makes dilated neighborhoods readily accessible to serve as the crossscale prior. Mathematically, a dilated window P iDW is defined as: ÿ෠ ௕௢௡௘ ÿ෠

where P iDW ∈ Rk×3 , N represents the k nearest neighbors of the down-sampled point pi on the decoded skeleton ˆ Pbone. k is set to 8 in our experiments.

Feature Compaction

Higher dimensional features lead to an increase in arithmetic coding complexity, due to the presence of longer symbol sequence to be coded. Thus, a squeezing operation is used by leveraging a simple fully-connected layer. Mathematically,

where F i skin ∈ R1×C , f i skin ∈ R1×c . Note that the skin features are stretched back by another Linear layer after arithmetic decoding. We set C=128 and c=16 in our experiments.

Cross-Scale Entropy Modeling

A uniform scalar quantizer is used in this work, which is replaced with an additive uniform noise during training [Ball´e et al., 2016; Jamil et al., 2023]. Let the quantized skin features be ˜ fskin = Q(fskin), then it is further modeled as:

where Pθ represents the entropy model parameterized by θ, L (Φi ) refers to the Laplacian distribution of quantized feature ˜ fskin with parameter Φi = (µi , σi ), and U (- 12 , 12 ) denotes the uniform distribution ranging from [- 12 , 12 ]. Here, the parameter Φi can be estimated from dilated window by a network that contains a GraphConv layer and a regression head MLP:

Finally, the expected bit rate for the skin features can be written as:

where N denotes the number of points of the input point cloud.

DWUS: Dilated Window-based Up-Sampling.

ÿ෠ ௕௢௡௘

3.4 DilatedWindow-based Up-Sampling (DWUS)

Unlike the progressive multi-scale fashion used in the previous point models [Huang et al., 2022; He et al., 2022], we fully exploit the single-scale strategy to reduce the complexity: First, the skin features that are attached to the skeleton undergo fast refinement, owing to the small quantity of features M, which is orders of magnitude less than the original input scale N; Then, lightweight Folding operation is devised to generate point coordinates based on shallow MLPs.

Fast Feature Refinement

Figure 6 details the fast feature refinement block, where the Dilated Window-based Convolution (DWConv) is introduced to integrate information from dilated windows. The reuse of the built dilated window avoids the recomputation of the spatial graph [Wang et al., 2019; Li et al., 2023b], thus provides an economized graph structure for feature convolution. To be specific, features of the points within the corresponding dilated window are gathered into groups based on the provided dilated index. Then, GraphConv is applied to aggregate the gathered groups to produce refined features.

Folding-based Point Generator

Backed by the refined features, a lightweight folding-based point generator is devised to generate point coordinates with fast speed, as shown in Fig. 7. To be specific, we first define the Folding operation as a combination of MLP and reshape, where MLP is used to upscale the input features and reshape operation is used to fold the output dimensions. The first Folding transforms each skin feature from a 1 × C vector to a grid matrix of Rmax × D. Then, the rows of the grid will be randomly sampled to the shape of R ×D, where R ∈ [1, Rmax]. The sampling technique allows an adjustable number of generated points, which is crucial for the variablerate control mechanism, as will be detailed in Sec. 3.5. Then, the down-sampled feature grid is concatenated with the input skin features, followed by another Folding function to generate point coordinates.

Inverse Aligning

The inverse aligning operation mirrors the adaptive aligning used in the encoder. Each reconstructed window ˆ P iAW is shifted to the original position and rescaled back, to assemble into a completed reconstructed result ˆ P. Mathematically,

where ˆ d represents the scale factor recalculated by ˆ Pbone, as described in Eq. 1.

3.5 Variable-Rate Control

A single-model-variable-rate solution is suggested based on the local window size that can be flexibly handled by the point model. Similar to the node-size adjustment of the G-PCC Trisoup codec, we modulate the size K of the queried local windows to adapt to different bit rates. Inspired by [You et al., 2022], we set M = ⌊ 2N K ⌋, i.e., a denser skeleton is grown for smaller windows to capture finer details at higher bitrate budget, while a sparser skeleton is presented with larger windows to naturally reduce the bit rate. At the decoder, we employ the feature sampling technique to reconstruct the window under the given size K, by adapting the parameter R of the point generator to ⌊ Kr ⌋, where r is fix to 4 in our implementation.

3.6 Loss Function

We follow the conventional rate-distortion trade-off as our loss function:

where DCD(P, ˆ P) refers to the Chamfer Distance between input point cloud P and reconstructed point cloud ˆ P, Rskin refers to the bit rate as described in Eq. 11.

4 Experiments

4.1 Experimental Setup

Training Dataset. We limit the training process on the ShapeNet [Chang et al., 2015] training set, which consists of 35,708 point clouds, each generated by uniformly sampling 8k points from a CAD model.

Test Dataset. Both large-scale indoor scenes and outdoor maps are considered for testing. The indoor point cloud scenes includes Stanford Large-Scale 3D Indoor Spaces Dataset (S3DIS) [Armeni et al., 2016] and ScanNet [Dai et al., 2017]. The outdoor point cloud maps are generated from KITTI [Behley et al., 2019], following [Wiesmann et al., 2021]. Details of used test sets are provided in Tab. 1.

Settings. We implement our model using Python 3.10 and Pytorch 2.0. Adam optimizer is used with an initial learning rate of 0.0005 and a batch size of 1. We train our model only once for 140,000 steps under the local window size of 128. The λ that balances the rate and distortion is set to 10-4 . Down-sampled bones are compressed losslessly by G-PCC predictive tree. All experiments are conducted on an Intel Core i9-9900K CPU and one RTX 2080Ti GPU.

Benchmarking Baselines. We compare our method with state-of-the-art rules-based methods: the default Octree codec and improved Trisoup codec of the lastest G-PCCv23 [WG 7 and Coding, 2023], which are denoted as "G-PCCv23" and "G-PCCv23 (T)", respectively; and learning-based methods: OctAttention [Fu et al., 2022], IPDAE [You et al., 2022], and 3QNet [Huang et al., 2022]. All learning-based methods are retrained on the same dataset as our method, and all test samples are normalized to the coordinate range of [0, 1023] (a.k.a., 10-bit precision) for ease of fair comparison. In addition, we compare Depoco [Wiesmann et al., 2021] and DPCC [He et al., 2022] following their recommended test conditions in Sec. 4.4.

4.2 Quantitative Comparison

Rate-Distortion Performance. Figure 8 shows the ratedistortion curves of different methods and Tab. 2 demonstrates the quantitative results using BD-PSNR and BD-Rate metrics. It can be seen that the proposed Pointsoup achieves the best rate-distortion performance, providing 60%∼64% bitrate reduction over the G-PCCv23 anchor.

Subjective Visual Quality. Figure 1 shows visualized results for an example indoor scene, where our method reconstructs the point cloud surface with a uniform point distribution that shares the same nature as the ground truth. An example of an outdoor point cloud map is presented in Fig. 10, in which finer details can be observed in our output (e.g., higher fidelity vehicle shapes and vegetation outlines).

Computational Complexity. As can be seen from Fig. 9 and Tab. 2, our method reports the lowest decoding latency, e.g., 90∼160× faster than the G-PCCv23 Trisoup decoder and 3∼5× faster than the Octree decoder. Moreover, the encoding time of the proposed Pointsoup is also significantly faster than the Trisoup, presenting a manageable encoding complexity. In addition, our network is fairly small with 761k parameters (about 2.9MB), which is much lighter than other point models such as 3QNet (85MB) and IPDAE (68∼516MB for each bitrate point).

4.3 Customized Training Domain

The training process in the above section is limited to the ShapeNet dataset. Intuitively, it is worth considering a training domain that closely resembles the test scenario to enhance reconstruction accuracy. Therefore, we retrain each model on S3DIS Area 1∼5 and test them on Area 6 again. As seen from Fig. 8 (S3DIS∗ ), the performances of Pointsoup and OctAttention are significantly improved, due to the similar patterns shared between training and test samples. Unexpectedly, both IPDAE and 3QNet exhibit a severe degradation, possibly indicating their inadequate capacity in handling complex training samples [Wang et al., 2021b].

4.4 Scenario Dependent Comparison

Depoco [Wiesmann et al., 2021] and D-PCC [He et al., 2022] are representative point models for large-scale point cloud compression. However, they do not support the generalization from a small-scale training set (e.g., ShapeNet) to largescale test frames. In this subsection, we follow their training/testing conditions for a comparative study, i.e., we verify our model using the respective training and test datasets as suggested in their papers. As seen from Tab. 3, our method significantly outperforms Depoco and D-PCC in terms of rate-distortion performance. It is worth noting that, despite the faster coding speed of Depoco, it comes at the expense of severely compromised reconstruction quality.

4.5 Ablation Study

Adaptive Aligning. It is imperative to consider the adaptation of the point densities as they vary with the number of points and the size of the scenes. The quantity-based aligning [You et al., 2022] is another reasonable way of density adaptation, but it only considers the influence of the number of points, while neglecting the impact of the spatial volume of the point cloud. For instance, they do not adapt well to narrow hallways, as evidenced in Fig. 11. On the contrary, we

0.546 / 57.192dB / 0.759s

Table 5: Ablation study on feature compaction. S3DIS Area 6 is used for test. G-PCCv23 serves as the anchor. AD refers to the time of arithmetic decoding.

infer local densities from down-sampled bones, which provides better aligned results by exploiting cross-scale priors.

Predictive Tree vs. Octree. Octree is another optional codec to compress down-sampled bone points. However, since predictive tree is particularly designed for low complexity decoding, using the octree will decelerate the decoding speed (about 2∼3× slower), as shown in Fig. 12, despite some rate-distortion benefits (7.979% BD-BR gain).

Network Components. As shown in Tab. 4, several key modules are disabled individually to examine the validity of the components. Note that the attention block is changed to MLP, and the DWEM is replaced by the basic factorized prior model [Ball´e et al., 2016] during study. Results show that disabling the DWEM module will lead to 47.243% BD-Rate loss relative to the original Pointsoup, which demonstrates the significant efficiency of the cross-scale entropy modeling.

Feature Compaction. We use a linear layer to squeeze skin feature to a compact representation, which speed up the arithmetic coding of the features. As shown in Tab. 5, the feature compaction operation greatly improves the speed of the decoding, reducing the arithmetic decoding time from 22 ms to 4 ms. In addition, the compaction yields a more efficient representation, leads to a slight bitrate reduction.

Please refer to our supplementary material for more comparisons and ablation studies.

5 Conclusion

This paper proposes an efficient learning-based geometry codec, dubbed Pointsoup, aiming at large-scale point cloud scenes. The Pointsoup demonstrates state-of-the-art compression efficiency with significantly low decoding latency and variable-rate controllability, making it a promising option for AI-based PCC solutions. Downstream tasks (e.g., object detection) may consider working directly on compressed domain without the need of a complete reconstruction. Source code and supplementary material are available at https://github.com/I2-Multimedia-Lab/Pointsoup.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (No. 62272227).