MM-PCQA: Multi-Modal Learning for No-reference Point Cloud Quality Assessment

1 Introduction

Point cloud has been widely adopted in practical applications such as virtual/augmented reality [Park et al., 2008], automatic driving [Cui et al., 2021], and video post-production [Mekuria et al., 2016b] due to its ability of representing the 3D world. Consequently, plenty of works have been carried out to deal with point cloud classification [Grilli et al., 2017; Ku et al., 2018; Wang and Jia, 2019; Vora et al., 2020; Xie et al., 2020; Yoo et al., 2020; Chen et al., 2020], detection [Cui et al., 2021], and segmentation [Cheng et al., 2021;

Liu et al., 2022b]. However, point cloud quality assessment (PCQA) has gained less attention. PCQA aims to predict the visual quality level of point clouds, which is vital for providing useful guidelines for simplification operations and compression algorithms in applications such as metaverse and virtual/augmented reality (VR/AR) [Fan et al., 2022a] to not negatively impact users' quality of experience (QoE). Moreover, the point clouds representing vivid objects/humans are usually more complex in geometric structure and contain large amounts of dense points along with color attributes, which makes the PCQA problem very challenging.

Generally speaking, PCQA methods can be divided into full-reference PCQA (FR-PCQA), reduced-reference (RRPCQA), and no-reference PCQA (NR-PCQA) methods according to the involvement extent of the reference point clouds. However, the pristine reference point clouds are not always available in many practical situations, thus NR-PCQA has a wider range of applications. Hence, we focus on the NR-PCQA in this paper. Reviewing the development of NRPCQA, most metrics are either based on the point cloud features extracted by statistical models [Zhang et al., 2021a; Zhang et al., 2022c] and end-to-end neural networks [Liu et al., 2022c] or based on the projected image features obtained via hand-crafted manners [Meynet et al., 2020; Yang et al., 2020b; Alexiou and Ebrahimi, 2020] or 2D convolutional neural networks (CNN) [Liu et al., 2021b; Fan et al., 2022b; Zhang et al., 2022d]. Such methods fail to jointly make use of the information from 3D point clouds along with 2D projections, thus resulting in unsatisfactory performance.

Therefore, to boost the performance of PCQA, we propose a multi-modal learning strategy for NR-PCQA, which extracts quality-aware features not only from the 3D point clouds but also from the 2D projections. There are two main reasons to adopt this strategy. First, point clouds can be perceived in both 2D/3D scenarios. We can view point clouds from 2D perspective via projecting them on the screen or directly watch point clouds in 3D format using the VR equipment. Thus multi-modal learning is able to cover more range of practical situations. Second, different types of distortions have divergent visual influence on different modalities. As shown in Fig. 1, the structural damage and geometry downsampling are more obvious in the point cloud modality while the image modality is more sensitive to texture distortions caused by color quantization and color noise. Moreover, it is easier to extract quality-aware semantic information from the image modality. Thus, the proposed multi-modal learning fashion can make up for the deficiencies and take advantage of both modalities. Further, considering that the local patterns such as smoothness and roughness are very important for quality assessment, we first propose to split the point cloud into sub-models rather than sampling points for analysis, which has been previously adopted for extracting the low-level pattern features of the point cloud [Chetouani et al., 2021; You and Gao, 2021]. The image projections are acquired by rendering the point clouds from several viewpoints with a fixed viewing distance to maintain the consistency of the texture scale. Then a point cloud encoder and an image encoder are used to encode the point clouds and projected images into quality-aware embeddings respectively, which are subsequently strengthened by symmetric cross-modality attention. Finally, the quality-aware embeddings are decoded into quality values with fully-connected layers. The main contributions of this paper are summarized as follows:

• We propose a no-reference Multi-Modal Point Cloud Quality Assessment (MM-PCQA) to interactively use the information from both the point cloud and image modalities. To the best of our knowledge, we are the first to introduce multi-modal learning into the PCQA field.
• To preserve the local patterns that are vital for visual quality, we propose to split the point cloud into submodels rather than sampling points as the input of the point cloud encoder. To better incorporate the multimodal features, we employ cross-modal attention to model the mutual relationship between the quality-aware features extracted from two modalities.
• Extensive experiments show that MM-PCQA achieves the best performance among the compared state-of-theart methods (even including the FR-PCQA methods). The ablation studies reveal the contributions of different modalities, the patch-up strategy, and cross-modal attention, demonstrating the effectiveness of the proposed framework.

2 RelatedWork

2.1 Quality Assessment for Point Cloud

In the early years of PCQA development, some simple pointbased FR-PCQA methods are proposed by MPEG, such as p2point [Mekuria et al., 2016a] and p2plane [Tian et al., 2017]. To further deal with colored point clouds, point-based PSNR-yuv [Torlig et al., 2018] is carried out. Since the point-level difference is difficult to reflect complex distortions, many well-designed FR-PCQA metrics are proposed to employ structural features and have achieved considerable performance, which includes PCQM [Meynet et al., 2020], GraphSIM [Yang et al., 2020b], PointSSIM [Alexiou and Ebrahimi, 2020], etc.

To cover more range of practical applications and inspired by NR image quality assessment [Zhang and Liu, 2022; Zhang et al., 2022a; Zhang et al., 2021b], some NR-PCQA methods have been proposed as well. Chetouani et al. [Chetouani et al., 2021] extract patch-wise hand-crafted features and use classical CNN models for quality regression. PQA-net [Liu et al., 2021b] utilizes multi-view projection for feature extraction. Zhang et al. [Zhang et al., 2022c] use several statistical distributions to estimate quality-aware parameters from the geometry and color attributes' distributions. Fan et al. [Fan et al., 2022b] infer the visual quality of point clouds via the captured video sequences. Liu et al. [Liu et al., 2022c] employ an end-to-end sparse CNN for quality prediction. Yang et al. [Yang et al., 2022] further transfer the quality information from natural images to help understand the point cloud rendering images' quality via domain adaptation. The mentioned methods are all based on single-modal information, thus failing to jointly incorporate the multi-modal quality information.

2.2 Multi-modal Learning for Point Cloud

Many works [Radford et al., 2021; Cheng et al., 2020] have proven that multi-modal learning is capable of strengthening feature representation by actively relating the compositional information across different modalities such as image, text, and audio. Afterwards, various works utilize both point clouds and image data to improve the understanding of 3D vision, which are mostly targeted at 3D detection. Namely, AVOD [Ku et al., 2018] and MV3D [Chen et al., 2020] make region proposals by mapping the LiDAR point clouds to bird's eye view. Then Qi et al. [Qi et al., 2018] and Wang et al. [Wang and Jia, 2019] attempt to localize the points by proposing 2D regions with 2D CNN object detector and then transforming the corresponding 2D pixels to 3D space. Pointpainting [Vora et al., 2020] projects the points into the image segmentation mask and detects 3D objects via LiDAR-based detector. Similarly, PI-RCNN [Xie et al., 2020] employs semantic feature maps and makes predictions through point-wise fusion. 3D-CVF [Yoo et al., 2020] transfers the camera-view features using auto-calibrated projection and fuses the multi-modal features with gated fusion networks. More recently, some works [Afham et al., 2022; Zhang et al., 2022b] try to make use of multi-modal information in a self-supervised manner and transfer the learned representation to the downstream tasks.

3 ProposedMethod

The framework overview is clearly exhibited in Fig. 2. The point clouds are first segmented into sub-models and put into the point cloud encoder θP . The projected images are directly rendered from the colored point clouds and put into the image encoder θI. Subsequently, the quality-aware encoder features are optimized with the assistance of symmetric crossmodality attention. Finally, the features are concatenated and decoded into the quality values via the quality regression.

3.1 Preliminaries

Suppose we have a colored point cloud P = {g(i), c(i)}Ni=1 , where g(i) ∈ R1×3 indicates the geometry coordinates, c(i) ∈ R1×3 represents the attached RGB color information, and N stands for the number of points. The point cloud modality ˆP is obtained by normalizing the original geometry coordinates and the image modality I is generated by rendering the colored point cloud P into 2D projections. Note that ˆP contains no color information.

3.2 Point Cloud Feature Extraction

It is natural to transfer mainstream 3D object detectors such as PointNet++ [Qi et al., 2017] and DGCNN [Wang et al., 2019] to visual quality representation of point clouds. However, different from the point clouds used for classification and segmentation, the high-quality point clouds usually contain large numbers of dense points, which makes it difficult to extract features directly from the source points unless utilizing down-sampling. Nevertheless, the common sampling methods are aimed at preserving semantic information whereas inevitably damaging the geometry patterns that are crucial for quality assessment. To avoid the geometry error caused by the down-sampling and preserve the smoothness and roughness of local patterns, we propose to gather several local sub-models from the point cloud for geometric structure feature representation.

Specifically, given a normalized point cloud ˆP, we employ the farthest point sampling (FPS) to obtain Nδ anchor points

𝑁𝑁𝐼𝐼

Figure 4: Illustration of the projections rendering process. Camera is rotated along the circular pathway with intervals of 2π NI to capture NI projections.

{δm}Nδ m=1 . For each anchor point, we utilize K nearest neighbor (KNN) algorithm to find Ns neighboring points around the anchor point and form such points into a sub-model:

where S is the set of sub-models, KNN(⋅) denotes the KNN operation and δm indicates the m-th farthest sampling point. An illustration of the patch-up process is exhibited in Fig. 3. It can be seen that the local sub-models are capable to preserve the local patterns. Then a point cloud feature encoder θP (⋅) is employed to map the obtained sub-models to qualityaware embedding space:

where F lP ∈ R1×CP indicates the quality-aware embedding for the l-th sub-model Sl, CP represents the number of output channels of the point cloud encoder θP(⋅), and ̃ FP ∈ R1×CP is the pooled results after average fusion.

3.3 Image Feature Extraction

NI image projections are rendered from the distorted colored point clouds from a defined circular pathway with a fixed viewing distance to keep the texture consistent as shown in Fig. 4:

where the pathway is centered on the point cloud's geometry center, R indicates the fixed viewing distance and the NI image projections are captured with intervals of 2π NI . The projections are rendered with the assistance of Open3D [Zhou et al., 2018]. Then we embed the rendered 2D images into quality-aware space with the 2D image encoder:

where F t I ∈ R1×CI denotes the quality-aware embedding for the t-th image projection It, CI represents the number of out-

�𝐹𝐹𝐼𝐼

�𝐹𝐹𝑃𝑃

Figure 5: Illustration of the symmetric cross-modality attention module (SCMA). The point cloud embedding ˆ FP is used to guide the attention learning of image embedding and the image embedding ˆ FI is used to guide the attention learning of point cloud embedding respectively.

put channels of the 2D image encoder θI(⋅), and ̃ FI ∈ R1×CI is the pooled results after average fusion.

3.4 Symmetric Cross-Modality Attention

As shown in Fig 1, the single modality may be incomplete to cover sufficient information for quality assessment, thus we propose a symmetric attention transformer block to investigate the interaction between the visual quality features gathered from the point cloud and image modalities. Given the intra-modal features ̃ FP ∈ R1×CP and ̃ FI ∈ R1×CI from the point clouds and images respectively, we adjust them to the same dimension with linear projection:

where ˆ FP ∈ R1×C′ and ˆ FI ∈ R1×C′ are adjusted features, WP and WI are learnable linear mappings, and C ′ is the number of adjusted channels. To further explore the discriminate components among the modalities, the multi-head attention module is utilized:

where Γ(⋅) indicates the multi-head attention operation, β(⋅) represents the attention function, hµ is the µ-th head, and W, WQ, WK, WV are learnable linear mappings. As illustrated in Fig. 5, both point-cloud-based and image-based quality-aware features participate in the attention learning of the other modality. The final quality embedding can be concatenated by the intra-modal features and the guided multimodal features obtained by the symmetric cross-modality attention module:

where ⊕(⋅) indicates the concatenation operation, Ψ(⋅) stands for the symmetric cross-modality attention operation, and

ˆ FQ represents the final quality-aware features (Ψ( ˆ FP , ˆ FI) ∈ R1×2C′ , ˆ FQ ∈ R1×4C′ ).

3.5 Quality Regression &Loss Function

Following common practice, we simply use two-fold fullyconnected layers to regress the quality features ˆ FQ into predicted quality scores. For the quality assessment tasks, we not only focus on the accuracy of the predicted quality values but also lay importance on the quality rankings. Therefore, the loss function employed in this paper includes two parts: Mean Squared Error (MSE) and rank error. The MSE is utilized to keep the predicted values close to the quality labels, which can be derived as:

where qη is the predicted quality scores, q′η is the quality labels of the point cloud, and n is the size of the mini-batch. The rank loss can better assist the model to distinguish the quality difference when the point clouds have close quality labels. To this end, we use the differentiable rank function described in [Sun et al., 2022] to approximate the rank loss:

where i and j are the corresponding indexes for two point clouds in a mini-batch and the rank loss can be derived as:

Then the loss function can be calculated as the weighted sum of MSE loss and rank loss:

where λ1 and λ2 are used to control the proportion of the MSE loss and the rank loss.

4 Experiments

4.1 Databases

To test the performance of the proposed method, we employ the subjective point cloud assessment database (SJTUPCQA) [Yang et al., 2020a], the Waterloo point cloud assessment database (WPC) proposed by [Liu et al., 2022a], and the WPC2.0 database [Liu et al., 2021a] for validation. The SJTU-PCQA database includes 9 reference point clouds and each point cloud is corrupted with seven types of distortions (compression, color noise, geometric shift, down-sampling, and three distortion combinations) under six strengths, which generates 378 = 9×7×6 distorted point clouds in total. The WPC database contains 20 reference point clouds and augments each point cloud with four types of distortions (down-sampling, Gaussian white noise, Geometry-based Point Cloud Compression (G-PCC), and Video-based Point Cloud Compression (V-PCC)), which generates 740 = 20×37 distorted point clouds. The WPC2.0 databases provides 16 reference point clouds and degradaes the point clouds with 25 V-PCC settings, which generates 400 = 16×25 distorted point clouds.

4.2 Implementation Details

The Adam optimizer [Kingma and Ba, 2015] is utilized with weight decay 1e-4, the initial learning rate is set as 5e-5, and the batch size is set as 8. The model is trained for 50 epochs by default. Specifically, We set the point cloud sub-model size Ns as 2048, set the number of sub-models Nδ = 6, and set the number of image projections NI = 4. The projected images with the resolution of 1920×1080×3 are randomly cropped into image patches at the resolution of 224×224×3 as the inputs (the white background is removed from the projected images). The PointNet++ [Qi et al., 2017] is utilized as the point cloud encoder and the ResNet50 [He et al., 2016] is used as the image encoder, where the ResNet50 is initialized with the pre-trained model on the ImageNet database [Deng et al., 2009]. The multi-head attention module employs 8 heads and the feed-forward dimension is set as 2048. The weights λ1 and λ2 for LMSE and Lrank are both set as 1.

Following the practices in [Fan et al., 2022b], the k-fold cross validation strategy is employed for the experiment to accurately estimate the performance of the proposed method. Since the SJTU-PCQA, WPC, and WPC2.0 contain 9, 20, 16 groups of point clouds respectively, 9-fold, 5-fold, and 4fold cross validation is selected for the three database to keep the train-test ratio around 8:2. The average performance is recorded as the final results. It's worth noting that there is no content overlap between the training and testing sets. We strictly retrain the available baselines with the same database split set up to keep the comparison fair. What's more, for the FR-PCQA methods that require no training, we simply validate them on the same testing sets and record the average performance.

4.3 Competitors and Evaluation Criteria

14 state-of-the-art quality assessment methods are selected for comparison, which consist of 8 FR-PCQA methods and 6 NR-PCQA methods. The FR-PCQA methods include MSEp2point (MSE-p2po) [Mekuria et al., 2016a], Hausdorffp2point (HD-p2po) [Mekuria et al., 2016a], MSE-p2plane (MSE-p2pl) [Tian et al., 2017], Hausdorff-p2plane (HDp2pl) [Tian et al., 2017], PSNR-yuv [Torlig et al., 2018], PCQM [Meynet et al., 2020], GraphSIM [Yang et al., 2020b], and PointSSIM [Alexiou and Ebrahimi, 2020]. The NRPCQA methods include BRISQUE [Mittal et al., 2012a], NIQE [Mittal et al., 2012b], IL-NIQE [Zhang et al., 2015], ResSCNN [Liu et al., 2022c], PQA-net [Liu et al., 2021b], and 3D-NSS [Zhang et al., 2022c]. Note that BRISQUE, NIQE, IL-NIQE are image-based quality assessment metrics and are validated on the same projected images. Furthermore, to deal with the scale differences between the predicted quality scores and the quality labels, a five-parameter logistic function is applied to map the predicted scores to subjective ratings, as suggested by [Antkowiak et al., 2000].

Four mainstream evaluation criteria in the quality assessment field are utilized to compare the correlation between the predicted scores and MOSs, which include Spearman Rank Correlation Coefficient (SRCC), Kendall's Rank Correlation Coefficient (KRCC), Pearson Linear Correlation Coefficient (PLCC), Root Mean Squared Error (RMSE). An excellent quality assessment model should obtain values of SRCC, KRCC, PLCC close to 1 and RMSE to 0.

Performance Discussion

The experimental results are listed in Table 1, from which we can make several useful inspections: a) Our method MMPCQA presents the best performance among all 3 databases and outperforms the compared NR-PCQA methods by a large margin. For example, MM-PCQA surpasses the second place NR-PCQA method by about 0.05 (0.91 vs. 0.86 (ResSCNN)) on the SJTU-PCQA database, by 0.14 (0.84 vs. 0.70 (PQA-net)) on the WPC database, and by 0.05 (0.80 vs. 0.75 (ResSCNN )) on the WPC2.0 database in terms of SRCC. This is because MM-PCQA enforces the model to relate the compositional quality-aware patterns and mobilize under-complementary information between the image and point cloud modalities to optimize the quality representation by jointly utilizing intra-modal and cross-modal features; b) There are significant performance drops from the SJTU-PCQA database to the WPC and WPC2.0 databases since the WPC and WPC2.0 databases contain more complex distortion settings, which are more challenging for PCQA models. MM-PCQA achieves a relatively smaller performance drop than most compared methods. For example, the SRCC and PLCC values of MM-PCQA drop by 0.07 and 0.08 respectively from the SJTU-PCQA database to the WPC database. However, the top-performing PCQA methods except 3D-NSS experience a larger performance drop over 0.1 on both SRCC and PLCC values. Therefore, we can conclude that MM-PCQA gains more robustness over more complex distortions; c) The image-based handcrafted NR methods BRISQUE, NIQE, and IL-NIQE obtain the poorest performance. This is because such methods are carried out for evaluating the natural scene images, which have a huge gap from the projected images rendered from the point clouds. In all, the overall experimental performance firmly validates our motivation that multi-modal learning can help the model better understand the visual quality of point clouds.

4.4 Contributions of the Modalities

As described above, MM-PCQA jointly employs the features from the point cloud and image modalities and we hold the hypothesis that multi-modal learning helps the model to gain better quality representation than single-modality. Therefore, in this section, we conduct ablation studies to validate our hypothesis. The performance results are presented in Table 2. The performance of both single-modal-based model is inferior to the multi-modal-based model, which suggests that both point cloud and image features make contributions to the final results. After using SCMA module, the performance is further boosted, which validates the effectiveness of crossmodality attention.

4.5 Ablation for the Patch-up Strategy

To prove that the patch-up strategy is more suitable for PCQA, we present the performance of using farthest point sampling (FPS) strategy as well. To make the comparison even, 12,288 = 6×2048 points (the same number of points as contained in the default 6 sub-models) are sampled for each point cloud and the results are shown in Table 3. It can be seen that the patch-up strategy greatly improves the performance when only point cloud features are used. The reason is that the sampled points are not able to preserve the local patterns that are vital for quality assessment. Moreover, when the point cloud contains more points, the sampled points may not even maintain the main shape of the object unless greatly increasing the number of sampling points.

4.6 Cross-database Evaluation

The cross-database evaluation is further conducted to test the generalization ability of the proposed MM-PCQA and the experimental results are exhibited in Table 4. Since the WPC database is the largest in scale (740), we mainly train the models on the WPC database and conduct the validation on the SJTU-PCQA (378) and WPC2.0 (400) databases. From the table, we can find that the proposed MM-PCQA better generalizes learned feature representation from the WPC database and achieves much better performance than the most competitive NR-PCQA methods. Furthermore, the WPC→WPC2.0 MM-PCQA is even superior to all the competitors directly trained on the WPC2.0 database.

4.7 Number of 2D Projections and 3D sub-models

We try to investigate the contributions of the point cloud and image branch by varying the number of the 3D sub-models and input 2D projections. The performance results are exhibited in Table 5. We can see that employing 6 sub-models and 4 projections yields the best performance on major aspects for the point cloud and image branch respectively. With the number increasing, the features extracted from sub-models and projections may get redundant, thus resulting in the performance drop.

4.8 Different Feature Encoders

In this section, we present the performance of different feature encoders. The popular 2D image backbones VGG16 [Simonyan and Zisserman, 2014], MobileNetV2 [Sandler et al., 2018] and ResNet50 [He et al., 2016] are used for demonstration while the mainstream point cloud backbones PointNet++ [Qi et al., 2017] and DGCNN [Wang et al., 2019] are also included. The results are shown in Table 6. It can be found that the proposed PointNet++ and ResNet50 combination is superior to other encoder combinations. Additionally, the performance of different backbones are still competitive compared with other NR-PCQA methods, which further confirms the effectiveness of the proposed framework.

5 Conclusion

This paper proposes a novel multi-modal learning approach for no-reference point cloud quality assessment (MMPCQA). MM-PCQA aims to acquire quality information across modalities and optimize the quality representation. In particular, the point clouds are patched up to preserve the important local geometric structure patterns. Then the projected images are employed to reflect the texture distortions. PointNet++ and ResNet50 are utilized as the feature encoders. Symmetric cross-modal attention is further employed to make full use of the cross-modal information. Experimental results show that MM-PCQA reaches a new state-of-the-art on the SJTU-PCQA, WPC and WPC2.0 databases. Extensive ablation studies further demonstrate the potency of the proposed multi-modal learning framework.

Acknowledgments

This work was supported in part by NSFC (No.62225112, No.61831015), the Fundamental Research Funds for the Central Universities, National Key R&D Program of China 2021YFE0206700, and Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102).