RDPA: Real-Time Distributed-Concentrated Penetration Attack for Point Cloud Learning

1 Introduction

Numerous studies have demonstrated that 3D point cloud deep neural networks are extremely vulnerable to deception by adversarial examples [Xiang et al., 2019; Lee et al., 2020; Cao et al., 2021]. Most existing attack methods primarily perturb the entire point cloud, offering the advantage of not necessitating complex region selection or local processing. However, the added noise may propagate throughout the whole point cloud, thus making it challenging to implement in the physical domain.

∗Corresponding author.

Partial point attack approaches [Kim et al., 2021; Zheng et al., 2019; Wicker and Kwiatkowska, 2019; Zheng et al., 2023] focus on leveraging the fewest points to achieve the best attack efficiency. A common attack framework usually consists of two parts: point selection and point perturbation. Specific to these two aspects, by re-examining the characteristics of previous approaches, we discover two fundamental principles:

• The selection of attacked points should consider not only the favourable visual salience but also the proper position concentration, thus to acquire effective structural destruction on the basis of remaining imperceptible.
• The perturbation of target points should form meaningful structures rather than outliers.

An example of the first criterion is shown in the top row of Figure 1. Assuming a set of point clouds distributed in squares, point selection strategies that solely focus on visual salience may result in attacked points being evenly distributed along contours [Qi et al., 2017a; Zheng et al., 2019]. After removing the contour points with salient features, the point cloud still remains square in shape, but making it difficult to fool the robust neural network. On the other hand, when choosing only one salient region to attack, it indeed leads the neural network to make erroneous decisions. However, this approach causes particularly obvious damage to the structure because the selected points are too concentrated. For the second criterion, as shown in the bottom row of Figure 1, if we do not constrain the system to execute perturbation, the adversarial sample will have a large number of individual outliers whose attack success rate can be rapidly decreased by only imposing a simple Gaussian filter. Specifically, when applying two widely used defense methods, simple random sampling (SRS) and statistical outlier removal (SOR) [Zhou et al., 2019], the attack success rates against PointNet [Qi et al., 2017a] on ModelNet40 [Wu et al., 2015] using current SOTA attack approaches [Kim et al., 2021; Zheng et al., 2023] significantly declines from 89.41% and 92.35% to 17.60%, 3.77% and 50.32%, 16.73% respectively. Thus, the attack penetration must be considered in the framework.

Based on the analysis of above two principles, we propose a novel real-time distributed-concentrated penetration attack (RDPA) framework for point cloud learning, which not only can quickly generate high-quality adversarial point cloud but also show penetration ability against mainstream defenses. Specifically, in the point selection phase, different from previous works mainly focusing on the salience of the individual point or region, we propose a distributed-concentrated point selection (DPS) strategy which is easier to concentrate salient points containing rich local information in a few tiny regions. In the point disturbance phase, we design a perturbation network based on the multi-scale penetration loss (Lmsp), which can generate adversarial samples with as few outliers as possible only through a single forward propagation.

In general, the contributions of this paper are as follows:

• By re-examining the characteristics of previous partial point attack approaches leading to the performance improvement, we propose two fundamental principles for an effective attack scheme, in which the penetration ability is taken into account for the first time.
• A novel distributed-concentrated point selection (DPS) strategy and multi-scale penetration loss (Lmsp) are proposed to improve the attack efficiency and penetration ability against mainstream defenses.
• Comprehensive experiments show that the proposed attack framework is capable of providing SOTA results on various benchmarks. We also provide extensive ablation studies to examine the effect and robustness of different parts.

2 RelatedWorks

2.1 3D Point Cloud Attacks

Adversarial attacks on 3D point cloud models can be broadly classified into optimization-based, gradient-based, and dropbased methods. Optimization-based attacks, such as those of [Xiang et al., 2019] and [Carlini and Wagner, 2017], focus on perturbing point clouds by optimizing objective functions. Techniques such as adding custom loss terms [Tsai et al., 2020], using binary random vectors [Kim et al., 2021], and encoding high-dimensional data into a compact latent space [Lee et al., 2020] have also been explored. [Zhou et al., 2020] applied generative networks for the generation of adversarial examples.

Gradient-based methods modify the input by using model gradients. Early works introduced curvature losses [Wen et al., 2020], while [Goodfellow et al., 2014; Madry et al., 2017] refined gradient computation. Autoencoders were used by [Hamdi et al., 2020] to generate perturbations, and [Moosavi-Dezfooli et al., 2017; Naseer et al., 2019; Poursaeed et al., 2018] extended this approach. [Ma et al., 2020] proposed the Joint Gradient Based Attack (JGBA) and [Yang et al., 2019] introduced the Pointwise Gradient Enhanced Momentum (MPG) to improve the transferability of the attack. [Huang et al., 2022] adapted I-FGSM [Zhao et al., 2020] and [Tao et al., 2023] developed a black-box attack based on decision boundaries. [Zheng et al., 2023] targeted salient regions in point clouds for more effective attacks.

In dropping-based attacks, methods like Drop200 [Zheng et al., 2019] discard points based on importance scores. [Yang et al., 2019] further refined this by calculating gradients before maximum pooling in PointNet, while [Wicker and Kwiatkowska, 2019] classified models and used latent translations for keypoint definition to improve attack precision.

2.2 3D Point Cloud Defenses

Defensive strategies aim to purify adversarial samples or enhance model robustness. Purification methods include SOR, which uses upsampling networks [Zhou et al., 2019], IFDefense with implicit function networks [Wu et al., 2020], Adv3Diff combining diffusion and denoising [Zhang et al., 2023b], and IT-Defense, which employs invariant transformations [Zhang et al., 2023a].

To strengthen the robustness of the classifier, various data augmentation techniques have been proposed. PointMixup [Chen et al., 2020] and PointCutMix [Zhang et al., 2022]

blend point clouds, while RSMix [Lee et al., 2021] combines rigid subsets. Other methods include experiments on data augmentation [Sun et al., 2022], self-attention for feature extraction [Dong et al., 2020], and lattice classifiers [Li et al., 2022]. [Ding et al., 2023] introduced CAP, a framework that enhances both semantic and structural information in point clouds.

3 Method

As illustrated in Figure 3, the novel real-time distributedconcentrated penetration attack (RDPA) framework begins with employing the distributed-concentrated point selection (DPS) strategy as Separator to separate the attacked target points Ph and the remains Pl. Subsequently, both the original point cloud P and Ph are fed into the perturbation network named Disruptor, which can generate the adversarial subset Ph-adv only through a single forward propagation. Then, we merge Ph-adv and Pl to get the final adversarial point cloud Padv. By inputting Padv into the attacked classifier, we can calculate the misclassification loss Lmis, while comparing P and Padv determines the disturbance distance loss Ldis. Additionally, we design a novel multi-scale penetration loss Lmsp according to Padv.

3.1 Distributed-Concentrated Point Selection Strategy

Existing point selection strategies only focus on a single salient point or a single salient region. As shown in Figure 2, if only consider the salience when selecting points to attack, they are basically scattered in point cloud surface, such as critical subset strategy [Qi et al., 2017a] and saliency map algorithm [Zheng et al., 2019]. The disadvantage of these methods are that once disturbed points are defended, original points located on the subsurface will form a new profile with their neighboring points, which is why the attack success rate of method [Kim et al., 2021] will rapidly decline in the face of defenses. Another drawback is that the points to be attacked are too dispersed, resulting in minimal changes to the point cloud structure. In addition, as shown in the visualization result of local adversarial method [Zheng et al., 2023], choosing to attack a single salient region will cause serious deformation of the point cloud structure, which violates the principle of imperceptibility because the salient points are too concentrated.

Based on the analysis of above deficiencies, we consider both salience and concentration when selecting the attacked target points. Meanwhile, in order to prevent salient points from being concentrated in a certain region, we propose a distributed-centralized point selection (DPS) strategy which fully follows the first principle of the partial point attack. As shown in the final part of Figure 2, salient points have both the distributed characteristic and the advantage of centralized attacks. The unique advantage of DPS is that it can be effectively against minor defenses without causing excessive damage to the point cloud structure.

Specifically, we employ the covariance matrix to measure information in the nearest region of a point. By choosing appropriate neighbor numbers, we are able to concentrate information-rich salient points within a few tiny regions, which is a simple but effective method. Formally, a point cloud P is defined as a collection of N unordered points, denoted as P = {pi|i = 1, 2, ..., N} , P ∈ RN×3 , where each point pi is represented by a vector containing its (x, y, z) coordinates. To identify the n nearest points to a given point pi, we employ the k-nearest neighbor (knn) algorithm, resulting in a set denoted as knn (pi, n) = {pij|j = 1, 2, ..., n}. The mean vector of point pi is computed as

while the covariance matrix of the local region is defined as

which is a 3 by 3 real symmetric matrix.

Analyzing the eigenvalues and eigenvectors of the covariance matrix allows us to discern the primary magnitude and direction of change within the local structure of the point cloud. Let λi1,λi2 , and λi3 be the descending eigenvalues of the matrix respectively, where λi1 provides the richest local information because it indicates the magnitude of change along the major direction. To offer more nuanced insights into structural changes, we consider both the ratio of three eigenvalues and the max eigenvalue. Therefore, we define the salient score of point pi as

where ϵ is a constant set to prevent the denominator from being 0, and we set ϵ = 1 for consistency. When the differences between the three eigenvalues are significant, the left-hand side of the dot product ensures that the overall expression increases as these differences widen. Simultaneously, the righthand side ensures that si increases with the growth of the max eigenvalue.

After calculating the salient score si for each point pi, we first arrange them in descending order. Then, we define the top M points as the attacked target points Ph and the remaining N -M low-score points as Pl, where Ph ∈ RM×3 , Pl ∈

R(N-M)×3 . They have the following relationship:

As shown in the lower right part of Figure 3, for ∀ph ∈ Ph, pl ∈ Pl, we have sh ≥ sl.

3.2 Perturbation Network Based on the Multi-Scale Penetration Loss

After the separator extracts Ph and Pl from the original point cloud, we aim to design a network to generate Ph-adv , which can effectively meet the flexibility requirements for real-time attacks because it only requires a single forward pass. Meanwhile, to meet the second principle of our proposed partial point attacks, we design a novel multi-scale penetration loss to train the perturbation network. The specific structure of the perturbation network and its loss function is as follows:

Design of Disruptor

Regarding the design of the disruptor, we address the learning of both global and local features. This process involves three components: the original point cloud encoder Ep, the attacked target points encoder EPh , and the decoder DPh . Specifically, we employ the single-scale grouping set abstraction (SSGSA) algorithm [Qi et al., 2017b] for learning global features which involved a carefully designed coarse-to-fine five-layer farthest point sampling processes. In the i-th layer, the number of sampling points is N 2i , and the number of aggregated feature channels is 64·2i-1 . For learning local features, we employ a simple three-layer perceptron exclusively, with each layer outputting a shape of M × (64 · 2i-1 ). For the decoding process, the final output is the Ph-adv after perturbation. The entire encoder-decoder process is illustrated in Figure 3, its mathematical expression as follows:

Multi-Scale Penetration Loss

In order to make our attack framework more penetrating in the face of mainstream defense strategies, we propose Lmsp to constrain adversarial examples with as few outliers as possible. To identify outliers more accurately in local ranges of different densities, we begin with computing the multiscale average k-nearest neighbor distance Di for each point pi. In the following equations 6 to 9, we uniformly set n = 2t , t = 1, 2, 3:

followed by the calculation of the mean ¯D and standard deviation Σ of all distances on every scale within the point cloud:

Based on the multi-scale means and standard deviations, we define the multi-scale penetration loss Lmsp as follows:

Joint Loss Function

On the premise of ensuring the penetration of adversarial samples, we must balance the high attack success rate and good visual quality. Therefore, we propose a novel joint loss function, which consists of Lmis, Ldis, and Lmsp three terms:

where α, β are weight parameters.

Our work primarily focus on untargeted attacks rather than targeted because these are two different patterns. Untargeted attacks are mainly to deceive the network, while target attacks are to induce. Specifically, let F represents the classifier network for the point cloud, and Fi(P) denotes the likelihood of the point cloud being classified into the i-th class. Our attack can increase the likelihood of misclassification by the classifier, that is argmaxiFi(P) ̸= argmaxi′Fi′(Padv). The defination of Lmis is shown as follows:

where θ represents the optimization factor, usually with θ = 0; it denotes the correct label of the original point cloud P. Given the limited number of points affected by perturbation, we employ the Hausdorff distance to determine Ldis, defined as follows:

4 Experiments

4.1 Experiment Setup

Dataset and Attacked Classifier

We take ModelNet40 [Wu et al., 2015] and PointNet [Qi et al., 2017a] as the basic experimental dataset and classification network. Specifically, ModelNet40 comprises 12,311 CAD models across 40 common object categories, where 9,843 objects were allocated for training and 2,468 served as the testing set. Similar to [Qi et al., 2017a], we initially sample 1,024 points by the farthest point sampling (FPS) for each point cloud and then normalize them to fit within a sphere with a radius of 1. Due to the unified processing of data input, the disruptor can conduct additional training on multiple point cloud datasets to satisfy the needs of specific adversarial samples.

Implementation Details

When calculating salient scores, we utilize knn to select a local point count of 30 and define the attacked target subset comprising 36 points. For Disruptor, we employ Adam optimizer and CosineAnnealingLR scheduler, setting the learning rate to 0.001 and the cosine annealing cycle to 20. We train Disruptor on PyTorch using an NVIDIA RTX 3090 Ti GPU, over 400 epochs with a batch size of 64. Within the joint loss terms, we conducted an exhaustive search [Kim et al., 2021] for the parameters in Eq. 10 and empirically set them as α = 5 and β = 50. For the definition of Lmsp, we take γ=1.

4.2 Comparison and Evaluation

We evaluate RDPA from four primary dimensions: attack success rate (ASR), distance, generation time and disturbance point number. When evaluating ASR, we consider not only the performance of the classification network without any defense, but also the performance after incorporating mainstream defenses: SRS and SOR [Zhou et al., 2019], where SRS randomly samples and discards 500 points and SOR follows the same setup as [Zhou et al., 2019]. We measure distance employing Hausdorff and Chamfer distances, with the generation time excluding perturbation network training but including saliency score computation. In addition, we compare RDPA with six SOTA partial point attack methods: random occlusion (RO) [Wicker and Kwiatkowska, 2019], iterative significant occlusion (ISO) [Wicker and Kwiatkowska, 2019], critical subset based point dropping (CS-D) [Zheng et al., 2019], saliency map based point dropping (SM-D) [Zheng et al., 2019], minimal attack examples (MAE) [Kim et al., 2021] and adaptive local adversarial (Al-Adv) [Zheng et al., 2023].

As depicted in Table 1, in the absence of any defense, our RDPA exhibits the highest attack success rate at 93.52%, which is 1.17% higher than the second place method AlAdv and 4.11% higher than the third place method MAE. Even when facing the defenses SRS and SOR, the attack success rates of RDPA still remain high at 89.58% and 81.39%. However, in such a condition, the attack success rates of AlAdv and MAE drop significantly from 92.35% and 89.41% to 50.32%, 16.73% and 17.60%, 13.77%. Furthermore, RDPA takes an average of only 0.03 seconds to generate a single adversarial sample, allowing it to attack 2000 point cloud frames per second, which fully meets the flexibility requirements of real-time dynamic scenarios. At the same time, the minimal cost of 36 perturbation points provides significant guidance for practical attacks in the physical domain. While RDPA slightly lags behind ISO and MAE in Hausdorff distance and Chamfer distance, this discrepancy arises from their iterative querying of the classification network for enhanced results, which is a challenging process to the real-time attack because it is very time consuming.

Additionally, we validate RDPA on ScanObjectNN [Uy et al., 2019] setting as in [Kim et al., 2021], which is a high quality real-world dataset and include 15000 objects. As illustrated in Table 1, the comparison results on ScanObjectNN are similar to those on ModelNet40, which demonstrates the robust scalability of RDPA across diverse datasets. Compared with the hand-crafted dataset, the scanned dataset in the real world is more likely to achieve a high attack success rate, because the data itself has certain irregularity.

4.3 Ablation Study

The Effect of Each Part

In order to evaluate the proposed DPS, Lmsp and EP , we carry out a study on ModelNet40 against PointNet. As can be seen from Table 2, when the encoder EP is omitted, the attack success rate decreases drastically by more than 10%. Removing the loss term Lmsp slightly improves the attack success rate without defense, but reduces it by 23.95% against SOR defense. To validate DPS, we replace it with the point selection strategy based on saliency map [Zheng et al., 2019], resulting in decreases of 3.20%, 5.25%, and 4.46% under no defense, SRS defense, and SOR defense. This is because DPS considers both point salience and appropriate concentration, while Lmsp avoids the occurrence of individual outliers, and EP captures the global feature of the original point cloud.

The Size of KNN for DPS

The k value of knn employed in DPS plays a crucial role in the distribution of salient points. As shown in table 3, when k is set to a small value, such as 5, salient points tend to disperse along the contour surface, whose attack success rate is similar to that of the point selection strategy based on saliency map [Zheng et al., 2019]. However, as k increases, salient points begin to cluster. Imagine that when k becomes infinitely large, i.e., k = N (where N is the total point number of a point cloud), salient points will cluster within a local region because each point uses the entire point cloud in calculating, where RDPA had been degraded to Al-Adv method. Therefore, to achieve both distributed characteristics and concentrated attack advantages for salient points, clustering them into several tiny regions is most appropriate. From the experimental data in table 3, setting k to 30 results the best distribution and attack success rate.

The Analysis of Attacked Point Number

As shown in Figure 4, the experiment reveals that as the number of attack points increases, the attack success rate (ASR) rises, the Hausdorff distance decreases, and the Chamfer distance slightly increases. However, when the number of attack points exceeds 36, the success rate begins to decline. Beyond 128 points, although the success rate improves slightly, the excessive number of attack points becomes impractical and unnecessary. Consequently, selecting 36 points as the optimal attack number ensures a high success rate while maintaining low interference costs and imperceptibility.

4.4 Robustness Analysis

Transferability

To validate the transferability of adversarial samples, we perform black-box attacks on four other classification networks, namely PointNet++ [Qi et al., 2017b], DGCNN [Wang et al., 2019], CurveNet [Xiang et al., 2021], and PCT [Guo et al., 2021]. At the same time, we compare RDPA with Al-Adv. PointNet is the pioneer designed to address the disorder nature and rigid transformation invariance of point cloud, while PointNet++ builds on PointNet as a foundational layer and incorporates sampling-grouping layers. To further strengthen the geometric relationship, DGCNN performs edge convolution on the neighborhood graph and CurveNet groups the point cloud into multiple curves. PCT applies attention mechanisms to point cloud based on Transformer.

As demonstrated in table 4, when no defense, the blackbox attack success rates of RDPA consistently exceed 15%, while that of Al-Adv is below 3%. This is because DPS is based on analyzing the structure of the point cloud, rather than relying on the specific classification network being attacked. And, during the point perturbation phase, Al-Adv employs an iterative query method, which means the generated adversarial samples lack generality. In the face of SRS, the black-box attack success rates surpasses that of no defense because the removal of points disrupts the structure of the point cloud, which can be exploited by adversarial samples. Therefore, SRS is not suitable as the black-box defense mechanism. In addition, despite lower black-box attack success rates with SOR defense compared to no defense, RDPA still outperforme Al-Adv.

Visual Quality

Figure 5 illustrates typical instances of adversarial samples for three classes of point clouds: airplane, chair, and guitar. Since the attack methods RO, ISO, CS-D, and SM-D are based on dropping mechanisms, their adversarial samples appear relatively sparse. Specifically, methods RO and ISO tend to drop points from the entire point cloud, whereas methods CS-D and SM-D are more inclined to drop points from specific local regions, such as the head of an airplane or the legs of a table. MAE is inclined to attack contour points and generates a large number of individual outliers. Al-Adv targets individual salient region, causing severe disruption to the point cloud structure and similarly generating outliers. In contrast, RDPA chooses to attack multiple tiny regions, enhancing the imperceptibility of adversarial samples. Furthermore, the perturbed points formmeaningful structures rather than outliers, which makes RDPA equally penetrating against mainstream defenses. These analyses highlight the impacts of diverse methods, demonstrating the trade-off of RDPA between structural disruption and imperceptibility.

5 Conclusion

In conclusion, this paper first proposes the partial point attack strategy should pay attention to not only the selection and disturbance of points but also the penetration of defenses, which identifies two fundamental principles: first, the selection of attacked points should consider both the visual salience and the proper position concentration; second, the perturbation of target points should form meaningful structures rather than outliers. Further, we introduce the novel RDPA framework leveraging the DPS strategy and the Disruptor design based on Lmsp, which achieves a 93.52% attack success rate by perturbing only 3.5% of points. Even when faced with the mainstream defenses SRS and SOR, the attack success rates still remain at 89.58% and 81.39%. In addition, RDPA requires an average of only 0.03 seconds to generate a single adversarial sample, fully meeting the requirements for real-time attacks. In the future, we plan to extend RDPA to real-world applications, particularly focusing on enhancing the robustness of large-scale models.

Acknowledgements

This work is supported in part by the National Natural Science Foundation of China (No. 62471415); the Natural Science Foundation of Fujian Province of China (No. 2023J01004).