MC3D-AD: A Unified Geometry-aware Reconstruction Model for Multi-category 3D Anomaly Detection

1 Introduction

Anomaly Detection (AD) is a critical task for quality control in the manufacturing industry. Early research has concentrated on 2D image data and has achieved significant advancements [Zavrtanik et al., 2021; Zhang et al., 2023; You et al., 2022; Lu et al., 2023]. With the increasing demand for highprecision industrial products, 3D-AD [Bergmann et al., 2022; Liu et al., 2023] has garnered growing attention from researchers, and its objective is to identify and localize anomalous points or regions from 3D point cloud data.

∗Corresponding author

Point cloud data exhibits the characteristics of disorder, sparseness, and structurelessness, which pose great challenges for anomaly detection. Existing 3D-AD methods [Ye et al., 2024] can generally be categorized into: feature embedding-based and reconstruction-based ones. For feature embedding-based approaches, methods like Reg3D-AD [Liu et al., 2023] and Group3-AD [Zhu et al., 2024] have demonstrated their effectiveness by extracting feature embeddings from normal samples. While reconstruction-based methods, such as IMRNet [Li et al., 2024] and R3DAD [Zhou et al., 2024b], focus on learning key features to restore point cloud data, achieving anomaly detection by calculating reconstruction errors.

Additionally, in light of the rich information hidden in multimodal data, multimodal 3D-AD has also attracted much attention from researchers. Some methods, such as BTF [Horwitz and Hoshen, 2023], extract statistical information from both RGB and depth modules to perform anomaly detection. Recently, deep learning-based approaches, such as M3DM [Wang et al., 2023] and CMPF [Cao et al., 2024], have shown promising results by learning feature representations frommultimodal data.

Despite achieving very appealing results, some challenges still exist: (1) The trained models are task-specific and lack the generalization to different tasks. In other words, they are required to train an individual and distinct model for each category, seriously limiting their practicality. (2) Existing reconstruction-based methods may fail to learn the intrinsic features for reconstruction, leading to the problem of "identity shortcut", where the input is directly copied for output without considering its content [You et al., 2022]. As verified in 2D images [You et al., 2022; He et al., 2024; Lu et al., 2023], this phenomenon is significantly amplified in the setting of multi-category anomaly detection. To address these challenges, we propose a unified geometry-aware reconstruction model for 3D anomaly detection named MC3DAD. Different from previous methods which require training a privately owned model for each category, our method aims to train only one unified model to perform 3D anomaly detection for all categories (See Figure 1).

In light of that point cloud anomalies are usually manifested as irregularities and abnormalities in local geometry structures, we introduce an adaptive geometry-aware masked attention to improve the local feature representation for different categories. It explicitly computes geometric variations within the neighborhood of points and intentionally masks key features selected using their quantified geometric information. To accurately identify anomalies from point cloud data, a transformer-based [Vaswani et al., 2017] architecture is adopted, which incorporates point cloud position embeddings [Li et al., 2023] as global queries and the adaptive geometry-aware masked attention to reconstruct feature tokens for multi-category anomaly detection and localization.

The main contributions of this paper are summarized as follows:

• To perform multi-category 3D anomaly detection, we present a unified framework based on feature reconstruction. To the best of our knowledge, it is the first time to explore multi-category 3D anomaly detection by training only one model.
• To learn robust and generalized representations across categories, we propose a novel adaptive geometry-aware masked attention, which explicitly captures neighborhood geometry information for representation, facilitating the extraction of reconstruction features and also enhancing the interpretability of the model.
• To achieve accurate anomaly detection, we design a local and global geometry-aware transformer, which is reinforced by the proposed adaptive geometry-aware masked attention, thereby providing the ability to reconstruct point clouds from different categories.
• Extensive experiments are conducted to verify the effectiveness of the proposed model, and very impressive results are achieved, with an object-level AUROC improvement of 3.1% over the state-of-the-arts singlecategory model on Real3D-AD and 9.3% on AnomalyShapeNet, respectively.

2 RelatedWork

2.1 Feature Embedding-BasedMethods

Feature embedding-based methods[Liang et al., 2025] extract features from normal samples to form a memory bank using pre-trained models and identify anomalies by comparing test sample features with those in the memory bank. Reg3DAD [Liu et al., 2023] used the pre-trained PointMAE [Pang et al., 2022] to extract normal features from registered point cloud data and constructed a memory bank to store both global geometric and local coordinate features for anomaly detection. Group3AD [Zhu et al., 2024] introduced contrastive learning for clustering groups to ensure intra-cluster compactness and inter-cluster uniformity, leveraging grouplevel features stored in a memory bank to detect anomalies. 3D-ST [Bergmann and Sattlegger, 2023] adopted a studentteacher framework to perform feature matching between two networks for 3D anomaly detection. M3DM [Wang et al., 2023] utilized contrastive learning to align RGB and depth modalities, creating a fused representation and a threelevel memory bank to jointly enhance detection performance. CPMF [Cao et al., 2024] projected 3D point clouds into 2D images from multi-view and considered image features as global semantic information to complement 3D features, thereby establishing a pseudo multimodal memory bank for anomaly detection. PointAD [Zhou et al., 2024a] aligned local and global features extracted from 2D projections of 3D point clouds using a pre-trained vision-language model, enabling zero-shot 3D anomaly detection.

2.2 Reconstruction-BasedMethods

Reconstruction-based methods try to encode normal point cloud data into informative feature representation and restore these features into the original form, with points exhibiting high reconstruction errors identified as anomalies. IMRNet [Li et al., 2024] enhanced PointMAE by incorporating geometric-preserving downsampling and random masking to improve reconstruction fidelity for anomaly detection. R3DAD [Zhou et al., 2024b] leveraged PointNet to iteratively reconstruct fully masked point clouds using a diffusion process, enabling precise localization of abnormal regions. Shape-Guided [Chu et al., 2023] introduced dual memory banks to store normal features extracted from RGB and 3D modalities and reconstructed the input sample at the feature level to achieve robust anomaly detection. Although achieving encouraging results, these methods need to train a taskspecific model for each category. Therefore, it is highly desired to develop a unified all-in-one model for all categories.

3 The Proposed Approach

3.1 ProblemDescription

For multi-category 3D anomaly detection, the available data in the training phase contains point cloud samples from multiple categories, i.e., Ptrain = {P 1 train , P 2 train , · · · , P c train}, where P i train denotes the training data from the i-th category and only have normal samples, and c is the number of categories. In the testing phase, the data to be detected includes both normal and anomalous point cloud samples from different categories, i.e., Ptest = {P 1 test , P 2 test , · · · , P c test}. The objective is to train a unified model for multiple categories using only normal training data.

3.2 Overview Framework

A key challenge in multi-category 3D anomaly detection is to develop a unified representation method to simultaneously adapt to different categories. To this end, we propose a novel geometry-aware reconstruction framework for multi-category 3D anomaly detection. The overall framework is presented in Figure 2, which consists of three main components: Adaptive Geometry-aware Masked Attention (AGMA), Local Geometry-aware Encoder (LGE), and Global Query Decoder (GQD). Each component is described in the following sections.

3.3 Adaptive Geometry-Aware Masked Attention

The representation of normal point cloud data is the key factor for the success of 3D anomaly detection. Existing reconstruction-based methods use the mask attention mechanism to improve the representation ability. Nevertheless, due to the high variation and complexity of point cloud data, they may fail to learn the intrinsic features to represent normal point cloud samples, leading to the phenomenon of "identity shortcut". To address this problem, we propose an AGMA module, which explicitly computes neighborhood geometry information for representation, providing not only better reconstruction ability but also model interpretability.

Specifically, given a point cloud P = {p1, p2, · · · , pn} with each pi ∈ R3 , the group centers Pcenter = {p1, p2, · · · , pm} can be obtained by sampling the point cloud through the Furthest Point Sampling (FPS), which can be expressed as:

To capture the local geometric information hidden in neighborhood structure, the adaptive neighborhood of group centers is introduced and can be expressed as:

where ∥ · ∥2 denotes the 2-norm, and r represents the neighborhood radius. Due to the varying scales of point clouds across different categories, an adaptive radius is employed to ensure that the number of points in the neighborhood of each group center point remains consistent. The calculation of r can be expressed as:

where | · | denotes the cardinality of a set, pnearest i is the nearest neighbor of the pi, and η is a scaling factor to adjust the size of the neighborhood.

Subsequently, to reflect the geometric information within the neighborhood structure, we define the normal vector and curvature for each group center point pi, which can be formulated as:

where FEVmin means finding the eigenvector corresponding to the smallest eigenvalue, λi and λmin represents the i-th and the minimum eigenvalue of the covariance matrix, respectively, and Covi is the covariance matrix of group center point pi, which is defined as:

where µi is the centroid of the neighborhood Nr(pi) of pi. Based on the calculated normal vector Ni ∈ R3 and curvature Ci, we can further define an index to quantify the variation in geometric information:

where ∠(·, ·) denotes the angle between two vectors, V arnorm i , V arcruv i , and V argeom i represent the degree of change in the normal vector, curvature, and geometric information within the adaptive neighborhood, respectively, and the α and β are hyper-parameters to balance the values.

Intuitively, this geometric change information can be used to guide the learning of representation for reconstruction. Therefore, an improved mask attention mechanism shown in Figure 3 is introduced, aiming to mask some key feature tokens to enhance representation ability.

Specifically, geometric variation information is first extracted for group center points. Then, points with larger and smaller values are randomly selected, respectively, according to the ratio ρ of tokens to be masked, and their corresponding group feature tokens are masked during attention calculation. With this mask attention mechanism, feature representation ability can be significantly improved, and the interpretability of the model is accordingly enhanced.

3.4 Local Geometry-Aware Encoder

Since anomalies usually account for a small area of the entire point cloud, thus encoding local features is essential for improving anomaly detection performance. To encode local group features with geometric information, we propose an LGE to incorporate neighborhood geometric variation information into the feature encoding process. Specifically, a point group is formed by applying k-Nearest Neighbors (KNN) to each group center point pi, which can be defined as:

This operation is similar to patch extraction in 2D images, aiming to extract local features from data sequences. Subsequently, the point groups G = {G1,G2, · · · ,Gg} are input into the feature extractor F to generate group-level feature tokens Ftok ∈ Rg×c , where g is the number of groups and c is the number of channels. Concurrently, the group centers Pcenter are passed through a MultiLayer Perceptron (MLP) σ to obtain position embeddings Fpos ∈ Rg×c , which serve as the Transformer's positional encodings. Additionally, before inputting the feature tokens Ftok into the LGE, feature jittering[You et al., 2022; Bengio et al., 2013] is adopted by adding perturbation to the features. This promotes the representation and reconstruction capability through the denoising process. The perturbation for the i-th group feature can be expressed as:

where N(·, ·) denotes the normal distribution, and γ is a scaling factor to control the intensity of perturbation on the group feature token.

Finally, the group feature tokens with added noise are fed into the LGE for local feature encoding. The LGE is composed of N sequential blocks, each consisting of an AGMA module and a Feed Forward Network (FNN), where the FNN is implemented by a 2-layer fully connected MLP. By incorporating the AGMA module, the LGE module can encode robust local features with geometric awareness, facilitating subsequent decoding and reconstruction.

3.5 Global Query Decoder

Effectively encoding local features only does not ensure accurate and complete anomaly localization. Moreover, it is evident that global information can positively guide the reconstruction process and improve decoding ability. To this end, we propose a GQD module, which leverages global queries to improve anomaly localization. Specifically, the previously obtained position embeddings Fpos are considered as global queries to be fed into an AGMA. Subsequently, the results are further added with the position embeddings again and fed into another AGMA, followed by an FNN as in the LGE.

To improve the decoding and reconstruction ability, the GQD stacks N repeated blocks. In each block, the local encoding features from the LGE and the position embeddings acted as global queries are input into the first AGMA. Then, the output of the previous block is combined with the localglobal features obtained from the first AGMA and fed into the second AGMA. This interaction promotes feature fusion between blocks and facilitates feature decoding and reconstruction. Finally, the GQD outputs the reconstructed feature tokens F rec ∈ Rg×c , which is optimized by the MSE loss:

(b) P-AUROC(↑)

During the testing phase, the test point cloud is first registered and grouped for the feature extractor to generate feature tokens. Then, the proposed reconstruction model tries to encode and decode them into the original form. The reconstruction difference is normalized and subjected to average pooling to obtain the final pixel-level anomaly score Sp, which can be expressed as:

where Norm denotes the min-max normalization, and AvgPool means the operation of average pooling with the kernel size of 1*512. The anomaly score indicates the likelihood of the point being anomalous, and the maximum value of Sp is used as the object-level anomaly score So.

4 Experiment

4.1 Experiment Settings

Datasets. (1) Real3D-AD [Liu et al., 2023] is a highresolution point cloud anomaly detection dataset consisting of 1,254 samples from 12 object categories. Each category has only four training samples but contains anomalies with varying shapes and sizes. (2) Anomaly-ShapeNet [Li et al., 2024] is a synthetic point cloud anomaly detection dataset containing 1,600 samples across 40 categories. Each sample contains between 8,000 and 30,000 points, with the anomalous region accounting for 1% to 10% of the entire point cloud. Due to the large number of categories, this dataset is more challenging for multi-class anomaly detection.

Comparison Methods. Our method adopts the setting of multi-category anomaly detection, where only one model is uniformly trained for all categories. Because existing methods can not apply to multiple categories directly, the compared methods in the experiment use the single-category configuration, wherein privately owned models are separately trained for each category. The proposed method is compared with some representative methods, including BTF [Horwitz and Hoshen, 2023], M3DM [Wang et al., 2023], PatchCore[Roth et al., 2022], CMPF [Cao et al., 2024], Reg3D-AD[Liu et al., 2023], Group3AD [Zhu et al., 2024], IMRNet [Li et al., 2024], and R3D-AD [Zhou et al., 2024b].

Implementation Details. PointMAE pre-trained on ModelNet408K [Wu et al., 2015] is adopted as the feature extractor of our method. The AdamW optimizer is used in the training process, and the learning rate is initially set to 0.0001 and dropped to 0.00001 after 800 epochs. The batch size and the maximum number of epochs are set to 1 and 1000, respectively. The hyperparameters α, β, η, and ρ for AGMA are set to 1, 10, 7, and 0.4, respectively. The number of stacked blocks N in LGE and GQD is set to 4. Our method is performed on PyTorch 1.13.0 and CUDA 11.7 with an NVIDIA A100-PCIE-40GB GPU.

Evaluation Metrics. In the experiments, the Area under the Receiver Operating Characteristic Curve (AUROC, ↑) is used to assess the performance of object-level anomaly detection and pixel-level anomaly localization.

4.2 Main Results

Results on Real3D-AD. The comparison results of MC3DAD and the existing methods are shown in Table 1. It can be observed that MC3D-AD achieved SOTA performance, with an AUROC of 0.782 at the object level and 0.768 at the pixel level. These results are 3.1% and 1.0% higher than those of the second-best method, which is a task-specific model that carefully tunes for each category.

Results on Anomaly-ShapeNet. The experimental results of MC3D-AD on the Anomaly-ShapeNet dataset are presented in Table 2. Despite the increased complexity of this dataset, which comprises 40 categories, MC3D-AD still achieved SOTA performance. Specifically, the object-level AUROC for anomaly detection reached 0.842, outperforming the second-best single-category method by 9.3%. Additionally, the pixel-level AUROC for anomaly localization attained 0.748, which is 8.0% higher than the second-best single-category approach. These results clearly demonstrate the generalization capabilities of MC3D-AD in multi-class anomaly detection setting.

4.3 Ablation Studies

Table 3 presents the results of the experiments evaluating the effectiveness of the proposed modules. Specifically, LGE refers to a model that incorporates only the LGE module, without the AGMAmodule. LGEAGMA refers to the application of AGMA within the LGE framework, while GQDAGMA indicates the addition of AGMA to the GQD module. The results show that AGMA plays a crucial role in capturing point cloud geometry features, significantly enhancing its reconstruction capability. This leads to a significant increase in anomaly detection performance, with a 9.4% improvement in O-AUROC. On the other hand, LQD provides additional guidance for the model's reconstruction process, further improving anomaly localization and yielding an 8.3% improvement in P-AUROC. AGMA can be seamlessly integrated into both LGE and GQD, resulting in an overall performance enhancement across all metrics.

4.4 Analysis of Hyper-Parameters

Our method introduces two key parameters η and ρ to control the size of the adaptive neighborhood and the proportion of masked tokens, respectively. As shown in Figure 4, an inappropriate η value, whether too small or too large, can lead to inaccurate quantization of geometric variation, so a balanced value η = 7 is set to ensure stable performance. Similarly, the mask proportion in the attention mechanism requires careful tuning: a lowmask ratio hinders the learning of robust reconstruction, while a high mask ratio increases the difficulty of reconstruction. In the experiments, the parameter ρ is set to 0.4, at which the best performance is reached.

4.5 Visualization

Figure 5 shows the heatmap visualization results of our MC3D-AD on the Real3D-AD. It is observed that MC3D-AD accurately detects and localizes anomalous regions within the point cloud, clearly demonstrating its effectiveness. In addition, Figure 6 provides the heatmap visualization of the proposed AGMA, which encapsulates the geometric information extracted from the point cloud. The blue regions indicate that the geometric information of the point cloud varies slowly within the adaptive neighborhood, while the transition from green to yellow and red indicates a gradual increase in the change of geometric information within the neighborhood. By intentionally masking blue or red areas during training, the reconstruction ability of our method is greatly improved.

5 Conclusion

In this paper, we propose a unified reconstruction framework for multi-category anomaly detection. We introduce an adaptive geometry-aware guided mask attention module, where geometric variation information is captured for robust and generalized representation of different categories. Additionally, we design a geometry-aware transformer with global position embeddings and local mask attention to learn robust reconstructed features. Experiments on benchmark datasets show that our method outperforms existing approaches, achieving state-of-the-art performance. However, further research is needed to exploit the utilization of geometric variation information and develop more robust and efficient frameworks for multi-category anomaly detection.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (Grant Nos. 62476171, 62476172, and 62206122), the Guangdong Basic and Applied Basic Research Foundation (Grant No. 2024A1515011367), the Guangdong Provincial Key Laboratory (Grant No. 2023B1212060076), the Tencent "Rhinoceros Birds" - Scientific Research Foundation for Young Teachers of Shenzhen University, and the Shenzhen Institute of Artificial Intelligence and Robotics for Society.