Multi-Modal Point Cloud Completion with Interleaved Attention Enhanced Transformer

1 Introduction

Point cloud, as a common 3D data format, is widely used in real-world scenarios such as autonomous driving [Chen et al., 2024], medical treatment [Liu et al., 2023], and robotics [Cheng et al., 2022]. Given its importance and versatility, many researchers are currently interested in 3D computer vision. Owing to inherent limitations of 3D sensor devices and environmental factors, point cloud data collected in real-world settings is often sparse and incomplete, which adversely impacts downstream tasks such as point cloud segmentation [Du et al., 2024c; Liang et al., 2023; Du et al., 2024b], registration [Zhang et al., 2023b; Mu et al., 2024], and detection [Pei et al., 2023; Zhang et al., 2021b].

* Corresponding author

Figure 1: An illustration of our core idea. Subfigure (a) illustrates that existing methods typically employ a cross-attention mechanism to fuse point cloud and image data, where information transfer is unidirectional (from the image to the point cloud). In contrast, Subfigure (b) demonstrates that the proposed method utilizes an interleaved attention mechanism to enable bidirectional information transfer between the point cloud and image modalities.

Single-modal point cloud completion [Fei et al., 2022; Tesema et al., 2024], restoring the incomplete input to 3D objects with complete shapes, is proposed to address this challenge. From the perspective of the generated completion results, existing methods can be divided into two categories. Some approaches [Huang et al., 2020; Alliegro et al., 2021; Yu et al., 2021] focus on generating missing shapes, while others [Yuan et al., 2018; Zhou et al., 2022; Wang et al., 2024] directly produce complete geometric structures. Both usually encode the incomplete input into a global representation. The difference lies in the decoding part; the former usually restores the global representation to the shape of the missing area, while the latter restores the global representation to a complete 3D object. While the above methods have achieved remarkable results, they are still difficult to recover enough details of the missing areas due to the incompleteness of partial input [Zhang et al., 2021a; Zhang et al., 2022].

To address this challenge, a straightforward approach introduces additional modality information to assist point cloud completion. Zhang et al. [Zhang et al., 2021a] propose a view-guided point cloud completion (ViPC) framework that, for the first time, combines image information with point cloud information for point cloud completion. Notably, the major challenge in view-guided point cloud completion is how to efficiently utilize image modality. To address this challenge, CSDN [Zhu et al., 2024] proposes a novel IPAdaIN module to transfer global shape information of image modality to that of point cloud modality. Different from CSDN, most methods such as XMFNet [Aiello et al., 2022], CDPNet [Du et al., 2024a], and EGIINet [Xu et al., 2024] adopt a cross-attention mechanism to fuse point cloud and image modalities. As shown in Figure 1, although the above methods have achieved certain performance, they do not fully utilize the image information and ignore that the structural information in the point cloud representation can enhance the geometry of the image representation.

To address above challenges, we propose the interleaved attention enhanced Transformer (IAET) consisting of three parts, i.e., token embedding, bidirectional token supplement, and coarse-fine decoding. Different from previous methods that use a cross-attention mechanism to fuse point cloud and image modalities in a unidirectional manner, we adopt a bidirectional information transfer strategy to enhance the interaction between modalities. Specifically, we employ existing 2D and 3D feature extractors to generate image tokens and point cloud tokens. Subsequently, we propose the interleaved attention mechanism which enables mutual supplementation between the point cloud and image tokens. Finally, we utilize the supplemented image tokens to guide the transformation of point cloud tokens into the completion results. IAET demonstrates state-of-the-art performance on the multi-modal point cloud completion benchmark datasets. The main contributions of this paper are as follows.

• For multi-modal point cloud completion, we suggest the interleaved attention enhanced Transformer (IAET) which aims to fully use image information to assist the point cloud completion process.
• We propose an interleaved attention mechanism to enable bidirectional information supplement across modalities, that is, enhancing point cloud representations with richer semantic detail and infusing image representations with supplementary geometric information.
• We propose a view-guided upsampling module introducing image information to enhance the generation of more detailed and accurate completion results. Specifically, this module utilizes image information as a query to re-

trieve relevant point cloud representations, enabling it to capture point clouds' local geometric structures.

2 RelatedWork

2.1 Single-modal Point Cloud Completion

PCN [Yuan et al., 2018] is introduced as the first end-to-end framework for point cloud completion. Building on PCN, TopNet [Tchapmi et al., 2019] proposes a tree-structured decoder that allows generation of completion results with arbitrary resolutions. Subsequently, Liu et al. [Liu et al., 2020] develop a morphing and sampling network that achieves completion results with smooth surfaces. PF-Net [Huang et al., 2020] marks a notable advance by focusing on generating the shape of missing regions using a multi-resolution encoder and a point pyramid decoder. To further refine missing shapes, PoinTr [Yu et al., 2021] pioneers the use of a full Transformer [Vaswani et al., 2017] architecture for point cloud completion, signaling the potential of Transformers for this task. Recently, SDT [Zhang et al., 2023a] introduces a skeleton-detail Transformer that emphasizes skeleton points in coarse completion results, while Seedformer [Zhou et al., 2022] applies an upsample Transformer to increase point density. More recently, PointAttN [Wang et al., 2024] presents a fully attention-based network architecture to enhance the completion of partial objects.

2.2 Multi-modal Point Cloud Completion

The development of devices such as cross-modal sensors has enabled multi-modal point cloud completion. As pioneers in this field, Zhang et al. [Zhang et al., 2021a] are the first to construct a multi-modal point cloud completion dataset incorporating both point cloud and image data and propose ViPC for multi-modal point cloud completion. A key challenge in multi-modal point cloud completion lies in effectively leveraging image information to aid point cloud completion. To address this challenge, CSDN [Zhu et al., 2024] introduces a novel IPAdaIN module, which facilitates the transfer of global shape information from images to point clouds. XMFNet [Aiello et al., 2022] first uses the cross-attention mechanism to fuse modality and point cloud representation, achieving excellent performance. CDPNet [Du et al., 2024a] and EGIINet [Xu et al., 2024] also use the cross-attention mechanism to fuse the two modalities. Specifically, the former uses a two-stage network to complete partial input. The latter focuses on the extraction and aligning of two modalities' features. Compared with ViPC and CSDN, XMFNet, CDPNet, and EGIINet, which use the cross-attention mechanism to fuse image and point cloud features, have better completion performance. However, the information transmission approach that utilizes the cross-attention mechanism to fuse point clouds and images is unidirectional, specifically transferring information from the image to the point cloud. This approach does not fully exploit the potential of the image data and overlooks the fact that the geometric information inherent in the partial point cloud could enhance the geometric representation of the image modality. IAET aims to explore the impact of bidirectional information transmission between point cloud and image on multi-modal point cloud completion.

3 Method

3.1 Overview

As shown in Figure 2, IAET mainly consists of three parts, i.e., token embedding, bidirectional token supplement, and coarse-fine decoding. During the token embedding stage, a partial point cloud P and a full image I are transformed into point cloud tokens FP and image tokens FI using PointNet++ [Qi et al., 2017b] and ResNet [He et al., 2016], respectively. In the bidirectional token supplement stage, interleaved attention mechanisms are applied to enhance the point cloud tokens and image tokens, resulting in supplemented point cloud tokens F 0 P and supplemented image tokens ˜ FI. Finally, in the coarse-fine decoding stage, the supplemented point cloud tokens F 0 P are processed through a coarse decoder, producing a coarse point cloud P0. The coarse point cloud P0, along with the supplemented point cloud tokens F 0 P and supplemented image tokens ˜ FI , is then refined using three view-guided upsampling modules to generate the completion outcome P3. Next, we will provide a detailed introduction to the two parts, i.e., bidirectional token supplement and coarse-fine decoding, along with their key components.

3.2 Bidirectional Token Supplement

Existing methods usually use a cross-attention mechanism to fuse the representation of point clouds and images. This unidirectional method of information transmission fails to fully utilize the rich information embedded in images and overlooks the geometric features of point clouds, which have the potential to enhance image representation. Unlike previous approaches, IAET employs a bidirectional token supplementation strategy during the fusion of the two modalities. This strategy facilitates the generation of a more informative point cloud representation while incorporating geometric information into the image representation. As illustrated in Figure 2, this process is implemented through multiple interleaved attention mechanisms.

Interleaved Attention Mechanism. To concisely illustrate the principle underlying the interleaved attention mechanism, we designate the inputs to this mechanism as point cloud tokens FP and image tokens FI, as depicted in Figure 3. Initially, the point cloud enhances the information content of the image. In this process, image tokens FI serve as the query, while point cloud tokens FP serve as the key and value. The corresponding mathematical formulation is as

where MLP(·) is the multi-layer perceptron. Then, the geometric information HP is obtained from the query Q1I , key K1P , and value V 1 P , and the formula is as

where Softmax(·) is the Softmax function and √d1 is the dimension of query Q1I , key K1P , and value V 1 P . Finally, the geometric information extracted from point cloud tokens is supplemented to the image tokens, and its formula is as

where ˜ FI is the supplemented image tokens. Subsequently, a similar operation is performed to facilitate the information transfer from supplemented image tokens to point cloud tokens. In this step, point cloud tokens FP are utilized as query, while supplemented image tokens ˜ FI serve as key and value. The corresponding mathematical formulation is as

After that, we use complete image tokens with geometric information to supplement incomplete point cloud tokens and its formula is as

where √d2 is the dimension of query Q2P , key K2 I , and value V 2 I . Finally, point cloud tokens are supplemented with extracted image information HI , and its formula is as

The output of an interleaved attention mechanism is supplemented point cloud tokens ˜ FP and supplemented image tokens ˜ FI .

In summary, the proposed bidirectional token supplement integrates several interleaved attention mechanisms to enable effective mutual supplementation of information between point clouds and images. Moreover, the enriched image representation is leveraged to guide the subsequent point generation process, ensuring the comprehensive utilization of image information.

3.3 Coarse-Fine Decoding

This section provides a detailed explanation of the final stage of IAET. In the coarse-fine decoding stage, IAET utilizes the supplemented point cloud tokens F 0 P to reconstruct a coarse point cloud P0 through deconvolution and MLPs. Subsequently, as shown in Figure 2, the coarse point cloud P0, supplemented point cloud tokens F 0 P , and supplemented image tokens ˜ FI are processed through three view-guided upsampling modules to produce the final completion result P3.

View-Guided Upsampling Module. To fully leverage image information, IAET embeds the supplemented image tokens into the proposed upsampling module. This enables the use of image data to guide the capture of fine-grained details within the point cloud tokens. As illustrated in Figure 4, to succinctly explain the principle of the view-guided upsampling module, we consider the inputs of the i-th module to be image tokens ˜ FI ∈ RM×C , point cloud tokens F iP ∈ RN×C , and a low-resolution point cloud Pi ∈ RN×3 . First, we use cosine similarity to calculate the similarity matrix A ∈ RN×M between each point cloud token and the image token. The formula is as

where fi ∈ R1×C and ˜ fj ∈ R1×C denote i-th point cloud token and j-th image token, respectively. Softmax(·) denotes the Softmax function. After that, the image tokens and point cloud tokens are processed through an MLP to generate the queries and keys, while the point cloud is processed via PointNet [Qi et al., 2017a] to derive the values. The formula is as

where Q ∈ RM×C , K ∈ RN×C , and V ∈ RN×C denote queries, keys, and values. Each query qi(i = 1, 2, . . . ,M) represents the shape features of a local region within the image, while each key ki(i = 1, 2, . . . ,N) represents the geometric features of a local region in the point cloud. Next, we employ the K-nearest neighbor (KNN) algorithm to identify the K nearest keys for each query, with the goal of retrieving relevant point cloud shapes corresponding to each image shape. Specifically, we use Euclidean distance to quantify the similarity between a query and a key, and the resulting values are passed through a Softmax function to compute the corresponding weights W ∈ RM×K×C . Then, point cloud features are aggregated based on the shape of each image and its formulation is as

where ˜ V ∈ RM×C contains M values and wij ∈ R1×C denote j-th weight produced by i-th image token. Finally, the point cloud tokens F i+1 P ∈ RµN×C output by the i-thmodule are produced, and its formulation as

And a high-resolution point cloud Pi+1 ∈ RµN×3 output by the i-th module is produced, and its formulation as

where tile(·) is the copy operation aiming to copy the coordinates of the point cloud Pi µ times.

In conclusion, the coarse-fine decoding framework of IAET consists of a coarse decoder and several view-guided upsampling modules to generate the completed point clouds. Specifically, image information is integrated into the proposed view-guided upsampling module, where image tokens serve as queries to guide the generation of offset values for high-resolution point clouds. This approach is designed to produce fine-grained completion results with sufficient local details.

KITTI. KITTI [Geiger et al., 2012] is a real-world dataset. KITTI used in this experiment was created by Wu et al. [Wu et al., 2025] Specifically, it contains 156 different car samples, each of which contains an incomplete point cloud and a complete image sizing 224×224. Since KITTI does not contain real labels, we train each method on the car category of a multi-modal point cloud benchmark [Wu et al., 2025] and test them on KITTI.

4.2 Experimental Settings

Metrics. Consistent with all previous multi-modal methods [Zhang et al., 2021a; Zhu et al., 2024; Aiello et al., 2022; Du et al., 2024a; Xu et al., 2024], we use CD and F-Score as evaluation metrics. CD is the same as formula (12). F-Score is a harmonic mean of precision and recall. Precision measures the number of points in the predicted point cloud that are close to the ground-truth point cloud, while recall quantifies the number of points in the ground-truth point cloud that are covered by the predicted point cloud. The formula of FScore is as

where ϵ > 0 is a threshold, P and ˆ P are predicted point cloud and ground-truth point cloud, and I(·) is the indicator function. Consistent with previous methods [Zhang et al., 2021a; Zhu et al., 2024; Aiello et al., 2022; Du et al., 2024a; Xu et al., 2024], we set ϵ = 0.01.

Implementation Details. All experiments are performed on a NVIDIA RTX A6000. We use the Adam [Kingma and Ba, 2015] optimizer with an initial learning rate of 0.001. The learning rate decayed every 20 epochs with a decay rate of 0.7. Our method converges after 200 epochs with a batch size of 64.

3.4 Loss

Consistent with previous methods, IAET uses chamfer distance (CD) [Fan et al., 2017] as the loss function, and its formula is as

where P and ˆ P are the completion result and ground truth, respectively. CD is commonly used to quantify the similarity between two point clouds. Since IAET employs a coarseto-fine decoding strategy, it generates completion results at multiple resolutions, namely P0, P1, P2, and P3. The ground truth corresponding to each resolution is utilized as the final loss function to train the model and specific formulation is as

where ˆ Pi(i = 0, 1, 2, 3) denotes the ground truth corresponding to the resolution of completion results Pi(i = 0, 1, 2, 3), which are generated using the farthest point sampling [Qi et al., 2017b] strategy.

4 Experiments and Analyses

4.1 Datasets

ShapeNet-ViPC. ShapeNet-ViPC [Zhang et al., 2021a] is a multi-modal dataset containing images and point clouds. It contains 13 categories and a total of 38,328 objects. Each object contains 24 views, so ShapeNet-ViPC contains 919,872 samples in total. Each sample consists of a partial point cloud, a complete point cloud, and an image sizing 138 × 138. Same as with previous methods [Zhang et al., 2021a; Zhu et al., 2024; Aiello et al., 2022; Xu et al., 2024], we conduct known category experiments on 8 categories and generalization experiments on the remaining categories.

4.3 Results on ShapeNet-ViPC

Results on Known Categories. In eight known categories, we compare the proposed IAET with all multi-modal point cloud completion methods, such as ViPC [Zhang et al., 2021a], CSDN [Zhu et al., 2024], XMFNet [Aiello et al., 2022], CDPNet [Du et al., 2024a], and EGIINet [Xu et al., 2024], and some representative single-modal point cloud completion methods, such as AtlasNet [Groueix et al., 2018], FoldingNet [Yang et al., 2018], PCN [Yuan et al., 2018], TopNet [Tchapmi et al., 2019], PF-Net [Tchapmi et al., 2019], MSN [Liu et al., 2020], GRNet [Xie et al., 2020], PoinTr [Yu et al., 2021], SDT [Zhang et al., 2023a], SeedFormer [Zhou et al., 2022], and PointAttN [Wang et al., 2024]. Table 1 presents the quantitative experimental results. Our method has the best result in all categories. Figure 5 presents the qualitative comparison of the experimental results. The first, second, third, fourth, fifth, and sixth rows correspond to the airplane, cabinet, car, lamp, sofa, and watercraft categories, respectively. In the airplane, cabinet, and sofa categories, only our method generates objects with smooth surfaces, whereas the completion results from other methods exhibit outlier points. In the car category, our method is the only one capable of generating smooth surfaces. In the lamp example, only our method can generate the missing lamp arm, while other methods seem unable to recover this shape. A similar observation can be made in the watercraft category. Despite the challenges in generating irregular shapes such as sails, only our method successfully reconstructs portions of the sail structures.

4.4 Generalization Ability Evaluation

Results on Unknown Categories of ShapeNet-ViPC. To assess the generalization capability of the proposed method on unseen categories, we follow the experimental setup of previous studies [Zhu et al., 2024; Xu et al., 2024], training on eight known categories and testing on four unknown categories. Specifically, we compare the performance of IAET against all existing multi-modal point cloud completion methods, such as ViPC [Zhang et al., 2021a], CSDN [Zhu et al., 2024], XMFNet [Aiello et al., 2022], CDPNet [Du et al., 2024a], and EGIINet [Xu et al., 2024], and some representative single-modal point cloud completion methods, such as PF-Net [Tchapmi et al., 2019], MSN [Liu et al., 2020], GRNet [Xie et al., 2020], PoinTr [Yu et al., 2021], SDT [Zhang et al., 2023a], SeedFormer [Zhou et al., 2022], and PointAttN [Wang et al., 2024]. Tables 2 and 3 present the quantitative results of the experiments. Our method achieves the best performance across four unknown categories, demonstrating its superior generalization ability. Figure 6 presents a qualitative comparison of the experimental results, with the first, second, and third rows corresponding to the monitor, speaker, and cellphone categories, respectively. In the monitor example, IAET generates a more compact screen along with detailed features of the bracket. For the speaker sample, only IAET is able to produce a circular global shape. Additionally, IAET successfully generates a complete cell phone shape in the cellphone sample.

Results on Real-Scene KITTI. To assess the effectiveness of IAET in real-world scenarios, we compare the qualitative results of all multi-modal point cloud completion methods, such as ViPC [Zhang et al., 2021a], CSDN [Zhu et al., 2024], XMFNet [Aiello et al., 2022], CDPNet [Du et al., 2024a], and EGIINet [Xu et al., 2024], on the KITTI dataset. As shown in Figure 7, each row corresponds to a real car sample. Only IAET generates completion results that closely resemble the vehicle shape depicted in the image.

4.5 Ablation Study

This section evaluates the effectiveness of IAET's core components from a quantitative perspective. As shown in Table 4, the first row corresponds to IAET without interleaved attention mechanisms, while the second row represents IAET without view-guided upsampling modules. The experimental results indicate that both components are critical to the overall performance of IAET.

5 Conclusion

We propose IAET, a novel framework for multi-modal point cloud completion. To fully leverage image information and enhance its contribution to point cloud completion, we propose an interleaved attention mechanism to facilitate the mutual supplement of point cloud and image features. Furthermore, we propose a view-guided upsampling module to utilize image information effectively, guiding the generation of high-quality completion results. Extensive qualitative and quantitative experiments demonstrate that IAET achieves state-of-the-art performance on multi-modal point cloud completion benchmarks.

In future work, we aim to enhance the performance of multi-modal point cloud completion with more modalities and large-scale model technologies.

Acknowledgments

This work is supported by the National Natural Science Foundation of China (No.U21A20473, 62032022, 62376141, 62176244) and the Key Technologies Program of Taihang Laboratory in Shanxi Province, China (THYF-JSZX24010600).