Top-I2P: Explore Open-Domain Image-to-Point Cloud Registration Using Topology Relationship

1 Introduction

Image-to-point cloud (I2P) registration is a fundamental task in computer vision [An et al., 2024a]. Given an image and a point cloud, it establishes the pixel-to-point correspondences to estimate the six degrees of freedom (6-DoF) camera poses [Wang et al., 2021]. I2P registration is critical for applications, such as visual localization, visual navigation, multisensor calibration, augmented reality (AR), object retrieval, and object pose estimation [Cheng et al., 2023b].

Deep learning is a crucial technique for I2P registration, as it effectively bridges the modality gap between 2D images and 3D point clouds. To improve the capability of neural networks for I2P registration, researchers tend to revise the neural networks in image registration (i.e., SuperGlue [Sarlin et al., 2020]) and point clouds registration (i.e., Geotrans [Qin et al., 2022]) for I2P registration [Pham et al., 2020].

Although current I2P registration approaches [Wang et al.,

2021; Li et al., 2023] have made progress, domain generalization on unseen scenes remains a major challenge for existing learning based I2P methods, especially in the open-domain I2P registration scenario. For a better illustration, we conducted a case study of 2D3D-MATR [Li et al., 2023] on the 7-Scenes [Glocker et al., 2013], as shown in Fig. 1. We found that the quality of correspondences of 2D3D-MATR is not accurate in unseen scenarios, making it unsafe for downstream applications (i.e., 6 DoF visual localization [Kim et al., 2023] and visual navigation [van Dijk et al., 2024]). This underscores the need to explore open-domain I2P registration.

In 2024, Wang et al. were the first to discuss open-domain I2P registration with presenting a neural network named as FreeReg [Wang et al., 2024]. Their method is computationally expensive (9 seconds per pair, ≥20 GB GPU memory) and lacks precision due to its use of a loose RMSE threshold (0.3m). These limitations make it unsuitable for real-time or high-accuracy applications. Thus, it is essential to explore a more effective and accurate I2P registration method in the open domain.

This paper addresses open-domain I2P registration from the perspective of topology relationship [Egenhofer, 1993]. We first explore the connection between topology relationship and I2P registration, and find that topology relationship reflect the sparse connection between pixels and points. It shows the salient potential in enhancing cross-modality feature interaction in the open domain. Based on this insight, we address the problem of learning and leveraging topology relationship between the heterogeneous 2D and 3D spaces, and present a registration framework that incorporates topology relationship reasoning and topology feature interaction.

Furthermore, to effectively build and leverage topology relationship, we propose a dedicated neural network Top-I2P that incorporates correction-based topology reasoning (CTR) and fast topology feature interaction (FTI) modules to balance feature capability and inference speed. Extensive experiments are conducted on the 7-Scenes [Glocker et al., 2013], RGBD-V2 [Lai et al., 2014], ScanNet [Dai et al., 2017], and self-collected datasets. Results demonstrate that the proposed Top-I2P achieves a superior inlier ratio (IR) and registration recall (RR) than state-of-the-art methods in open domain testing.

Our core contribution is rethinking pixel-to-point matching mechanism from a topology relationship viewpoint. To achieve this goal, we design an I2P registration network based on topology relationship, and it has better performance in the open domain.

2 RelatedWorks

We briefly overview the mainstream literature related to I2P registration and topology relationship.

2.1 Learning based I2P registration

I2P registration aims to build pixel-to-point correspondences ⟨q, p⟩ from image and point cloud where ⟨q, p⟩ is defined as:

where q ∈ R2 represents a 2D pixel in image I, and p ∈ R3 represents a 3D point in point cloud P. T is a transformation from the world coordinate system to the camera coordinate system. π(·) is a projection operator. As T is unknown in I2P registration task setting, ⟨q, p⟩ is determined in a featurematching manner:

where d(·, ·) is a normalized L2 distance and δ is a threshold. f 2D q and f 3D p are features of q and p. φ(·, ·) is the network of I2P registration. I ∈ RH×W×3 is an image, P ∈ RN×C is a point cloudwithC-dimensional features. f 2D q , f 3D p are queried from FI, FP at q, p. Current works focus on designing φ(·, ·).

In 2019, Feng et al. [Feng et al., 2019] was the first to develop a network to learn the descriptors of image patches and point cloud patches. Triplet loss [Schroff et al., 2015] is used to eliminate the adverse effects of a limited number of inliers. Li et al. presented Deep-I2P to interact with the global features from images and point clouds [Li and Lee, 2021]. Their approach predicts the overlap region in 3D space, which improves the stability of I2P registration. Wang et al. polished the image registration neural network [Dusmanu et al., 2019]

as P2-Net for I2P registration [Wang et al., 2021]. Building on work [Li and Lee, 2021], Ren et al. refined feature interaction scheme with transformers and established a neural network Corr-I2P [Ren et al., 2023], which improves visual localization accuracy in the outdoor scenes. Kim et al. established a neural network EP2P-Loc to deal with visual localization in the pre-built map [Kim et al., 2023]. Li et al. found that P2-Net [Wang et al., 2021] suffers from a few inliers, and they attempted to solve it by designing 2D3D-MATR [Li et al., 2023]. It takes 2D-3D patch matching as guidance for the accurate registration. Taking Corr-I2P [Ren et al., 2023] as baseline, Zhou et al. exploited a refined circle loss [Sun et al., 2020] and a differential perspective-n-point (PnP) loss to improve the geometric consistency of I2P correspondences [Zhou et al., 2023]. Wu et al. leveraged diffusion model in de-nosing pixel-to-point correspondences [Wu et al., 2024b].

Despite the advancements in I2P registration, there is limited research targeting open-domain scenarios. To address this gap, we propose Top-I2P to leverage topology relationship for robust I2P registration in a challenging environment.

2.2 Learning with topology relationship

Topology relationship are the fundamental relationships 1 of the space [Egenhofer, 1993]. As these relationships are invariant to topology transformation, many researchers exploit the topology relationship in computer vision and robotics.

Wang et al. developed a topology reasoning benchmark for high definition (HD) mapping task [Wang et al., 2023]. They developed an entity-specific similarity measure to evaluate the similarity of two topology graphs. Chen et al. studied semantic topology reasoning for the domain generalization task [Chen et al., 2022]. Domain invariance features are learnt by reasoning semantic topology graphs from crossdomain data. Cheng et al. used topology relationship for pedestrian re-identification [Cheng et al., 2023a]. A multicamera logical topology graph is built according to the cameras' localizations and image features. This topology graph is used in a graph neural network (GNN) to predict pedestrian detection results. In the autonomous driving scenario, Wu et al. used multilayer perceptron (MLP) to construct topology reasoning networks to predict lane-lane and lane-traffic topology relationship [Wu et al., 2024a]. Li et al. designed a graph contrastive learning scheme. Its highlight is to learn both graph-level and subgraph-level topology isomorphism knowledge [Li et al., 2024]. Towards 3D indoor scene understanding, Zhang et al. designed a graph to record the relation of place-place and place-object. This graph benefits various downstream tasks, such as place recognition and 3D reconstruction [Zhang et al., 2024].

In summary, topology relationship provide a lightweight, fundamental, and stable representation of the 3D space. This concept has garnered attention across various fields, not limited to computer vision, robotics, and autonomous driving. In this paper, we study topology relationship for open-domain I2P registration.

1 This term is used to describe the disjoint, meet, overlaps, covers, contains, equal, converBy, and inside relationships of two sets.

2.3 Discussions

The use of topology relationship in I2P registration remains an open question of interest. Li et al were the first to apply topology relationship by predicting the intersection relation of 2D and 3D patches [Li et al., 2023]. However, their approach employs a simplistic patch partitioning scheme, which relies on 2D square patches and 3D spherical patches. This limitation results in suboptimal performance, particularly in unseen scenes, as shown in Fig. 1. To address these shortcomings, we consider a patch with the general shape and use a visual fundamental model for patch partition, enabling a broader applicability and improved performance compared to the prior method [Li et al., 2023].

3 Open-Domain I2P Registration

We first analyze the bottleneck in open-domain I2P registration. Then, we illustrate why topology relationship can solve this bottleneck and provide a complete framework. After that, we implement this framework by designing Top-I2P.

3.1 Bottleneck analysis and topology relationship

Why is the current I2P registration method unstable (as seen in Fig. 1(b))? The instability arises primarily from the lack of pixel-level feature interaction. We illustrate this issue in depth. Without feature interaction, image or point cloud feature extractors operate independently (as shown in Fig. 2(a)), making the 2D and 3D features prone to overfitting on training data. To overcome the overfitting, one scheme is to incorporate a feature interaction module as shown in Fig. 2(b).

It facilitates message exchange between the cross-modality data, and minimizes the feature disparity between correspondences, aiding in domain-invariant features learning [Wu et al., 2021].

Pixel-level feature interaction is the ideal scheme to model the relationships of each pixel to each point. Pixel-level crossattention is its core component, and it constructs a matrix A ∈ RN×HW that records the weights between each pixel in I and each point in P (as shown in Fig. 2(c), (d)).

However, A becomes prohibitively large. For a 640 × 480 image and a point cloud with 105 points, A contains 1010 elements, which is impractical to store and compute. Hence, pixel-level feature interaction represents a major bottleneck for open-domain I2P registration.

To address this bottleneck, we find that topology relationship between 2D and 3D spaces reflect the sparsity of A. For one pixel q and one point p, if their neighboring regions have no interaction (i.e., N2D(q)∩π(T◦N3D(p)) = ∅), there is no need to interact features between q and p. From Eq. (5) in Sec. 3.2, we construct A only with nearly 104 elements, 106 times smaller than the original scheme. Hence, topology relationship significantly reduce the computational burden of pixel-level feature interaction.

3.2 A framework with topology relationship

As topology relationship show significant potential in feature interaction, we attempt to leverage topology relationship and derive a lightweight and efficient framework for opendomain I2P registration, where an overview of the framework is shown in Fig. 3. Its sub-modules are discussed as follows.

Topology graphs construction. Using the segmentation anything model (SAM) in image and point cloud [Kirillov et al., 2023; Zhou et al., 2024], I and P are decomposed as arbitrary shapes without overlaps:

where A and B are the patch numbers. Pi and Ij are visualized with different colors in Fig. 3. Complete graphs GP , GI are constructed from {Pi}Ai=1 and {Ij}Bj=1 where Pi and Ij are graph nodes. GI and GP are named as 2D and 3D graphs.

Besides, we discuss the reason why 2D and 3D patches are obtained by SAM instead of the regular shapes (i.e., 2D square and 3D sphere) or 3D superpoint and 2D superpixel. First, unlike regular shapes, patches segmented by SAM have a semantic correlation, helpful to topology relationship reasoning. Second, SAM is faster than the superpoint and superpixel algorithms. Third, as pixel distance is not aligned with metric distance, regular shapes suffer from the scale selection [Li et al., 2023], while patches segmented by SAM do not.

Topology relationship reasoning. This step is to learn a sparse A by topology relationship reasoning on GI and GP . As GI and GP are heterogeneous graphs on 2D and 3D spaces, learning A is a challenging task. We simplify this task in such an approximate way. For p ∈ Pi and q ∈ Ij, if π(Pi) ∩ Ij = ∅, f 2D q has no contribution to f 3D p . Thus, we can approximately construct a sparse A as:

where 1 and 0 are matrices with full of ones or zeros. A can be further collapsed into a small-sized boolean matrix Atop ∈ BA×B , Atop,ij = I(π(Pi) ∩ Ij) where I(·) is an indicator function. After figuring out the aim of topology relationship reasoning, this procedure can be represented as:

where ϕtop(·, ·) is a self-defined topology relationship reasoning network. ˆAtop ∈ [0, 1]A×B describes confidence score of whether π(Pi) ∩ Ij .

Topology feature interaction. For the open-domain I2P registration, topology feature interaction is a crucial step to eliminate cross modality difference from f 2D q and f 3D p with the guidance of GI , GP , and ˆAtop. This step contains (i) 3D-to-2D message passing and (ii) 2D-to-3D message passing.

At first, 3D-to-2D message passing is defined in a moving average manner (suppose q ∈ Ij):

where α is a smooth coefficient. ¯f 3D i is the average feature in region Pi. Thus, 3D-to-2D message passing is to update f 2D q with average 3D features in regions that π(Pi) ∩ Ij ̸= ∅.

Similar to Eqs. (7) and (8), 2D-to-3D message passing is defined as:

Finally, with the updated f 2D q and f 3D p , whether ⟨q, p⟩ is a pixel-to-point correspondence can be determined via Eq. (2).

More discussions. We further discuss two issues of the proposed method. First, we only utilize a single topology relationship (i.e., intersection) in this paper. Actually, the general topology relationship has multiple relationships. 4 intersection model (4IM) [Egenhofer, 1993] is a traditional model to represent the multiple topology relations of two 2D sets A and B:

where A◦ and ∂A are interior and boundary sets of A. However, boundary-to-boundary and boundary-to-interior relation is difficult to predict from π(Pi) and Ij. An et al. once used boundary-to-interior relationship for I2P registration [An et al., 2024b], but they needed 3D object detector to understand 3D scene. Their method cannot be utilized in the general open scene. So, we simplify the 4IMmodel and only consider the intersection of two sets. It increases the prediction accuracy of topology reasoning in Eq. (6).

Then, we analyze why the proposed framework is robust to the open domain. Actually, topology relationship is a sceneindependent feature, because it is only variant to the shapes relationships and robust to the unseen scenes. Thus, the topology relationship can be used as an important cue to the opendomain I2P registration.

3.3 Framework implementation

Sec. 3.2 provides an abstracted framework to leverage the topology relationship. To implement this framework in an actual scene, we propose a neural network Top-I2P with detailed computation procedures. Pipeline is provided in Fig. 4.

Topology graphs. We begin by discussing the construction of graphs GP and GI. 2D and 3D backbones are used to extract features from I and P. Following 2D3D-MATR [Li et al., 2023], ResNet [He et al., 2016] and KPConv [Thomas et al., 2019] are used to construct 2D and 3D backbones. Backbone features of I and P are extracted as FBI and FBP , respectively. In the meanwhile, using 2D and 3D SAM, I and P are segmented into non-overlapping regions, as shown in Eq. (4). For each point cloud patch Pi, we compute the average coordinate ¯pi and the average feature ¯f 3D i ; For each image patch Ij , we compute the average coordinate ¯qj and average feature ¯f 2D j . ¯f 3D i (i = 1,...,A) are vertices in GP , its adjacent matrix A3D is constructed by exp(-∥¯pi-¯pj∥22 ). In the same way, ¯f 2D j (j = 1,...,B) are vertices in GI, its adjacent matrix A2D is constructed by exp(-∥¯qi - ¯qj∥22 ). Due to the setting of SAM, the patch number is no more than 128, so that the graph construction step can be done in real-time.

Correction-based topology reasoning (CTR). In the next, we illustrate the structure of ϕtop(GI , GP ). Due to the modality difference of I and P, GI and GP are heterogeneous graphs, which brings the challenge of ˆAtop prediction. To deal with this issue, we design a simple yet effective CTR module in Fig. 4. Graph convolution network (GCN) layers [Kipf and Welling, 2017] are used to extract topology features x = GCN2d(GI ) ∈ RB×c and y = GCN3d(GP ) ∈ RA×c . To eliminate the modality difference, we attempt to correct features by estimating feature shift ∆x and ∆y:

where mlp 2d and mlp 3d are two MLP layers. Then, topology features are updated as x′ = x + ∆x and y′ = y + ∆y, respectively. ˆAtop is computed using a normalized cosine distance from x′ to y′ . We find that most elements in the boolean matrix Atop are zero. So, we exploit a differentiable Sinkhorn layer [Sinkhorn, 1967] to regularize ˆAtop. And loss in CTR module is designed as [Sarlin et al., 2020]:

where mask is the indices in Atop with ones, Nmask is the number of elements in Atop that is one.

Fast topology feature interaction (FTI). The topology feature interaction in Eqs. (7) and (9) cannot meet the fast I2P registration demand, although it greatly reduced the computation burden. The reason why Eqs. (7) and (9) are inefficient lies in pixel-wise and point-wise computation. To speed up this procedure, we design the FTI module. The core idea behind FTI is to replace f 2D q and f 3D p as ¯f 2D j and ¯f 3D i in Eqs. (7) and (9). Then, feature interaction is rewritten in a matrix operation manner with MLP layers:

where ˆAtop,col and ˆAtop,row are the column and row normalized matrices from ˆAtop. It means that FTI focuses on interaction with average features. To improve the efficiency of feature interaction, FTR can be stacked in Top-I2P where the stack number is 3. After feature interaction, we update f 2D q by adding shift (¯f 2D j,new - ¯f 2D j ), update f 3D p by adding shift (¯f 3D i,new - ¯f 3D i ).

Training scheme. Single-stage training is unstable primarily due to the inaccurate predictions of topological relationships. To address this issue, we utilize a two-stage training scheme. First, we train backbone networks without CTR and FTI modules. The loss function is based on the I2P registration loss as defined in 2D3D-MATR [Li et al., 2023]. Second, based on the pre-trained backbone networks, we begin to learn the topological relationship prediction task and train both the CTR and FTI modules. It is achieved by training the whole Top-I2P architecture by adding LTopo loss defined in Eq. (13).

4 Experiments and Discussions

4.1 Configurations

Since limited research exists on open-domain I2P registration, we configure this task setting independently. First, we evaluate model generalization ability on the 7-Scenes dataset [Glocker et al., 2013]. One scene is chosen for training, while the remaining six scenes are used for testing. Image and point cloud pairs are created from the RGB-D data. Then, we evaluate the generalization ability on the RGBD v2 [Lai et al., 2014], ScanNet [Dai et al., 2017], and self-collected datasets.

The self-collected dataset was collected using an Intel RealSense D351 depth camera. We obtained the GT pose between images and point clouds based on an RGB-D-based 3D reconstruction toolbox 2 . In this case, all compared methods are trained on the 7-Scenes dataset [Glocker et al., 2013]. From Figs. 1 and 5, scenes in the different datasets are largely distinct, making them suitable for open-domain evaluation. Point clouds include RGB features.

To evaluate I2P registration performance, IR and RR are used as the main metrics. The root mean square error (RMSE) threshold is set to 0.025m and 0.1m for the strict I2P registration performance testing, while other metric thresholds follow the defaults in the literature [Li et al., 2023]. Subsequently, we discuss the details of Top-I2P. α is set to 0.86.

2 www.open3d.org

The stack number of the message passing submodule in FTI is 2. The first training stage consists of 5 epochs, while the second stage consists of 20 epochs. To generate the ground truth (GT) Atop, we set an intersection-over-union (IoU) threshold γIoU = 0.1. Atop,ij is set as 1 only if IoU(π(Pi) ∩ Ij) ≥ γIoU , and this procedure filters out several unimportant topology relations. The learning rate and optimizer of Top-I2P are the same in the literature [Li et al., 2023].

4.2 Comparison results

Current I2P registration methods, P2-Net [Wang et al., 2021] and 2D3D-MATR [Li et al., 2023], are used for comparison with Top-I2P. The notation A → B indicates that the model is trained on scene A but tested on scene B. Results on the 7-Scenes dataset [Glocker et al., 2013] are provided in Table 1, and results on RGBD-V2 [Lai et al., 2014], ScanNet [Dai et al., 2017], and self-collected datasets are provided in Table 2.

First, P2-Net suffers from significant overfitting to the training scene due to the absence of feature interaction modules. This limitation causes its backbone networks to generalize poorly to unseen scenes. Second, while 2D3D-MATR incorporates a feature interaction module, it operates only on patch-level features, limiting its ability to learn fine-grained pixel-level cross-modality interactions. This results in 2D3DMATR being superior to P2-Net while weaker than TopI2P. Third, Top-I2P achieves a more accurate feature interaction through its topology-based approach, which results in a higher number of inliers than 2D3D-MATR across different datasets (Fig. 6). The RR of Top-I2P is consistently and significantly better than that of 2D3D-MATR on all datasets. It suggests that topology relationship are helpful in improving the localization quality of pixel-to-point correspondences.

Furthermore, we compare the proposed Top-I2P with LCD [Pham et al., 2020], SuperGlue (Glue) [Sarlin et al., 2020], and FreeReg [Wang et al., 2024] on ScanNet [Dai et al., 2017]. The results of these methods are obtained from FreeReg [Wang et al., 2024], where the RMSE threshold is 0.3m. The results are provided in Table 3. Even with a stricter RMSE threshold, Top-I2P achieves an RR 12.1% higher than FreeReg [Wang et al., 2024]. It means that the generalization ability of Top-I2P is higher than that of FreeReg. Additionally, we compare Top-I2P with FCGF [Choy et al., 2019], FreeReg [Wang et al., 2024], and Diff-Reg [Wu et al., 2024b] on the RGBD-V2 dataset [Lai et al., 2014], where all models are trained using the train and test splits in literature [Li et al., 2023]. The results are provided in Table 4, and they show that Top-I2P outperforms state-of-the-art methods. Therefore, the above experiments suggest that Top-I2P achieves the best performance in the open-domain I2P registration task.

4.3 Ablation studies

In this section, we study the effect of the CTR and FTI modules in Top-I2P. First, the ablation study of the CTR module is provided in Table 5. [m, n] denotes two MLP neural network layers with m and n channels. Empty indicates that the model is the baseline [Li et al., 2023] if true. Sinkhorn refers to the usage of the Sinkhorn layer [Sinkhorn, 1967]. From Table 5, the Sinkhorn layer plays a dominant role in the CTR module, because it determines the accuracy of topology relationship prediction. This accuracy directly impacts the overall effectiveness of the CTR module. Thus, the performance of CTR depends on the prediction accuracy of π(Pi) ∩ Ij .

Second, the ablation study of the FTI module is shown in Table 6. NS is the stack number of the message passing submodule in FTI, as shown in Fig. 4. We fix NS and search for the optimal α in the range [0.80, 0.88]. The performance is relatively insensitive to α, as the FTI module primarily interacts with average features. We keep the optimal α and search for the best NS. When the stack number NS is excessively large, the average features in different patches tend to be similar, which reduces the discriminative ability of cross-modality features. The best α and NS are 0.86 and 2, respectively.

Third, we investigate the performance of the combination of the CTR and FTI modules. Results are provided in Table 7. Removing CTR (and replacing it with MLPs) causes a significant drop in topology prediction accuracy (Pred. Acc.), leading to degraded IR and RR. Similarly, without FTI, the model struggles to exploit topological cues for cross-modality learning. These results validate the necessity of both CTR and FTI.

Fourth, we evaluate the inference time and GPU memory usage performance of Top-I2P. Actually, there are two schemes to accelerate Top-I2P. First, we can skip 3D SAM during model inference, because 3D point cloud segmentation can be pre-computed in some downstream tasks related to I2P registration (i.e., visual localization). Second, we can use a lightweight 2D SAM during model inference. It can further reduce the runtime and memory usage. The results of the accelerated Top-I2P are shown in Table 8. It is found that the frame per second (FPS) of MATR [Feng et al., 2019] and the proposed Top-I2P are nearly the same.

Besides, we also analyze the failure case of the proposed I2P registration method. Although Top-I2P achieves the best I2P registration performance in the open-domain scenarios, Top-I2P still faces difficulties when input images lack texture or the point clouds have low resolution, as shown in Fig. 7. In such cases, even though the IR is higher than 2D3D-MATR, the 2D-3D correspondences become sparse. Thus, in future work, we attempt to refine the architecture to achieve accurate and real-time I2P registration performance.

5 Conclusions

In this paper, we addressed the challenge of open-domain I2P registration. At first, inspired by the potential connection between topology relationship and cross-modality feature interaction, we developed an I2P registration framework using topology relationship. After that, to efficiently construct and leverage topology relationship from heterogeneous 2D and 3D spaces, we designed Top-I2P with CTR and FTI modules. Extensive experiments on multiple datasets have demonstrated the superior performance of Top-I2P.

Limitation and future work. Top-I2P does not fully utilize other topology relationship, such as A◦ ∩ ∂B and ∂A ∩ ∂B in Eq. (11). We plan to refine this framework in the future.

Acknowledgments

This work is partially supported by the National Key R&D Program of China (Grant ID: 2024YFC3015302) and China Postdoctoral Science Foundation (Grant ID: 2024M761014).

Contribution Statement

Pei An and Jiaqi Yang have equal contribution in this work. You Yang is the corresponding author in this work.