Understanding Matters: Semantic-Structural Determined Visual Relocalization for Large Scenes

1 Introduction

Visual relocalization aims to determine the position and orientation of the camera within a known scene by visual cues. This task holds significant promise in various fields, such as robot navigation, augmented reality, and autonomous driving. In particular, image-based Scene Coordinate Regression (SCR) encodes the scene in the weight of neural networks and supports end-to-end inference, showing great potential for a wide range of applications.

With the same motivation, image retrieval [Camposeco et al., 2019a; Revaud et al., 2019] and pose regression [Brahmbhatt et al., 2018; Kendall and Cipolla, 2017] take the entire image as input and extract image-level features for matching [Ruan et al., 2023]. However, global features often lack local structural details, leading to insufficient relocalization accuracy. In contrast, methods based on feature matching [Sarlin et al., 2019; Sarlin et al., 2020a; Sattler et al., 2016a], which build explicit 3D geometry using structure-from-motion (SfM) [Schonberger and Frahm, 2016], achieve excellent performance. However, they require significant storage and extended processing time, ranging from several minutes to hours. Furthermore, they introduce privacy risks. With recent advances in deep learning, SCR

researches [Brachmann et al., 2023; Shotton et al., 2013; Wang et al., 2024] employ models to regress the 3D scene coordinates corresponding to each pixel in an image, subsequently estimating the camera pose using the Perspective-nPoint (PnP) algorithm. Consequently, only the raw query image is required during inference, eliminating the need to access previously trained data.

Due to constrained receptive fields, SCR methods struggle to differentiate similar local features in large-scale scenes, leading to decreased performance. Recent SCR methods focus on improving performance in such scenarios. ACE [Brachmann et al., 2023] trains four models (Poker) for partitioned large-scale scenes. HSC [Li et al., 2020] partitions the scene into smaller parts with hierarchical labels based on structural information, guiding the coarse-to-fine relocalization process. GLACE [Wang et al., 2024] encodes a single image into a global feature to distinguish similar features across different images. However, the methods above, which primarily focus on structural information, partition large scenes into small subscenes solely based on spatial distance, without considering scene understanding. This results in over-partitioned elements and high feature variances within subscenes. Semantic information plays a crucial role in scene understanding, guiding more effective scene partitioning.

To address the aforementioned challenges, we propose a Semantic-Structural Determined Visual Relocalization method. Specifically, semantic-structural scene partition learning is applied to divide the large scene into smaller subscenes, maintaining semantic consistency and structural continuity within each subscene through label assignments. Then, accelerated partition localization is performed with sampling-based learning. To better leverage the semantic and structural information in the labels and to avoid errors in relocalization caused by ambiguous areas, e.g. repetitive textures and flat regions, we put forward partition-determined pose refinement. Partition labels are encoded into embeddings and integrated with local features next, creating partition-aware features, and guiding the further discriminable point selection. Our contribution can be summarized as follows:

1. We propose a Semantic-Structural Determined Visual Relocalization method for scene coordinate regression. It partitions the scene into small parts with labels, ensuring semantic consistency and structural continuity, guiding further accelerated partition localization with sampling-based learning.
2. ii) We design a partition-determined pose refinement method to reduce the inaccuracy of pose estimation, which selects determinable points based on discriminability scores derived from partition-aware features.
3. iii) Our method achieves the state-of-the-art performance on Cambridge Landmark in a short training time when depth maps are provided, and can still achieve the competitive performance without depth maps.

2 RelatedWork

Typical methods of visual relocalization can be divided into four categories, e.g. image retrieval, pose regression, feature matching, and scene coordinate regression.

Image Retrieval. A set of mapping images with known poses and global descriptors serves as the known environment in retrieval-based methods. Given a query image, these methods search for the most similar image in the database using global descriptor matching [Arandjelovic et al., 2016; Revaud et al., 2019; Torii et al., 2015], and approximate the pose of the query image based on the top retrieved images [Camposeco et al., 2019a]. Since image-level descriptors are used, retrieval-based methods can scale to large scenes. Recently, some researches have combined retrieval methods with structure-based techniques and relative pose estimation.

Pose Regression. Instead of descriptor matching, pose regression [Brahmbhatt et al., 2018; Kendall and Cipolla, 2017; Kendall et al., 2015; Shavit et al., 2021; Revaud et al., 2019; Winkelbauer et al., 2021] uses neural networks to predict the absolute pose of the query image directly, namely absolute pose regression (APR). Some methods predict the relative pose between the query image and a mapping image. PoseNet [Kendall et al., 2015] regresses the pose from a query image with a neural network, but it takes several hours to train the model. Several variants of it have introduced improvements, ranging from the loss function [Kendall and Cipolla, 2017] to the network architecture [Walch et al., 2017; Wang et al., 2020; Shavit et al., 2021]. However, recent researches indicate that pose regression methods are more similar to image retrieval than to feature matching, resulting in their performance being surpassed by feature matching.

Feature Matching (FM). Methods based on feature matching [Sarlin et al., 2019; Sarlin et al., 2020a; Sattler et al., 2016a], also known as structure-based, perform well for visual relocalization. They establish 2D-3D correspondences between pixels in query images and 3D coordinates in the scene, usually employing descriptor matching. Thus, they need to reconstruct the known environment as a 3D point cloud through Structure-from-Motion (SfM), so that points can have several feature descriptors from different viewpoints. To scale to large scenes, commonly they need a large amount of storage and a long time for feature extraction. Some recent researches work on handling the storage problems. For instance, techniques such as storing fewer descriptors [Sattler et al., 2016b], compressing descriptors [Yang et al., 2022], and combining image retrieval with descriptor matching [Arandjelovic et al., 2016; Torii et al., 2015] are commonly used. Several approaches try to directly match the descriptors with the point cloud or mesh [Zhou et al., 2022a]. Given all of the above, large storage requirements and long descriptor generation time are unavoidable.

Scene Coordinate Regression. This family of methods regresses 3D coordinates for the 2D pixels in the query image instead of learning the entire pipeline. These methods do not rely on traditional feature detection and descriptor matching. Instead, they implicitly encode the scene information within a neural network [Brachmann et al., 2023; Brachmann et al., 2017; Brachmann and Rother, 2018; Brachmann and Rother, 2019; Brachmann and Rother, 2021; Cavallari et al., 2019a; Dong et al., 2022; Li et al., 2020] or random forests [Brachmann et al., 2016; Cavallari et al., 2017; Cavallari et al., 2019b; Shotton et al., 2013]. By doing so, they leverage learned representations of the environment to predict the 3D coordinates directly from the input image, without the need for explicit feature matching or descriptor-based techniques. These approaches can be considered privacy-preserving, as the implicit map cannot be regenerated without access to the training images of the scene. However, although SCR-based methods achieve high accuracy in small-scale scenes, their performance tends to degrade in large-scale environments. Existing methods, such as DSAC* [Brachmann and Rother, 2021], achieve state-of-theart accuracy but require several hours of model training. Recently, ACE [Brachmann et al., 2023] proposed a model that achieved high accuracy in a short time of training. However, its performance deteriorates in large-scale environments, requiring the assembly of more separate models for effective learning of such scenes.

3 Method

Our Semantic-Structural Determined Visual Relocalization method comprises two components. Semantic-Structural Partition Learning partitions the large scene into smaller subscenes, with labels assigned to each. This is followed by accelerated partition localization with sampling-based learning. The partition-determined pose refinement enhances features with partition labels and selects discriminable points.

3.1 Semantic-Structural Partition Learning

In this section, the scene is partitioned into small parts preserving semantic consistency and structural continuity, which are assigned labels. Subsequently, a label classifier is trained on the sampled feature set.

Semantic-Structural Label Assignment. Recent works have shown that in large scenes with many similar local features, it is difficult for SCR to form an effective mapping from features to 3D coordinates. Previous studies, such as HSC [Li et al., 2020], in Fig. 2 (a-I), employ scene partitioning strategies that disrupt the semantic information of the scene, leading to over-partitioned elements and large feature variances in each subscene. Apparently, a scene partitioning strategy that groups features with similar semantic information into the same category can maintain semantic consistency within the feature set. We propose our scene partitioning and point labeling strategy next. Semantic features are extracted from the images with the backbone.

where Pi ∈ RCI×HP×WP is a patch of image, and FB is the backbone that extracts semantic features fi ∈ RCf from image patch Pi with network parameters wB.

To ensure semantic consistency, we perform semantic-level clustering of the features. Large-scale scenes often contain many similar local patches. The limited ability of the network to differentiate features from these patches constraints its performance. Since these patches may be far apart in the scene, incorrect matching of similar patches can lead to significant errors, emphasizing the necessity of structural continuity. We partition the large scene into smaller subscenes, ensuring both semantic consistency and structural continuity within each subscene. We take the semantic feature fi and the corresponding 3D coordinates yi of the point as input and output a label vector that determines the cluster assignment.

Recent work [Li et al., 2020] proposes a hierarchical label assignment approach. During the inference stage, although the subscene label is correctly predicted, the predicted region label has a decisive impact on the final result. Our strategy assigns two labels to each point simultaneously, ensuring the independence of the two partitions. This decoupling approach reduces the dependence on a single label and enhances the robustness of the results. The process is shown as follows:

where yi is the 3D coordinates corresponding to the feature, k1 ∈ {1, . . . , nk1 } represents the index of semantic clustering, k2 ∈ {1, . . . , nk2 } represents the index of structural clustering, µ1(k1) is the k1-th semantic cluster center and µ2(k2) is the k2-th structural cluster center, and (c1, c2)i represents a 2-dimensional label vector.

Standard k-means algorithms require significant time to cluster high-dimensional features. We use mini-batch kmeans for semantic consistency partitioning on features and standard k-means for structural continuity partitioning on coordinates. Here is the process of mini-batch k-means:

where fB is a batch of semantic features. We randomly select nk1 initial cluster centers µ1(1), µ1(2), . . . , µ1(nk1 ). Then we iteratively perform a random selection of small subsets to assign each feature to the nearest feature centroid and update the centroids based on the average of the features in the minibatch. This process repeats for several iterations or until the centroids stabilize. The mini-batch k-means uses only a minibatch of data to incrementally update the centroids, while the centroids of standard k-means are recalculated by taking the

where Pstr(k2) = {yi|c2(i) = k2} denotes the 3D coordinate set of points with the structural label k2. For the category with semantic label k1 and structural label k2, we compute the average of the 3D coordinates.

Accelerated Partition Localization with Sampling-based Learning. The features surrounding a point are more likely to belong to the same category as the point itself, both at the structural and semantic levels. Therefore, it is possible to predict the labels of other local points on the basis of the label of a given point. Furthermore, partitioning all points in the scene incurs significant time overhead. In HSC [Li et al., 2020], labels are assigned to every pixel in all images offline, resulting in poor transferability and high computational cost. Inspired by [Brachmann et al., 2023], we perform a sampling operation on the features of an image.

where N is the number of training images, S represents the sampling process, and B is the feature extractor.

In addition, we train a small classifier on the sampled feature set F to predict the label of a feature: \hat {o}_i=\mathcal {F}_L(fi;w_L),\ \text {and}\ fi\in F \label {equ7}, (7)

where FL is a small MLP head. And it is trained with the cross-entropy loss:

where oi denotes the one-hot label of pixel i. ˆ oi denotes the corresponding label probabilities.

3.2 Partition-determined Pose Refinement

In this section, we present the process of pose refinement, which includes partition-aware feature enhancement and discriminable point selection for pose refinement.

Partition-aware Feature Enhancement. The quality of semantic features significantly affects the accuracy of SCR models [Nguyen et al., 2024]. Although we assign semantic and structural labels to each feature in Equation 2, the clusters formed of semantic features with the same label are still not compact enough, as shown in Fig. 2 (b-II). As a result, semantic features from different categories remain relatively close in the feature space. The overlap reduces their separability, thereby affecting discriminability. To mitigate this, we introduce a feature fusion module that encodes labels into structure-semantic partition-aware embeddings and fuse them with the semantic features fi extracted from the image, to promote intra-class compactness and inter-class separation:

where ϕ represents the fusion function and ˆ fi represents the feature fi fused with the embedding. Specifically, ϕ is a small network module composed of MLP layers.

Discriminable Points Selection for Pose Refinement. Some fused features extracted from repetitive textures or flat regions still struggle to be mapped. We present a pixel-wise discriminability score ˆ si, which is generated by our prediction head, along with the predicted 3D coordinate ˆ yi:

where LH is a regression head. We define si = 1 + eˆ s to ensure that it is always greater than 1. The fused feature with a higher discriminability score is considered to have effective scene information, leading to an accurate prediction result. Due to the complexity of large-scale scenes, we find that using si to represent the discriminability of the mapping from features to 3D coordinates results in a large overall error. Therefore, we combine si with the reprojection error ˆ ℓ to ensure that si represents the discriminability of a point along the line connecting the imaging point and the actual pixel coordinates of that point: \hat {\ell }=||x_i-Kh^{-1}\hat {y}_i||_1, \label {equ11} (11)

where h is the pose of the camera and K is the camera calibrationmatrix. ˆ yi denotes the predicted 3D scene coordinates and xi denotes the ground-truth pixel coordinates.

where ˆ ℓi is the reprojection loss of pixel i, and α is a hyperparameter that controls the regularization term.

The partial derivatives of Equation 12 can be computed with respect to its two parameters:

where si is always greater than one to ensure that model training continually reduces ˆ ℓi and to prevent the gradient from vanishing. And the Equation 13 is positive only when si > α ℓi . It reaches its minimum when si = α ℓi . In other words, after the training process is complete, for the input features, the output should have a large si and a small ˆ ℓi or a small si and a large ˆ ℓi. Therefore, during the prediction process, si can be used to assess the reliability of the feature fi. However, since si varies depending on the magnitude of the loss, it cannot directly reflect the reliability of the prediction. For features with very small si, the probability of high reliability is very low. Therefore, an appropriate threshold needs to be defined to filter out points with low discriminability. This helps improve accuracy. In the regression head, the prediction layers of ˆ yi and ˆ si share almost all parameters.

3.3 Optimization

Combined with Equation 12, the reprojection loss is calculated as follows:

where P is a batch of points from the sampled point set. However, reprojection loss, which is based solely on observations of the same landmark from multiple viewpoints, is insufficient for the model to encode the map information within its parameters. Euclidean loss, which represents the Euclidean distance between predicted and ground-truth 3D coordinates, imposes strong constraints for coordinate prediction:

In reprojection loss, the 2D distances of 3D points are calculated after they are projected onto the camera plane, while the discrepancy between the predicted and true 3D coordinates is minimized by Euclidean loss. The combination of these two types of losses eliminates the need for implicit triangulation. The total loss function is given by:

The correctness of the label prediction is crucial to localization performance, and thus a large value γ should be set. We set β = 0.1 to ensure that the model does not overly focus on fitting the ground-truth 3D coordinates.

4 Experiment

4.1 Datasets

We conduct our experiments on 7Scenes [Shotton et al., 2013], 12Scenes [Valentin et al., 2016], Cambridge Landmarks [Kendall et al., 2015] and Wayspots [Brachmann et al., 2023]. Experiments are conducted on the first three datasets with RGB and RGB-D. On Wayspots, experiments are conducted with RGB. 7Scenes and 12Scenes are indoor relocalization datasets. They contain 7 and 12 indoor scenes respectively. For each scene, the dataset contains a set of RGB images along with their camera poses and depth maps.

Cambridge is an outdoor relocalization dataset. It contains 5 outdoor scenes. For each scene, the scale is large, reaching up to hundreds of meters. It also includes factors such as lighting, pedestrians, cars, and other influences, consistent with real-world conditions. The dataset is captured using a handheld device, resulting in complex motion trajectories.

Wayspots is a dataset with 10 small outdoor scenes, curated from a publicly available corpus of phone scans. The ground truth poses are reconstructed using SfM, and the original phone trajectories are registered to the SfM poses. Our method outperforms ACE and DSAC* without depth maps.

4.2 Implementation

Architecture. We implement our method in PyTorch, using the backbone of ACE [Brachmann et al., 2023] as the feature extractor. Based on the ACE prediction head, we add a label prediction head and a feature fusion module. Semantic labels are generated using mini-batch k-means, while structural labels are generated using k-means. We use a 6-layer MLP

head as the label classifier. It takes the features generated by the feature extractor as input and outputs the labels for each feature. The feature fusion module consists of several MLP layers for feature alignment, as well as a conditioning layer [Li et al., 2020] that performs linear modulation at each position. The image features and label embeddings are simply added together to form the partition-aware features.

Training. To accelerate the training process, we implement mini-batch k-means with CUDA. The batch size is set to 400K, and the maximum number of iterations is 200. We cluster 64 categories at the structural level and 2 classes at the semantic level. To better adapt to the scale of the scene, we define α in Equation 9 as 20, which yields relatively good results for both large and small scenes. Additionally, we set β in Equation 15 to 0.1. γ is defined as 100 to encourage the network to focus on label prediction accuracy. As described in Section 3.2, the 0.51 percentile of all discriminability scores is used as the threshold for feature filtering. We compare mapping times of ACE, GLACE, DSAC* and ours on NVIDIA GeForce RTX 2080 Ti. The results of other methods come from ACE [Brachmann et al., 2023].

4.3 Analysis

7 Scenes and 12 Scenes. As shown in Tab. 1, our approach achieves higher accuracy compared to ACE [Brachmann et al., 2023] and GLACE [Wang et al., 2024]. Mappings with a time of less than a quarter of an hour are labeled green, while ones smaller than 15 MB are labeled green. Our method achieves the best performance of these methods on 12Scenes, regardless of the existence of depth maps. Additionally, on 7Scenes, our method demonstrates a significant time advantage over DSAC*, while also achieving competitive results. Cambridge Landmarks. The advantages of our method are fully demonstrated in large-scale scenes. As shown in Tab. 2, our method outperforms state-of-the-art SCR methods [Brachmann et al., 2023; Wang et al., 2024] and is competitive with FMmethods. It is worth noting that effectively utilizing depth maps in large-scale scenes presents significant challenges, with DSAC* achieving poor performance when depth maps are provided. The results show that we effectively combine Euclidean distance loss and reprojection loss. Using Euclidean distances as a complement and enhancement to the

reprojection error has yielded favorable results.

Wayspots. We also evaluate our method on a small-scale outdoor dataset without depth maps. The main challenge of this dataset is that the scenes contain a large amount of repetitive textures or textureless regions. As shown in Tab. 3, on six of these scenes, we achieved the best performance, with an accuracy improvement of almost 10% points in the Lawn scene. On average, our method achieves an improvement of 2.9% over ACE, while maintaining a training time of 4 minutes.

4.4 Ablation Study

We also conduct ablation studies on the main design choices, modules, and training strategies of our approach on the Cambridge Landmarks dataset.

Scene Partitioning Strategies. As shown in Tab. 4, we compare our method w/ and w/o structural and semantic partitioning. Without the structural partitioning strategy, performance suffers due to structural discontinuity. Without the semantic partitioning strategy, performance shows a slight decline. Our semantic-structural partitioning strategy preserves both semantic consistency and structural continuity within each subscene, achieving the best results.

Feature Fusion Module. As shown in Tab. 4, with the feature fusion module, the prediction error decreases from 16.1 cm to 13.0 cm.

Thresholds for Discriminability Scores. As shown in Fig. 4, the threshold is expressed as a percentage (%) to prevent inconsistencies in discriminability scores in different scenes. As the threshold increases, the position error initially decreases and then increases. Here, we provide a reasonable explanation. When the threshold is small, only ambiguous points are defined as invalid. Excluding these points enhances the pose estimation. However, as the threshold increases, many highly distinguishable points are excluded, leading to instability in the results. As shown in Fig. 2, where c = 0.51, invalid points are concentrated in repetitive textures and flat regions such as grass and sky, while valid points are concentrated in highly distinguishable areas, such as buildings. In the other two scenes of the Cambridge Landmark dataset, ShopFacade and StMarysChurch, the error does not exhibit a clear trend with score variation. However, the error does not decrease. With the discriminability score, our method achieves better results.

5 Conclusion

We propose a novel scene coordinate regression method named Understanding Matters: Semantic-Structural Determined Visual Relocalization for Large Scenes. We partition the scene into small parts, ensuring semantic consistency and structural continuity, further leading the accelerated partition localization with sampling-based learning. Partitiondetermined pose refinement is then put forward to reduce the inaccuracy of pose estimation. Our method supports RGB-D and RGB camera relocalization in large-scale scenes. We can achieve the state-of-the-art performance within 13 minutes.

Acknowledgments

This paper is supported by the Science and Technology Project of Hainan Provincial Department of Transportation (Grant No.HNJTT-KXC-2024-3-22-02), the National Natural Science Foundation of China (Grant No. 62272018, 62206184), and Zhongguancun Laboratory.