A Structural-Analysis-Based Information Fusion for Multi-Modal Cross-View Geo-Localization

1 Introduction

Cross-View Geo-Localization (CVGL) aims to retrieve the corresponding location from large-scale aerial or remote sensing reference databases using ground-level observations as queries. Even when the Global Positioning System (GPS) is interfered with, CVGL can still ensure the stable operation of the navigation and positioning system. Therefore, an increasing number of researchers have begun to pay attention to and explore ways to enhance the application effectiveness of CVGL. However, the large viewpoint gap, scale variations, and heterogeneous appearances make cross-domain alignment challenging.

∗Corresponding author.

Current studies have primarily concentrated on uni-modal CVGL approaches using either cameras [Shugaev et al., 2024; Mi et al., 2024; Zhu et al., 2022; Shi et al., 2020b; Wang et al., 2022] or LiDAR [Shi et al., 2025]. However, single-modality data often lack sufficient discriminative features to distinguish between correct and incorrect matches. Firstly, the data from the ground-level perspective and the data from the aerial-level perspective have only a small amount of information overlapping. Secondly, there are numerous repetitive elements in a city area, making it difficult to distinguish the remote sensing images of these locations. Thirdly, when the sensor is disturbed, e.g., by sudden changes in light intensity, the performance of the system will be further compromised. To address this, recent work has explored multi-modal fusion of street-view imagery and LiDAR [Wang et al., 2025], showing promise in improving the performance of CVGL. However, current methods mainly rely on the network to implicitly learn the relationships between different modalities during the multi-modal feature fusion stage, lacking structural constraints. This limits the full utilization of multi-modal information, degrading the performance of the model. Additionally, existing methods treat all positive samples uniformly in the supervision stage. However, since positive samples are typically obtained from within a certain neighborhood range of the anchor point, the distances between different positive samples and the anchor vary considerably. As the distance increases, the feature similarity between the anchor and positive samples gradually decreases. If these variations are not distinguished during the training stage, the network may receive conflicting supervision signals, which can degrade the network's performance.

To overcome this limitation, we analyze the fusion process of multi-modal data from the perspective of information theory [Williams and Beer, 2010], and proposed a principle for guiding the design of multi-modal CVGL networks: In the design of multi-modal fusion networks, structured constraints should be imposed to explicitly decouple the fusion process, thereby avoiding unstructured mixing of information from different sources. Driven by this principle, we propose Decoupled Query Fusion (DQF). DQF uses role-specific (modalityspecific and cross-modal) learnable queries to organize the fusion process of multi-modal features, ensuring that the final fusion representation can more fully retain the information of both. Furthermore, to ensure that the learned metric embedding space better reflects real-world spatial relationships, we propose a Geo-aware Circle Loss. Our loss function will assign adaptive weights to the losses of positive sample pairs based on the geographical distance between them. Specifically, sample pairs with a closer geographical distance will be assigned a larger weight. This operation ensures that the descriptors generated by the network can better reflect the continuity of the geographical space, making the descriptors more discriminative.

The main contributions of this work are:

• We propose a Structural-Analysis-Based information fusion principle to guide the design of the network framework for multi-modal CVGL. This design strategy enables the network to have greater interpretability and allows it to better integrate multi-modal information.
• We present Decoupled Query Fusion, a novel fusion module that disentangles feature interactions through role-specialized learnable queries. Additionally, we propose a Geo-aware Circle Loss, which incorporates geographic priors into supervisory signals, enabling the model to better preserve spatial relationships.
• Extensive experiments demonstrate the superiority of our approach. On LiRSI-KITTI360 and KITTI360AG, our method achieves 75.4% and 44.9% relative improvements over the state-of-the-art methods in Recall@1, respectively. Furthermore, comprehensive ablation studies validate the effectiveness and necessity of each proposed component.

2 RelatedWork

2.1 Uni-Modal Cross-ViewGeo-Localization

CVM-Net [Hu et al., 2018], as an early exploratory work, adopts a Siamese network architecture and employs NetVLAD [Arandjelovic et al., 2016] for feature aggregation, improving the accuracy of cross-view retrieval through a ranking loss function. SAFA [Shi et al., 2019] reduces the geometric domain gap between ground-level panoramas and satellite images by performing polar coordinate transformation on satellite images, subsequently enhancing the discriminability of feature through spatial attention mechanisms. [Shi et al., 2020a] is the first to jointly train the tasks of position estimation and direction prediction for CVGL. [Regmi and Shah, 2019] employs a conditional generative adversarial network to transform ground-level panoramas into the aeriallevel domain, reducing the domain differences with satellite images. [Toker et al., 2021] proposes a multi-task framework to jointly optimize the view synthesis from satellite images to street-view images and the CVGL in an end-to-end manner. TransGeo [Zhu et al., 2022] adopts a Transformer architecture and models cross-domain correspondence through explicit position encoding. EgoTR [Yang et al., 2021] enhances the generalization ability of representations through selfattention and cross-attention mechanisms. DWDR [Wang et al., 2024] reduces the correlation between channels of embedded features through regularization. ConGeo [Mi et al., 2024] leverages contrastive learning to align features from ground-level observations at the same location but with different orientation and fields of view (FoVs), enabling a single model to achieve orientation invariance and FoVs robustness. L2RSI [Shi et al., 2025] significantly reduces the domain gap between LiDAR point clouds and satellite images by aligning them to a unified semantic space, achieving high-precision cross-scene localization through a particle-based estimation method.

2.2 Multi-Modal Cross-ViewGeo-Localization

Multi-modal CVGL aims to jointly leverage information from multiple ground-level sensors to alleviate the problem of insufficient discriminative information from single-modal observations in complex environments. However, compared to single-modal CVGL, multi-modal CVGL is still in the exploration stage. AGPlace [Wang et al., 2025] fuse ground-level cameras and LiDAR for CVGL, introducing Neural Ordinary Differential Equations to model the continuous cross-modal alignment process, providing pioneering exploration for the research of multi-modal CVGL.

2.3 Supervision for Cross-ViewGeo-Localization

Early work [Vo and Hays, 2016] explored the application of metric learning in CVGL, arguing that ranking constraints are crucial for the stability of retrieval. Based on this, CVMNet [Hu et al., 2018] proposed a weighted soft-margin ranking loss to improve the stability of training. ConGeo [Mi et al., 2024] introduced ground-to-ground constraints in addition to ground-to-aerial constraints. Specifically, it employs contrastive learning to align features of ground-level images from the same location but with different orientations and FoVs, thereby enhancing the robustness of model under ground-level viewpoint perturbations. ArcGeo [Shugaev et al., 2024] improved the retrieval accuracy under limited FoVs conditions by introducing cosine margin into the ArcFace loss. In multi-modal CVGL, AGPlace [Wang et al., 2025] proposed a multi-domain alignment loss, enabling the network to learn consistent representations between groundlevel fusion features and aerial-level features, as well as between individual modality features and aerial-level features, thereby improving overall model performance.

3 Methodology

3.1 ProblemFormulation

Previous research has shown that integrating LiDAR point clouds and remote sensing images into a semantic domain can effectively reduce the modal differences between them, thereby enhancing the generalization ability of the model. Specifically, we follow the semantic data processing procedure in [Shi et al., 2025] to obtain the semantic BEV view Sbev of the LiDAR point clouds and the semantic view Sa of the remote sensing image.

Formally, let G = {Ig, Pg, Sbev} denote the multi-modal ground-level input, where Ig and Pg represent the street-view image and raw point clouds, respectively. Let A = {Ia, Sa} represent the aerial-level input consisting of the satellite image and its semantic view. Our objective is to learn two parameterized mapping functions:

where zg, za ∈ RD are D-dimensional global embedding vectors for ground-level input and aerial-level input respectively, and θg, θa denote the learnable parameters of the ground-level and aerial-level stream networks, respectively.

The learning objective is to optimize the network parameters such that the distance between the global embedding vectors of matching pairs is minimized:

where D+ represents the set of positive ground-aerial pairs, d(·, ·) is a distance metric.

3.2 Architecture Overview

As shown in Figure 1, our network framework takes groundlevel data (street-view images, LiDAR point clouds, semantic BEV views) and aerial-level data (remote sensing images and their semantic views) as inputs, and learns discriminative global embedding vectors through contrastive learning to achieve robust cross-view retrieval.

Ground-Level Stream. In the ground-level stream, we employ different backbones to extract multi-scale features from street-view images and raw point clouds, respectively. To better integrate multi-modal information, we employ the DQF module to fuse features from both modalities at each scale. Subsequently, these multi-scale fused features are aggregated through L transformer layers and pooled via Generalized Mean (GeM) pooling to obtain the global fusion embedding vectors zgo for the street-view images and the raw point clouds. Concurrently, semantic features are extracted through a separate backbone and pooled using GeM pooling, resulting in the global embedding vector zgs of the semantic BEV view. Finally, we fuse zgo and zgs through an adaptive gated module to obtain the ground-level global embedding vector zg .

Aerial-Level Stream. The aerial-level stream follows a symmetric encoding and integration pipeline. Specifically, we employ different backbones with GeM pooling to process the original view Ia and semantic view Sa of the remote sensing images independently, yielding the global embedding vectors zao and zas, respectively. Similarly, we use an adaptive gated module to fuse zao and zas to obtain the aerial-level global embedding vector za.

3.3 Multi-Branch Feature Encoding

Street-View Image Encoder. For the street-view image Ig, we employ a 2D backbone Bimg to extract multi-scale features:

where F i img denotes the feature map of street-view image at scale i.

LiDAR Point Clouds Encoder. For the raw point clouds Pg, we employ a 3D backbone Bpc to extract multi-scale features:

where F i pc denotes the feature vectors of LiDAR point clouds at scale i.

Semantic BEV Encoder. We extract the features of the semantic BEV view Sbev using a 2D backbone, and then obtain its global embedding vector zgs through GeM pooling:

Aerial-Level Encoders. The aerial-level streamencodes Ia and Sa through dual backbones and GeM pooling:

3.4 Decoupled Query Fusion

Structural-Analysis-Based Information Fusion Principle

From an information-theoretic perspective, the information contained in the global embedding vectors generated by multi-modal data at the ground-level can be decomposed into three parts:

• Unique information: Information that exists only in street-view images or LiDAR point clouds, e.g., color information in images and 3D geometric structure of point clouds.
• Redundant information: Information that exists simultaneously in street-view images and LiDAR point clouds, such as the outlines of buildings and the boundaries of roads.
• Synergistic information: Information that can only be obtained through the interaction of street-view images and LiDAR point clouds, e.g., complete environment understanding in obstructed scenarios.

Obviously, in order to fully utilize the information of multimodal data, the final fused representation should retain the above three parts of information as much as possible. However, if the network framework lacks structural constraints, the model is prone to confuse the information of each part during the training process. This will cause the network to overly rely on one part of the information, resulting in the loss of other information. Therefore, we propose a StructuralAnalysis-Based information fusion principle: In the design of multi-modal fusion networks, structured constraints should be imposed to explicitly decouple the fusion process, thereby avoiding unstructured mixing of information from different sources.

Decoupled Query Fusion Module

Motivated by the Structural-Analysis-Based information fusion principle, we propose the Decoupled Query Fusion module, as shown in Figure 2. DQF separately extracts modalityspecific and cross-modal information through role-specific learnable queries.

For the i-th scale, the encoded token sequences of streetview images and point clouds are defined as Xi img ∈ RNimg×d and Xipc ∈ RNpc×d , respectively. DQF maintains three learnable query sets:

where Ku and Kc denote the numbers of modality-specific and cross-modal queries.

Modality-Specific Slot. Modality-specific queries extract information from their corresponding modalities through intra-modal interactions:

where CA(·) denotes the multi-head cross-attention taking (q, k, v) as query, key and value, respectively.

Cross-Modal Slot. Cross-modal queries capture redundant and synergistic information by sequentially interacting with both modalities:

Redundancy-Aware Gated Suppression. Currently, redundant information exists simultaneously in cross-modal queries and modality-specific queries. This would consume the limited representation capacity and prevent the final fusion representation from fully leveraging the information of multi-modal data. Therefore, we introduce the Redundancyaware Gated Suppression mechanism to reduce the redundant information in modality-specific queries, thereby ensuring that the redundant information is concentrated as much as possible in cross-modal queries:

Ri

img = CA(Z i

img , Z icm, Z icm),

(12)

where σ(·) is the sigmoid function, MLP(·) is a Multi-Layer Perceptron and ⊙ denotes element-wise multiplication.

Hybrid Slot Aggregation. The fused representation at scale i is obtained by concatenating the refined modalityspecific queries and the cross-modal queries:

3.5 Multi-Scale Feature Aggregation

The multi-scale fusion features obtained from Sec.3.4 will be concatenated together. Subsequently, we use the L-layer multi-head self-attention (MHSA) module to integrate the high-resolution detailed features and the low-resolution semantic features:

Subsequently, we aggregate the features into a global embedding vector through GeM pooling:

3.6 Semantic-Aware Gated Integration

To integrate the global embedding vectors of the original view and the semantic view, we employ an adaptive gated module. This mechanism enables the model to dynamically adjust the weight ratios between the two views according to the feature distributions of different samples, thereby achieving more flexible feature representation.

Ground-Level View. Given the zgo and zgs, we compute a channel-wise gating mechanism to obtain the fused global embedding vectors for the ground-level views:

Aerial-Level View. Similarly, given the zao and zas, we apply a channel-wise gating mechanism to obtain the fused aerial-level global embedding vector:

3.7 Loss Function

To ensure robust cross-view alignment and effective feature decoupling, we propose a composite objective function consisting of a geometry-aware contrastive loss and a multi-scale regularization term.

Geo-Aware Circle Loss

Standard metric learning losses treat all positive pairs uniformly. In the CVGL task, the positive samples are usually selected from the specific-sized spatial neighborhood around the anchor point. Therefore, the distances between different positive samples and the anchor point are different. If these differences are not distinguished during the training process, the network may receive conflicting supervisory signals, which will lead to a decline in its performance. To address this, we propose a Geo-aware Circle Loss, which calibrates the supervisory signal based on the spatial distance between the positive sample pairs.

We use a Gaussian kernel function to assign corresponding weights to positive samples of different distances:

where pig , pja ∈ R2 represent the Universal Transverse Mercator (UTM) coordinates of the ground-level and aerial-level samples, and h denotes the bandwidth parameter that controls the decay rate.

We adapt the Circle Loss [Sun et al., 2020] by integrating wij into the gradient re-weighting mechanism. The geometry-aware coefficients for positive pairs ˆαjp and negative pairs αkn are computed as:

where s represents the similarity score between sample pairs, Op and On denote the target boundaries for positive and negative scores, respectively. Consequently, the final geometryaware circle loss is formulated as:

where P and N denote the sets of positive and negative samples for the anchor sample i, respectively, γ is the scaling factor that controls the penalty strength, and ∆p, ∆n are the margins. By incorporating the geometric weight wij into ˆαjp , the loss function implements differentiated supervision on positive samples of different spatial distances. Specifically, the contribution of distant matches is weakened, while the contribution of close matches is enhanced. This mechanism guides the model to learn more fine-grained spatial correspondences.

Multi-Scale Decoupling Regularization

To ensure the structural independence between modalityspecific and cross-modal features during multi-modal fusion, we impose an orthogonality constraint. Specifically, for the i-th scale,

where ∥ · ∥F represents the Frobenius norm. The multi-scale regularization loss is formulated as:

Overall Loss Function

The complete training objective is formulated as a weighted combination of the geometry-aware contrastive loss and the decoupling regularization:

where the hyperparameter λ controls the trade-off between cross-view alignment accuracy and feature disentanglement quality.

4 Experiment

4.1 Datasets

We evaluate our proposed approach on LiRSI-KITTI360 [Shi et al., 2025] and KITTI360-AG [Wang et al., 2025] benchmark datasets.

KITTI360-AG. KITTI360-AG is amulti-modal cross-view geo-localization dataset built upon KITTI-360 [Liao et al., 2022]. It correlates the ground-level modality with spatially aligned aerial imagery. Following [Wang et al., 2025], we conduct experiments on 7 sequences (00, 02, 03, 04, 05, 06, 09), using the first 85% of frames for training and the remaining 15% for testing. Following the protocol in LiRSI [Shi et al., 2025], we construct dense point cloud submaps at 5-meter intervals along the vehicle trajectory.

LiRSI-KITTI360. LiRSI-KITTI360 is a large-scale dataset derived from KITTI-360, designed for LiDAR-based place recognition using remote sensing imagery. To adapt this dataset for our research, we expand each LiDAR frame with corresponding street-view image. To evaluate the model's generalization capability, we set sequence 02 as the test set while using the remaining 8 sequences (00, 03, 04, 05, 06, 07, 09, 10) for training, as shown in Figure 3. To prevent data leakage, we remove the training samples located in the test remote sensing area.

4.2 Implementation Details

Our method is implemented in PyTorch. For feature extraction, we adopt MinkLoc3D [Komorowski, 2021] with a voxel size of 0.5 m as the 3D backbone and an ImageNet-pretrained ResNet18 [He et al., 2016] as the 2D backbone. The network is trained using the Adam optimizer for 100 epochs with a batch size of 16. We set the initial learning rates of 1 × 10-4 for the 3D branch and 1 × 10-3 for the 2D branch, both scheduled with CosineAnnealingWarmRestarts including a 5-epoch linear warmup. Positive pairs are defined using a spatial threshold of 30m. The circle loss adopts a scale factor γ = 80 and a margin m = 0.4. For the proposed module, we set L = 2, Ku = 64, Kc = 32, and h = 20.

4.3 Experimental Results

Evaluation Metrics

Following [Wang et al., 2025], we adopt Recall@K to evaluate the retrieval performance, where K ∈ {1, 5, 10}. Recall@K is defined as the proportion of queries that contain at least one ground-truth match among the top-K retrieved results. Specifically, for each ground-level query, we rank all candidates in the remote sensing image database by their feature similarity to the query. A retrieval is deemed successful if any of the top-K candidates lies within a spatial distance threshold from the query location. This metric directly reflects the usability of the model in practical application scenarios.

Main Results

We mainly compare our method with multi-modal methods, including MinkLoc++ [Komorowski et al., 2021], AdaFusion [Lai et al., 2022], MSSPlace [Melekhin et al., 2025], LCPR [Zhou et al., 2023], UMF [Garc´ıa-Hern´andez et al., 2024], and AGPlace [Wang et al., 2025]. Additionally, among the uni-modal CVGL methods, we also select the representative Sample4Geo [Deuser et al., 2023] and L2RSI [Shi et al., 2025] for comparison. For the experiments with Sample4Geo, we use street-view images as the ground-level input.

Results on KITTI360-AG. We validate the fitting and generalization capability of our approach on KITTI360-AG benchmark. On this dataset, we follow the experimental protocol from [Wang et al., 2025] and operate without orientation information. As shown in Table 1, our method outperforms existing state-of-the-art approaches across all evaluation metrics. Notably, our method achieves 62.3% on Recall@1, achieving a 44.9% relative improvement over L2RSI. This substantial improvement demonstrates our method can learn more discriminative global feature representations, highlighting its strong potential for real-world deployment.

Results on LiRSI-KITTI360. To further validate the generalization capability of our method, we conduct additional cross-scene experiments on the LiRSI-KITTI360 dataset. Specifically, we evaluate two experimental settings:

1. Orientation-Free: The model does not utilize any orientation prior information, relying solely on visual and geometric features for matching.

1. Orientation-Aware: The model incorporates orientation information to guide the retrieval process. To simulate the measurement errors of sensors (e.g., magnetometer) in real-world scenarios, we inject random perturbations within ±15◦ into the ground-truth orientation data.

As shown in Table 2, our method significantly outperforms all baseline methods in the orientation-free setting, demonstrating its superior capability in extracting rotation-invariant features. In the orientation-aware setting, our method also maintains a significant advantage. These results validate the strong cross-scene generalization ability of our approach. It is worth noting that after introducing directional information, Recall@1 and Recall@10 have increased from 16.6% and 41.6% to 40.7% and 76.2%, respectively. This highlights the critical role of orientation information in cross-scene retrieval task. To provide intuitive insights into the retrieval performance, we visualize the per-query results along the test trajectory in Figure 4.

4.4 Ablation Studies

To validate the effectiveness of individual components in our framework, we conduct ablation studies on the LiRSIKITTI360 dataset. All experiments are performed under the orientation-aware configuration. We investigate the contributions of two key components: the DQF module and the Geoaware Circle Loss. The quantitative results are presented in Table 3.

DQF. To evaluate the effectiveness of the DQF module, we remove it from the complete framework. The results are shown in the second row of Table 3. Without the DQF module, Recall@1 drops from 40.7% to 34.1%. This significant decrease demonstrates the module's crucial role in enhancing feature discriminability. This also provides an additional verification of the effectiveness of our proposed StructuralAnalysis-Based information fusion principle.

Geo-Aware Circle Loss. To evaluate the contribution of Geo-aware Circle Loss, we replace it with the standard Circle Loss while keeping all other components unchanged. The results are presented in the third row of Table 3. After removing the supervision weighting mechanism based on geographical distance, Recall@1 decreased by 6.8 percentage points. Compared to the standard Circle Loss which treats all positive sample pairs equally, Geo-aware Circle Loss provides more fine-grained supervision signals based on the geographical distance between samples. This enables the network to better distinguish between positive samples from different geographical locations and prevents the inappropriate alignment of samples with large spatial separations, leading to more discriminative feature representations.

5 Conclusion

In this work, we investigate multi-modal cross-view geolocalization. Specifically, we propose a Structural-AnalysisBased information fusion principle that guides the design of our DQF module. Furthermore, we introduce a Geo-aware Circle Loss that provides fine-grained supervision signals based on the geographical distance between samples. Extensive experiments demonstrate that our method significantly outperforms prior state-of-the-art approaches.

However, in the task of cross-scene generalization, our method relies heavily on orientation information to achieve strong retrieval performance, which constitutes a notable limitation. Future work will focus on extracting rotationinvariant features, aiming to maintain robust performance even in the absence of directional cues.

Acknowledgements

This work is supported in part by the National Natural Science Foundation of China (No. 62471415, 62401225); the Natural Science Foundation of Fujian Province of China (No. 2023J01004, 2025J09046, 2024J01115); the Natural Science Foundation of Xiamen of China (No. 3502Z202472018); the Jimei University Scientific Research Start-up Funding Project (No.ZQ2024034).

Contribution Statement

† denotes that Xu Yan and Xiaoran Zhang are co-first authors and contribute equally to this work.