Multimodal Image Matching Based on Cross-Modality Completion Pre-training

1 Introduction

Multimodal images, from different sensors like visible and infrared images, provide a more comprehensive understanding of the scene than single-modal images. They are valuable for advanced vision tasks, such as image fusion [Tang et al., 2022a; Ma et al., 2022], object detection and tracking [Zhao et al., 2023], and 3D reconstruction [Jiang et al., 2021]. However, multimodal images suffer from geometric distortions such as scale variations, rotations, and perspective distortions due to differences in imaging devices, which makes it hard for computers to analyze. Therefore, multimodal image matching is needed to establish the correspondence of feature points or regions between different modal images.

∗Corresponding author.

Multimodal images have significant modal differences in radiometric properties. For example, visible images capture the reflected light of object, while infrared images capture the thermal radiation [Tang et al., 2022b; Li et al., 2013]. This leads to inconsistencies in texture, contrast, and intensity of the images. The modal differences reduce the accuracy of feature extraction, and increase the matching difficulty. Current deep learning-based matching methods train their models on rich single-modal datasets and struggle to generalize to multimodal image scenarios, mainly due to the lack of largescale annotated datasets. These problems limit the practical application of multimodal image matching methods.

To overcome these problems, we propose XCP-Match, a multimodal image matching algorithm based on crossmodality completion pre-training. The training of XCPMatch has two phases. The first is the self-supervised pretraining phase. We develop a novel pre-training model using vision transformer (ViT) [Dosovitskiy et al., 2021], VGG [Simonyan and Zisserman, 2015], cross-attention fusion module and image reconstruction module. The pre-trained model has two branches to adaptively extract features from each input image. The task of this model is to perform crossmodality completion using a new masked image modeling method (MIM), which is reconstructing the masked parts of one modal image using the visible information in another modal image (the reference image). Pre-training forces the model to learn richer and in-depth semantic features across different modalities by cross-modality completion, which enhances the model's understanding of image content, leading to more accurate matching. Pre-training is performed on a real multimodal image dataset, where the image pairs are not fully aligned. To encourage the ViT encoder to focus on extracting features from the overlapping regions, we propose the attention-weighted contrast loss. The second is the supervised fine-tuning phase, which fine-tune the overall matching network using the MegaDepth dataset [Li and Snavely, 2018], based on the pre-training weights. The MegaDepth dataset is augmented to enhance the model's adaptability to modal differences. We construct a complete matching framework for XCP-Match, including multimodal and multiscale feature extraction module, coarse-level matching module, finelevel matching module, and subpixel refinement module, to achieve precise matching. The two-phase training encourages the model to learn deep cross-modal semantic features to improve adaptation to modal differences and does not require large annotated datasets. We evaluate XCP-Match on the public datasets, comparing it to state-of-the-art algorithms, and achieve the best performance.

In summary, this paper has the following contributions:

• We design a novel dual-branch pre-training model, and introduce unsupervised pre-training using a new MIM method to enhance the model's ability to extract multimodal image features.
• We design a novel attention-weighted contrast loss in pre-training to make the feature extraction focus on overlapping regions and overcome the problem of unaligned images in the dataset.
• We design a complete matching network suited for multimodal images and perform supervised fine-tuning with the augmented MegaDepth dataset for accurate optimization of matching.

2 RelatedWorks

2.1 Feature Detector-basedMatchingMethod

Feature detector-based matching methods involve three stages: features detecting, describing, and matching [Lowe, 2004; Bay et al., 2006; Li et al., 2024]. To overcome the nonlinear radiance differences (NRD) [Ye et al., 2022; Li et al., 2023a] of multimodal images, researchers have focused on improving the performance of feature detectors and descriptors, developing algorithms such as RIFT [Li et al., 2019] and RA-MMIR [Qiu et al., 2024]. Deep learning methods have also made some progress such as ReDFeat [Deng and Ma, 2023], which decouples detection from matching, and XPoint [Yagmur et al., 2024], using a pre-trained encoder for feature extraction while applying geometric constraints. SemLA [Xie et al., 2023] focuses on aligning key scene regions. However, these methods struggle to detect reliable keypoints in the scenario of lacking prominent features, which reduces the accuracy of subsequent matching.

2.2 Detector-Free MatchingMethod

Detector-free methods aim for pixel-level correspondences directly in input images, without using feature detector.

These methods uses transformers' capability [Vaswani, 2017] to handle long-distance dependencies and global context, and perform more robustly in images with large geometric distortions. LoFTR [Sun et al., 2021] achieves dense matching through a coarse-to-fine approach, while Efficient-LoFTR [Wang et al., 2024] improves efficiency and accuracy with attention mechanisms. DualRC [Li et al., 2023b] employs dual-resolution learning and neighborhood consensus. GRiD [Liu et al., 2024] addresses NRD and geometric distortions by finding pixel-level matches using reference points. However, these methods require large-scale annotated datasets, which are lacking in multimodal images.

2.3 Transformation Estimation-basedMethod

Transformation estimation methods include flow and homography estimation. Flow estimation methods estimate each pixel's spatial displacement for registration. Zhou et al. transfers knowledge from optical flow model to cross-modal flow estimation [Zhou et al., 2022]. Homography estimation methods predict image transformations. CrossHomo [Deng et al., 2024] uses multimodal homography estimation module to predict homography matrices accurately. Real-world transformations' complexity challenges these methods.

2.4 MIM in Image Matching

Inspired by BERT [Kenton and Toutanova, 2019] and ViT, self-supervised MIM methods have arisen in computer vision, training models to recover complete signals from unmasked data parts. MIM methods like MAE [He et al., 2022], MultiMAE [Bachmann et al., 2022], and MST [Li et al., 2021] excel in classification and also make breakthroughs in geometric tasks. CroCo [Weinzaepfel et al., 2022] reconstructs the masked image using another view of the same scene, suitable for geometric tasks. CroCo v2 [Weinzaepfel et al., 2023] builds on CroCo with enhancements. PMatch [Zhu and Liu, 2023] extends MIM to pairwise reconstruction, optimizing both encoder and decoder. Tuzcuo˘glu improves LoFTR based on PMatch to match infrared and visible images [Tuzcuo˘glu et al., 2024]. These developments highlight MIM's role in enhancing feature representation and narrowing the domain gap between pre-training and downstream tasks.

3 Methodology

Our method has four modules: multimodal and multiscale feature extraction module, coarse-level and fine-level matching module, and sub-pixel refinement module. We introduce a two-phase training strategy including self-supervised pretraining and supervised fine-tuning for matching. The meaning of the symbols is shown in Table 1.

3.1 Multimodal andMultiscale Feature Extraction Module

To address the challenges of modality and viewpoint differences in multimodal images, we propose the multimodal and multiscale feature extraction module (MM-FEM) based on VGG networks and ViT. To enhance the performance of MMFEM in extracting multimodal image features, we introduce a novel MIM method to pre-train MM-FEM.

Feature extraction: Given two multimodal images from the same scene, e.g., infrared image Iir and visible image Ivis, they are gridded into N = n2 non-overlapping patches and labeled as tokens for input to the MM-FEM. The MMFEM has two different asymmetric branches as shown in Figure 1. The inputs to the first branch are tokens from Ivis and Iir, respectively. The tokens of Ivis are fed to the VGG and the ViT encoder, respectively, while the tokens of Iir are fed only to the ViT encoder with shared weights. The ViT decoder fuses the features Fvis = Encoder(Ivis) and Fir = Encoder(Iir) from the encoders to enable the deep interaction of multimodal image information, resulting in a cross-modal feature map of size [B, N, C]:

A multibranch VGG with residual connections processes tokens from Ivis and generates the base feature map of size [B,C,H/2,W/2]: FVGG = VGG(Ivis), where H and W are the height and width of the original image, respectively.

Cross-attention fusion: To enhance the fusion of information from another modal image, we introduce a multi-headed cross-attention fusion (CAF) module to adaptively update the base feature map FV GG, which enables the model to learn the richer and more robust feature representations.

In CAF, we use FV iT as key and value, and use FVGG as query. First, reshape FVGG to size [B, (H ∗ W)/4,C]. The normalized attention score matrix M is computing by the following:M = Softmax(FV GG · F T V iT ). (2)

The enhanced feature for each location is computed by weighting and summing FV iT using the attention score matrix M. Since M varies with different modal inputs, it exhibits adaptability to newmultimodal images. Finally, the updated result is linearly transformed by a fully connected layer to yield the enhanced feature FCAF with the same shape of FV GG:

where FCAF is the enhanced feature map. For brevity, we omit some reshaping operations in equation.

Down-sampling: Input FCAF into the VGGwith different down-sampling scales to further extract the features of 1/4 and 1/8 original image resolution for the later coarse-to-fine matching.

As shown in Figure 1 (b), the structure of the second branch is same as the first branch. However, in the second branch, the tokens of Iir are fed to both the VGG and ViT. Additionally, we use rotational position embedding (RoPE) on the image tokes before the encoder in two branches, which injects the relative position information of features.

3.2 Coarse-level MatchingModule

The coarse-level matching module (CMM) uses the coarselevel feature map F 1/8 vis and F 1/8 ir of 1/8 resolution fromMMFEM to predict the matches at 1/8 scale.

Coarse-level feature interaction: We use the linear selfattention and cross-attention in LoFTR to interact F 1/8 vis and F 1/8 ir , outputting ˆ F 1/8 vis and ˆ F 1/8 ir , which captures the global dependencies between multimodal images.

Similarity matrix computation: Given ˆ F 1/8 vis and ˆ F 1/8 ir , the similarity matrix S is computed by the following:

where i and j are the indexes in ˆ F 1/8 vis and ˆ F 1/8 ir , respectively. Linear(·) denotes the linear layer, and ⟨·, ·⟩ denotes the inner product. γ is the temperature parameter.

Matching probability matrix computation: Apply softmax operation to S(i, j) to obtain the matching probability matrix Pk∈(0,1)(i, j) = softmax (S(i, ·))j. P0 and P1 are the matching probability matrices derived from softmax along the first and zeroth dimensions, respectively.

Coarse-level match acquisition: We use the threshold θc to filter out high-confidence elements and select the largest as the matching pair (˜i, ˜j):

The final coarse-level matching set is Mc = M0 ∪M1.

3.3 Fine-level MatchingModule

Based on Mc, the fine-level matching module (FMM) uses F 1/2 and F 1/4 to seek fine-level matches at 1/2 resolution to improve the matching accuracy. Given the significant texture differences between images of different modalities, FMM reevaluates coarse-level matches on a finer scale, capturing detailed texture information more accurately. In addition, FMM is also part of the image reconstruction module in pre-training because it can capture texture and structural information at different scales.

Feature preprocessing: To improve the interaction of F 1/2 and F 1/8 , we efficiently preprocess them. First, we cascade ˆ F 1/8 vis and F 1/8 vis along the channel dimension, and apply point-by-point convolution and deep convolution with 3 × 3 kernel to obtain ˜ F 1/8 vis = Conv3(Conv1( ˆ F 1/8 vis |F 1/8 vis )) with the same channel size as F 1/4 vis . Similarly, the same operation is used for ˆ F 1/8 ir and F 1/8 ir to obtain ˜ F 1/8 ir .

Local window extraction: The local window pair {f 1×1 vis , f 1×1 ir }, {f 3×3 vis , f 3×3 ir }, and {f 5×5 vis , f 5×5 ir } of each (˜i, ˜j) are extracted from { ˜ F 1/8 vis , ˜ F 1/8 ir }, {F 1/4 vis , F 1/4 ir }, and {F 1/2 vis , F 1/2 ir } using window sizes of 1×1, 3×3, and 5×5, respectively. We add the absolute position bias to each win-

dow before sending it to the next layer.

Multiscale and multimodal feature passing: To pass information between {f 1×1 vis , f 1×1 ir }, {f 3×3 vis , f 3×3 ir } and {f 5×5 vis , f 5×5 ir }, we perform a series of concatenation, linear layer, self-attention, cross-attention and splitting operations to obtain { ˆ f 5×5 vis , ˆ f 5×5 ir }.

Fine-level matching: For each (˜i, ˜j), compute the similarity matrix Sf between { ˆ f 5×5 vis , ˆ f 5×5 ir }, and apply a double softmax to obtain the fine-level match probability matrix P f :

Finally, the matching pair (ˆi, ˆj) with P f (i, j) greater than the threshold θf and all other elements in each (˜i, ˜j) is selected as the fine-level match Mf .

3.4 Subpixel Refinement Module

The goal of this module is to refine the fine-level matches to subpixel accuracy. We concate { ˆ f 5×5 vis , ˆ f 5×5 ir } at the finelevel match (ˆi, ˆj), and use the MLP layer and the Tanh function to jointly predict the local subpixel offsets for each match: {δvis, δir} = Tanh(MLP( ˆ f 5×5 vis | ˆ f 5×5 ir )). Finally, these offsets are added on the coordinates of (ˆi, ˆj) to obtain the subpixel-level matches:

where {Cvis,Cir} is the coordinate of (ˆi, ˆj) before subpixel refinement and { ˆ Cvis, ˆ Cir} is the coordinate after refinement.

3.5 Self-Supervised Pre-training

MIM method: XCP-Match is pre-trained with the pretraining model and MIM method. The goal of pre-training is to train the model to complete the masked parts of the image using another modal image as reference. The pre-training model is shown in Figure 1 (b). The inputs to first branch are the tokens of masked Ivis and Iir. The tokens of masked Ivis are fed to the VGG and the ViT encoder, while the tokens of Iir are fed only to the ViT encoder. The inputs to second branch are the tokens of masked Iir and Ivis. The feature map after MM-FEM processing is directly output to FFM. Since FFM uses feature map at different scales, we mask the feature map at each scale. To mask the input images, we randomly generate 64×64 masks using the binary method to cover 50% of the image. The remaining scales of feature map use the same masking method and rate, with only the masked patch size adjusted according to the scale ratio. To make the masked areas learnable, we replace them with CNN-processed learnable masked tokens at the corresponding positions.

Image reconstruction module: The masked feature map is processed by FMM to obtain { ˆ f 5×5 vis , ˆ f 5×5 ir }. To reconstruct the image, we designed linear layers to upsample { ˆ f 5×5 vis , ˆ f 5×5 ir } to the original image resolution: {ˆ I 10×10 vis , ˆ I 10×10 ir } = Linear( ˆ f 5×5 vis , ˆ f 5×5 ir ). The 10 × 10 image window of output is used to reconstruct the masked tokens of input at the same locations.

MIMloss function: To ensure that the output image is similar to the target image, the mean square error (MSE) between the target image window I 10×10 i and ˆ I 10×10 i is used as the similarity loss:

Attention-weighted contrast loss: The image pairs in the pre-training dataset are not perfectly aligned. To encourage the ViT encoder to focus on overlapping regions, we propose the attention-weighted contrast loss based on the contrastive learning. It pulls together the feature representations of different modalities in overlapping regions, while pushing away them in non-overlapping regions. This loss facilitates MMFEM to establish the common semantic space across different modalities.

The similarity matrix between the ViT encoder output features Fvis and Fir in the first branch is first calculated using matrix multiplication. Attention weights are derived by normalizing the similarity matrix with the softmax function, which are applied as the weighted average on Fvis to obtain the attention feature ¯ Fvis. The attentional weighted contrast loss is given by the following:

where B is the batch size, N is the number of image tokens. T is the temperature coefficient. sim(Fir, ¯ Fvis) is the positive sample similarity and simk is the similarity of all positive and negative sample pairs. Similarly, Lir ac can be obtained from the second branch.

Total pre-training loss is: Lpretrain = λsLs + λvis ac Lvis ac + λir acLir ac .

3.6 Supervised Fine-tuning

Based on the self-supervised pre-training weights, we perform supervised fine-tuning of the overall network using the dataset with ground-truth (GT). There are three loss functions for the fine-tuning.

Coarse-level matching loss: we use focus loss (FL) to supervise the matching probability matrix Pk∈(0,1) in CMM:

where ˆ P0 and ˆ P1 are the GT matching matrices for coarse matching. α and β are the weights that balance the two loss terms.

Fine-level matching loss: We design the fine-level matching loss to supervise P f in FMM:

where ˆ P f ˆi,ˆj is the GT fine-level matching matrix for (ˆi, ˆj).

Subpixel refinement loss: Given that the predicted matched point pairs have homogeneous coordinates (ˆxvi, ˆxir) in the normalized coordinate system, the subpixel refinement loss can be computed using the symmetric polar distance function:

where E is the GT essential matrix obtained using the camera pose. ∥v∥0:2 denotes the first two elements of the vector v.

Total fine-tuning loss is: Lfine = λcLc + λfLf + λsubLsub.

4 Experiments

4.1 Implementation Details

Self-supervised pre-training: In pre-training, we use the KAIST Multispectral Pedestrian dataset [Hwang et al., 2015] for training. Pre-training is conducted using the AdamW optimizer with a learning rate of 2.5 × 10-4 , a batch size of 2, a total of 15 epochs, and 30 hours of training on 2 NVIDIA GeForce RTX 4090 GPUs.

Supervised fine-tuning: In fine-tuning, we use the MegaDepth dataset [Li and Snavely, 2018]. To enhance the robustness to modal differences, we adopt the operation of XoFTR to augment data. Fine-tuning is conducted using the AdamW optimizer with a learning rate of 2.5×10-4 , a batch size of 2, a total of 25 epochs, and 125 hours of training on 2 NVIDIA GeForce RTX 4090 GPUs.

Hyperparameter settings: The thresholds in the matching network are set to: θc = 0.3, θf = 0.1. The settings in the loss function are set to: λc = 0.5, λf = 0.3, λsub = 104 , λs = 1, λvis ac = λir ac = 0.25.

4.2 Relative Pose Estimation

Dataset: To evaluate the performance of XCP-Match for relative pose estimation in visible-infrared image pairs, we test it on the METU-VisTIR dataset [Tuzcuo˘glu et al., 2024]. The dataset has thermal infrared and visible image pairs from six scenarios with GT camera poses. Four of these scenarios have both cloudy and sunny conditions, while the other two scenarios have only cloudy conditions.

Comparison scheme: XCP-Match processes the input images and generates matched point pairs. We use RANSAC [Fischler and Bolles, 1981] with a threshold of 3 to filter correct matching point pairs. During testing, the longer image side is set to 640 pixels to standardize sizes. We evaluate the methods independently on the six scenarios of the dataset to examine performance differences under different weather conditions. We use the area under curve (AUC) at 5◦ , 10◦ and 20◦ thresholds as evaluation metrics, measuring the maximum angular deviation from the GT in rotation and translation. We compared XCP-Match with the following publicly available methods: ReDFeat [Deng and Ma, 2023], GRiD [Liu et al., 2024], Xpoint [Yagmur et al., 2024], SemLA [Xie et al., 2023], XoFTR [Tuzcuo˘glu et al., 2024], DualRC [Li et al., 2023b] and Efficient-LoFTR [Wang et al., 2024].

Results: As shown in Table 2 and Figure 4, XCP-Match achieves significantly higher AUC than other algorithms at all thresholds for the cloudy-cloudy and cloudy-sunny datasets. XCP-Match also achieves a significant performance advantage in most scenarios. The performance on the cloudy-sunny dataset is lower than on the cloudy-cloudy dataset, due to increased image feature variation from light and temperature differences, which makes matching and pose estimation more challenging. Figure 3 (a) illustrates the qualitative results.

4.3 Homography Transformation Estimation

Dataset: To test XCP-Match for homography estimation, we use the RoadScene dataset [Xu et al., 2020b; Xu et al., 2020a]. We randomly generate a unique homography matrix and apply it to the original images as GT. The homography matrices include random translations of [-10, 10], random rotations of [-10,-10], random scaling of [0.8, 1.2], random shear angles of [-0.1, 0.1], and random perspective transformations of [-0.001, 0.001].

Comparison scheme: The evaluation metrics still use AUC. Since the homography matrix describes geometric transformations in planar scenes, we approximate the scene as planar to estimate the camera pose.

Results: As shown in Table 3 and Figure 3 (a), the AUC of XCP-Match are significantly higher than those of other methods at all thresholds, with the performance gap increasing as the threshold increases. Figure 3 (a) shows that XCP-Match accurately aligns the feature points between the source and target image, which ensures the transformed image maintains good geometric consistency and structural integrity, even with scale variations, perspective distortions and rotations.

4.4 Image Registration Test with Segmentation Annotations

Dataset: To test XCP-Match's performance for image registration, we evaluate it on the TriModalHuman dataset [Palmero et al., 2016]. It contains 5724 RGB-Depth-FIR triples across three indoor scenes, with human body segmentation annotations. We use RGB-FIR images and select only those with distinct human segmentations for testing. To avoid causing damage to the segmentation annotations, we only apply a small degree of random homography transformation.

Comparison scheme: We estimate the homography transformation based on image matching results and apply it to the segmentation annotations. The algorithm's performance is tested by evaluating the alignment between the transformed and target segmentation annotations. The evaluation metrics are LTA (Label Transfer Accuracy) and IoU (Intersection over Union), respectively. We add Cross-RAFT [Zhou et al., 2022] as the comparison algorithm.

Results: As shown in Table 4 and Figure 4, XCP-Match achieves the best performance in all three RGB-FIR indoor scenes. XCP-Match focuses on salient regions and generate denser and more homogeneous matches. Due to this characteristic, XCP-Match can better focus and align human segmentation annotations to achieve best results in LTA and IoU as shown in Figure 5.

4.5 Ablation Study

To verify the effectiveness of the modules and training strategy in XCP-Match, we perform the ablation experiments in the METU-VisTIR dataset. The results are shown in Table 5 and Figure 3 (b). (1) w/o pre-training: Training directly on MegaDepth without pre-training. The results show significant performance degradation, indicating the importance of pretraining for effective feature learning. (2) w/o pre-training model: Using only VGG in MM-FEM and removing other components. The results show a significant decrease in AUC, indicating its critical role in multimodal feature extraction and interaction. (3) w/o attention-weighted contrast loss: Removing this loss during pre-training leads to decrease in AUC, as this loss helps the ViT encoder focus on overlapping regions. (4) w/o RoPE: Removing the RoPE in ViT encoder. The results show that lacking position information negatively affects the algorithm's matching performance. (5) The results of Figure 3 (b) show that the absence of either component degrades matching performance, underscoring their importance for cross-modal feature learning.

5 Conclusion

In this study, our XCP-Match introduces a two-phase training strategy. In pre-training phase, we design a novel dualbranch pre-training model and MIM method to achieve crossmodality completion. The attention-weighted contrastive loss is designed to ensure the model focuses on overlapping regions. The fine-tuning phase is based on the augmented MegaDepth dataset to enhance its robustness against modal differences. XCP-Match constructs a complete matching framework. Our evaluations on public datasets demonstrate that XCP-Match outperforms state-of-the-art algorithms.