SSML-QNet: Scale-Separative Metric Learning Quadruplet Network for Multi-modal Image Patch Matching

1 Introduction

With the development of imaging sensor, more and more applications integrate multiple imaging sensors to perform the tasks with high performance requirements, such as military exploration, medical detection, and security monitoring [Barnea and Silverman, 1972]. As the key technology for these applications, multi-modal image matching has drawn increasing attention from the research community [Zagoruyko and Komodakis, 2015; Moreshet and Keller, 2021; Simo-Serra et al., 2015; Savinov et al., 2017], which aims to measure the similarities between two image patches. However, image matching, especially multi-modal image matching, is still an ill-posed problem that suffers from the significant diversities in visual appearance of different modal images.

∗ corresponding author

Multi-modal methods in general have been extensively explored [Own and Hassanien, 2002]. At an early stage, almost traditional image matching algorithms were based on handdesigned feature descriptors, such as SIFT [Lowe, 2004], PCA-SIFT [Ke and Sukthankar, 2004], SURF [Bay et al., 2006], and SSIF [Liu et al., 2008], etc. The above methods have achieved good performance on single modal image matching task, but their accuracy and robustness were still limited due to the significant appearance divergence between different modal images.

Recently, the methods based on deep learning technology have achieved great progress in this field. They can generally fall into two groups: descriptor learning [Simo-Serra et al., 2015; Balntas et al., 2016a; Savinov et al., 2017; Tian et al., 2017; Mishchuk et al., 2017; Quan et al., 2019] and metric learning [Zagoruyko and Komodakis, 2015; Han et al., 2015; Kumar BG et al., 2016; Baruch and Keller, 2021; Moreshet and Keller, 2021]. Descriptor learning-based methods generate global descriptor of input image patches through deep neural network, and measure their similarity by simple descriptor distance, then distinguish matching and nonmatching through a proper threshold. By contrast, metric learning-based methods adopt a metric network to convert image patch matching problem into a binary classification task (matching and non-matching), which consists of a feature extraction part and classification part. Compared to descriptor learning-based methods, metric learning-based methods are more flexible and effective, since they can simultaneously optimise feature representation and similarity measurement.

The existing metric learning-based methods have achieved a state-of-the-art performance in recent studies, but they only focus on learning invariant and discriminative features. Especially when measuring the similarity, they often take the features as a whole and largely overlook the fact that different scale features for a same image pair may have different similarity, which may lead to sub-optimal results only. To solve this problem, designing the fine-grained similarity metric is a promising solution. As motivated by the above analysis, we design a novel network architecture, which can encode the similarity of different scale features of the input image patch pair, as shown in Fig. 1.

The main contributions of this paper are summarized as follows:

1) A novel Scale-Separative Metric Learning Quadruplet network (SSML-QNet) is proposed for multi-model image patch matching. With a quadruplet network architecture, SSML-QNet can extract both relevant and irrelevant features of imaging modality. The proposed SSML (Scale-Separative Metric Learning) module separately encodes the similarity of different scale features. For each scale, SSML can accurately extract and measure cross-model consistent features by the operations of coordinate attention and Squeeze-andExcitation (SE) attention. It makes our model robust to appearance divergence caused by different imaging mechanisms.

2) Experiments on benchmark datasets (VIS-NIR, VISLWIR, Optical-SAR, and Brown) have verified that the proposed SSML-QNet outperforms other state-of-the-art methods. The mean value of false positive rate at true positive rate equal to 95% (FPR95) is reduced to 0.75, 1.56, 0.65 and 0.58 on VIS-NIR, VIS-LWIR, Optical-SAR and Brown dataset, respectively. The transferring experiments also show that our method has powerful ability of cross-dataset transferring.

The remainder of this paper is organized as follows. The proposed method is described in Section 2. Section 3 presents the experiment configuration, experimental results and analysis. Finally, the conclusion is given in Section 4.

2 Method

2.1 Overview

Fig. 2 shows the structure of the proposed Scale-Separative Metric Learning Quadruplet network, which is composed of three modules: quadruplet multi-model feature extraction module, scale-separative metric learning module, and multiscale feature fusion and prediction module. When a new multi-modal image pair arrives, the quadruplet multi-model feature extraction module utilizes two types of CNN subnetworks to generate the relevant and irrelevant features of imaging modality. Then both relevant and irrelevant features are fed into the scale-separative metric learning module to encode the similarity of different scale features for increasing the accuracy of metric learning. After that, the multi-scale feature fusion and prediction module firstly fuses multi-scale features and the final prediction score is generated by adopting three fully connected layers. The technical details above are presented in the sections as below.

2.2 Quadruplet Multi-model Feature Extraction

Due to distinct imaging mechanisms, there are vast differences in visual appearance between different multi-model images. To better extract and represent similar features and discriminative features between image patch pairs, the quadruplet multi-model feature extraction module is adopted. As shown in the left part of Fig. 2, it contains four branches with the same structure. When a new multi-modal image pair arrives, a Siamese sub-network formed by the top two branches sharing parameters takes them as input to encode the features irrelevant to imaging modality. And the bottom two branches unsharing parameters form a Pseudo-Siamese sub-network to encode image pairs' features related to imaging modality. For each branch, it consists of six convolution layers, whose details are shown in Table 1. Specially, an instance normalization is added before batch normalization of the first three convolution layers, which reduces the feature difference caused by the illumination variation and different imaging mechanisms. Finally, the feature maps generated by both Siamese sub-network and Pseudo-Siamese sub-network are concatenated together and then used as inputs to the SSML module.

2.3 Scale-Separative Metric LearningModule

Multi-scale feature integration strategy and proper attention mechanism are proved to be beneficial for increasing the accuracy of metric learning [Hou et al., 2021; Zhang et al., 2021]. Inspired by this fact, we propose a Scale-Separative Metric Learning Module (SSML), which encodes the similarity of different scale features of the input image patch pair respectively, and then integrates them together to increase the accuracy of metric learning. As shown in Fig. 1, the SSML module focuses on the relevant features and suppresses irrelevant features of imaging modality. We do not add any additional supervisory information for this module. It is only learned by the objective function using a multi-scale feature encoder mechanism.

As illustrated in the middle part of Fig. 2, the SSML module mainly contains three steps. Firstly, four convolution layers with different receptive fields (3 × 3, 5 × 5, 7 × 7, and 9 × 9) are utilized to generate four feature map groups. By splitting the input feature maps into four groups with different scales, SSML module can better measure the similarity of each scale. Then, coordinate attention (CA) [Hou et al., 2021] and Squeeze-and-Excitation attention (SE) is performed sequentially to encode coordination and channel-wise correlation for each feature map group. Finally, the four groups of feature map refined by CA and SE are regarded as the outputs of the SSML module. Visualization experimental result (Fig. 1) has shown that the similar features are highlighted for each scale by adopting the proposed SSML module, which makes the matching pairs and non-matching pairs becoming easier to be distinguished.

Specifically, given an input feature map F ∈ RL×H×W , the output of SSML module represented by F′ ∈ RL×H×W can be computed as:

where f n×n represents the convolution layer with kernel size of n × n. Fi ∈ RC×H×W ( i = 1, 2, 3, 4;C = L/4)

denotes one of four different scale feature maps. ⊗ is an element-wise multiplication. CA(·) represents the coordinate attention, SE(·) is the Squeeze-and-Excitation attention. F′i ∈ RC×H×W denotes scale-separative feature map.

The detailed structure of CA module presented in [Hou et al., 2021] is illustrated in the right bottom of Fig. 2. Given an input Fm ∈ RC×H×W , the spatial global average pooling kernel (H, 1) and (1,W) are performed along X-coordinate and Y-coordinate direction for each channel, respectively. Correspondingly, two feature maps with the size of C × H × 1 and C × 1 × W are generated. Then, these two feature maps are concatenated by convolving with the filter (1 × 1) along the channel. After that, a new feature map with the size of C/r × 1 × (H + W) is generate by performing Batch Normalization and Sigmoid operation. And through two sets of independent operations (a 1 × 1 convolution followed by a Sigmoid), the generated feature map is splited into two direction-aware attentions, i.e., Y-coordinate direction attention and X-coordinate direction attention with the size of C × H × 1 and C × 1 × W , respectively. Finally, these two attentions are multiplied with the input feature map Fm to generate the final output feature map. Based on the above descriptions, it decomposes coordinate attention into two one-dimensional feature encoding processes. In this way, it can capture long-range dependencies along one spatial direction and preserve high precise location information along another spatial direction. Therefore, CA module can capture long-range dependencies with precise positional information.

Mathematically, CA(·) is defined as follows:

where Fm(c, i, j) is the feature value of Fm at the position (c, i, j), in which c is the channel number, i and j denotes the X-coordinate and Y-coordinate, respectively. f 1×1 represents a 1 × 1 convolution layer. δ denotes Batch Normalization and σ is Sigmoid operator. Gh (c, h) and Gw (c,w) denote global pooling kernels to encode each channel along the X-coordinate and Y-coordinate direction, respectively. Gh and Gw denote the results after global pooling of the X-coordinate direction and Y-coordinate direction, respectively. Ft ∈ RC/r×1×(H+W) is the intermediate feature map that encodes spatial information in both X-coordinate and Ycoordinate direction. Then, Ft is split into two separate tensor along the spatial dimension, i.e. Fht ∈ RC/r×H×1 and Fwt ∈ RC/r×1×W , r is a reduction ratio for controlling the feature map size. Th ∈ RC×H×1 and Tw ∈ RC×1×W denote two attention vectors of the Y-coordination and Xcoordination direction, respectively.

Squeeze-and-Excitation can encode the relationship among feature channels by an attention vector, which is calculated among different channels of feature maps. The details about our implementation of SE model are as follows. The input feature maps firstly go through a global pooling layer, and output a vector with the same size as the number of input feature map channels. Then, a fully connected layer with 32 units followed by a ReLU activation function, a fully connected layer with C (the channel size of the input feature map) units, and a Sigmoid function are performed to generate the attention vector. Finally the input feature maps are weighted by the attention vector, and element-wise added with themselves to produce the channel-wise attentive features. Through the SE module, the feature maps contributing to the matching task are emphasized, and the others are restrained.

2.4 Multi-scale Feature Fusion and Prediction

Multi-scale feature fusion is very important for accurate prediction, which is denoted MFFP. Given the output feature map of SSML module F′ , we first carry out a coordinate attention operation on F′ . Then, an addition operation is performed to add F′ and CA attentive features together. After that, a 3 × 3 convolution is performed. Finally, three fully connected layers are adopted to predict the result. MFFP(·) can be described by the following formula.

where CA(·) is the coordinate attention as shown in Formula 2. F′ is the output feature map by SSML module. ⊕ is an element-wise addition operator. FC2, FC128, FC512 represent fully connected layers with 2, 128, and 512 units, respectively.

2.5 Loss Function

The image patch matching can be considered as a binary classification task (matching and non-matching). Cross-entropy loss is commonly used in classification task. In this paper, we adopt cross-entropy loss to train the network. In fact, we also considered other loss functions, including contrastive loss, hingle loss, and focal loss for experiments. However, these loss functions did not outperform cross entropy loss. Therefore, cross entropy loss Lenis more suitable for binary tasks like our image patch matching.

where y and ˆ y represent the ground truth and the predictive value, respectively.

3 Experiments

3.1 Datasets

To verify the effectiveness of the proposed method, we carry out experiments on public multi-model image datasets, including VIS-NIR, VIS-LWIR and Optical-SAR, as well as a single spectral multi-view stereo correspondence dataset named Brown.

1) VIS-NIR is a multi-modal image patch matching dataset [Brown and S¨usstrunk, 2011], which consists of more than 1,000,000 image patch pairs of visual spectrum and nearinfrared spectrum. One Half of these patch pairs are matching pairs and the other half are non-matching pairs. The size of each image patch is 64 × 64 pixels. Totally, there are nine categories in this dataset, i.e., Country, Field, Forest, Indoor, Mountain, Oldbuilding, Street, Urban and Water. Same as the methods [Baruch and Keller, 2021; Quan et al., 2021], the proposed model is trained on the Country category and test on the other categories. Since great differences among these categories, it is hard to obtain a good generalization performance on all test categories.

1. VIS-LWIR is a multi-modal dataset of visual spectrum (VIS) and long-wave infrared (LWIR) spectrum [Aguilera et al., 2015]. It contains 44 VIS-LWIR image pairs, which are strictly aligned in time and space. Similar to VIS-NIR dataset, we also crop image patch pairs from VIS and LWIR images centered on SIFT points. The patch size is 64 × 64 pixels. Following previous studies [Quan et al., 2021], our method is also trained on one half of the patch pairs and tested on the other half. There are significant appearance differences between VIS image and its corresponding LWIR image.
2. Optical-SAR is a multi-modal image patch dataset, which contains optical images and synthetic aperture radar (SAR) images. We generate these image patch pairs in the same way as VIS-NIR dataset. SEN1-2 [Schmitt et al., 2019] dataset contains 282,384 image pairs of optical images and corresponding SAR. Similar to [Quan et al., 2021], 583,180 image patch pairs are utilized for training and the other 248274 pairs for testing.

4) Brown is a single spectral multi-view stereo correspondence dataset [Brown et al., 2010]. It contains three subsets: Liberty, Notredame and Yosemite, which contains 100K, 200K, and 500K image patch pairs, respectively. For each subset, one half of the patch pairs are matching pairs with the same 3D point and the other half are non-matching pairs. The patch size is 64 × 64 pixels. Like the methods [Tian et al., 2017; Han et al., 2015; Zagoruyko and Komodakis, 2015], the proposed model is trained on one of three subsets and test on the other subsets.

3.2 Implementation Details

The code of the proposed model is implemented by Pytorch. It is trained with Adam optimizer, and the learning rate is 0.0001. The batch size is set to 128. The training time is set to 80 epochs, the momentum is initially set to 0.9 with the decay factor 0.9. The cross-entropy loss is adopted to train the network. To quantitatively evaluate the matching performance, the false positive rate at true positive rate (positive recall) equal to 95% (FPR95) is adopted.

3.3 Comparison with the State-of-the-Arts

1) Results on VIS-NIR Dataset: The proposed method is compared with the state-of-the-art image patch matching methods on the VIS-NIR dataset. Totally, there are seventeen comparison algorithms. Among these methods, SIFT [Lowe, 2004], GISIFT [Firmenichy et al., 2011], EHD [Aguilera et al., 2012], LGHD [Shechtman and Irani, 2007] are traditional hand-designed descriptor-based methods, which are limited by human prior-knowledge and have poor robustness and adaptability. PN-NET [Balntas et al., 2016a], L2-Net [Tian et al., 2017], and HardNet [Mishchuk et al., 2017] are descriptor learning-based methods. They focus on learning a representation that can enable the two matched features as close as possible, while making non-matched features far apart. Siamese [Simo-Serra et al., 2015], Pseudo-Siamese [Zagoruyko and Komodakis, 2015], 2-Channel [Zagoruyko and Komodakis, 2015], SCFDM [Quan et al., 2018], Hybrid [Baruch and Keller, 2021], Moreshet & K+ [Moreshet and Keller, 2021], Quan & W+ [Quan et al., 2021], AFD-Net [Quan et al., 2019], and MFD-Net [Yu et al., 2022] are all metric learning-based methods. As shown in Table 2, our method outperforms other comparison methods. Compared with the second-best method MFD-Net [Yu et al., 2022], the mean FPR95 value of our method is reduced by 0.15. Compared with Hybrid [Baruch and Keller, 2021], which is similar to our baseline and also has both Siamese sub-network and Pseudo-Siamese sub-network, the mean FPR95 value of our method is reduced by 1.79. It demonstrates that our method can effectively extract and measure the similarity of multimodel image patches by using the proposed SSML module and fusion strategy.

1. Results on VIS-LWIR Dataset: As shown in Table 4, we compare the proposed method with five state-ofthe-art methods on VIS-LWIR dataset, including Siamese [Simo-Serra et al., 2015], Pseudo-Siamese [Zagoruyko and Komodakis, 2015], 2-Channel [Zagoruyko and Komodakis, 2015], and Hybrid [Baruch and Keller, 2021]. Our proposed method achieves an excellent performance. The mean FPR95 value of our method is 1.56.
2. Results on Optical-SARDataset: As shown in Table 4, although there are significant appearance differences between optical images and SAR images, the mean FPR95 value of our method is 0.65. Compared with the other methods, the performance of our method is very excellent.
3. Results on Brown Dataset: To demonstrate the generalization ability of the proposal, we also test and compare SSML-QNet with other methods on Brown, i.e., a single spectral multi-view stereo correspondence benchmark dataset. As shown in Table 7, compared with the secondbest method Moreshet & K+ [Moreshet and Keller, 2021], the mean FPR95 value is significantly improved by 0.32. This improvement demonstrates that our method can effectively encode and evaluate the similarity between images of different views and has a better generalization ability.

From the above four experiments, the proposed method achieves much better performance than the other methods. It can demonstrate that our model is effective not only for multi-model images, but also for single-modal images. Note that compared with VIS-NIR and Optical-SAR datasets, the mean FPR95 value on VIS-LWIR dataset is lower. One possible reason is that the correspondence between VIS image and LWIR image is more diverse, since the thermal radiation energy of observed objects will vary with many factors, such as object status, material quality, environment temperature, and observation distance.

3.4 Ablation Study

To verify the effectiveness of each module, we conduct ablation experiments. "BL" means our baseline, consisting of only quadruplet multi-model feature extraction and fully connected layers. "SSML" is our scale-separative metric learning module. "CA" means adopting coordinate attention in the feature fusion stage. "Sia" and "Pse-Sia" represents only considering Siamese sub-network and Pseudo-Siamese subnetwork of our baseline, respectively.

As shown in Table 8, by adding SSML and CA into our baseline respectively, the mean FPR95 value is reduced by 1.28 and 0.23. By adding both of SSML and CA modules, the improvement becomes more significant, reaching 1.35. It can demonstrate the effectiveness of the proposed SSML module and the fusion way by using CA. When only considering Siamese or Pseudo-Siamese sub-network, the mean FPR95 value is 1.24 and 1.81, respectively. While considering both of them, the mean FPR95 value is significantly reduced to 0.73. Therefore, our quadruplet network structure is effective for multi-model image matching task.

3.5 Fusion Strategy Comparison

To verify the effect of our fusion strategy (CA), four typical fusion ways are adopted to compare with our method. These four comparison fusion ways are simple concatenation, element-wise sum fusion, multi-scale spatial feature attention module (MSSAM) [Zhang et al., 2022], and Transformer encode module, which are denoted as "Concat", "Sum", "MSSAM" and "TF", respectively. Similar to CA, Transformer encode module can also establish long-distance dependencies and obtain global context information. MSSAM can automatically learn the weight map of each scale feature group and effectively fuse the spatial detail information of different scale feature groups. We conduct an experiment to replace CA by other comparison fusion way based on our SSML-QNet, respectively. Specifically, the Transformer encoder has two layers and each layer consists of two multihead attention blocks. Same as Moreshet et al. [Moreshet and Keller, 2021], the VIT pretrained model is loaded for Transformer encoder To improve the image matching performance. The experiment results in Table 6 show that compared with Transformer encoder and MSSAM, the mean FPR95 value of our method is improved by 0.49 and 0.55 by using CA, respectively. Therefore, CA is more suitable for fusing the features extracted by SSML module.

3.6 Cross-dataset Transferring Performance

To evaluate the cross-dataset transferring performance of the proposed method, we select one dataset for testing, and adopt other three datasets to train three models, respectively. The experimental results are shown in Table 3 and Table 5. We can see that except the model trained on Optical-SAR dataset, the other models can achieve good cross-dataset transferring performance. The possible reason is that the imaging mechanism and features of SAR images are far apart from other modal images and our model effectively learn the features related to imaging modality through quadruplet multi-model feature extraction module. While, the models trained on Liberty of Brown and VIS-LWIR performs better on OpticalSAR dataset and gains the mean FPR95 value of 1.67 and 2.36, respectively. Experimental results demonstrate that our network has good generalization performance and robustness.

3.7 Computational Efficiency andMemory Usage

Table 9 shows the comparison of computation efficiency, memory usage and matching performance. All compared methods are tested on the same workstation (one RTX 3090Ti). Our method achieves the best FPR95 score while achieving competitive computational efficiency.

3.8 Image Matching Visualization Experiment

This section analyzes the visualization results based on the matching point pairs learned from the proposed SSML-QNet. We compare the proposed method with baseline method on three multi-modal dataset. As shown in Fig. 3, our method achieves an excellent performance. The visualization results of the baseline model are illustrated in Fig. 3(a), and that of the proposed SSML-QNet model are shown in Fig. 3(b). All modal images are processed by geometric transformation (rotation = 180◦ , translation = 2 pixels). The green lines represent matches and the red lines denote non-matches. The experimental results show that the proposed SSMLQNet achieves good results in VIS-NIR and VIS-LWIR image pairs, but achieves relatively less matching point pairs in SAR. There are two possible reasons. Firstly, the imaging mechanism and characteristics of SAR images are quite different from those of other modal images, which leads to poor generalization effect. Secondly, it may be difficult to detect more robust feature points in SAR images due to the influence of traditional detection operators in the early feature point detection.

4 Conclusion

In this paper, we proposed a scale-separative metric learning quadruplet network for multi-modal image patch matching, named SSML-QNet. It can effectively extract cross domain consistent features and measure feature similarity. The experiments show that our proposal method performs much better than the-state-of-the-art methods on three multi-modal datasets (VIS-NIR, VIS-LWIR and Optical-SAR ) and a single modal Brown dataset, and also has excellent cross-dataset transferring performance.

Acknowledgments

This work was supported by the National Natural Science Foundation of China under Grants U19B2037, 61971356, 62201473 and 62201467, by the National Natural Science Foundation of Shaanxi province under Grant 2021KWZ-03 and 2022JQ-686. Thank the providers of the dataset.