A Multi-Constraint Similarity Learning with Adaptive Weighting for Visible-Thermal Person Re-Identification

1 Introduction

Traditional person re-identification (ReID) [Zhong et al., 2018; Zheng et al., 2017; Yang et al., 2020b] aims at matching a specific query person from large-scale gallery images captured by RGB cameras. However, the quality of the visible (RGB) image will significantly decrease under poor illumination (e.g. night-time). To overcome this issue, many thermal surveillance cameras have been deployed to capture thermal images. Hence, we encounter the task of matching person samples between visible and thermal cameras, which is known as visible thermal person re-identification (VT-ReID).

∗Corresponding authors

Apart from the long-standing inter-and intra-class variations suffered in the traditional REID, VT-ReID further needs to deal with the large inter-modality discrepancy (Figure 1(a)).

The key solution for VT-ReID is learning a shared embedding space in which the features of two modalities can be matched. Therefore, different metric-learning-based methods have been proposed to reduce the inter-modality discrepancy and intra-modality variations. These methods can be mainly divided into two aspects based on the similarity constraint utilized during the learning phase: (1) Instance-toInstance (I2I): constraining the inter-and-intra modality similarity relationship of different training samples at the instance level [Feng et al., 2019; Hao et al., 2019c], and (2) Center-toCenter (C2C): constraining the relationship between the class centers from two different modalities [Liu and Tan, 2020; Zhu et al., 2020]. Despite their success, the former I2I-based methods mainly focus on constraining the relationship between instances while ignoring the overall global class-level characters. They will be easily affected by noise or hard sample-pairs, resulting in inferior performance and generalization. On the other hand, the latter C2C-based methods indirectly optimize the feature similarity and significantly ignore intra-class variations of different samples. This may lead the situation that the similarity between class centers of two different modalities are optimized, but the discrepancy of cross-modality (SI2I) still maintain large (Figure 1(b)). In a nutshell, these single constraint metric learning-based methods are insufficient to explore the rich data relationships among the large inter-modality discrepancy and intramodality variations. Besides, most of these methods treat different informative pairs equally and fail to explore the crossmodality informative pairs for similarity learning, incurring an inefficiency and ineffectiveness of optimization.

To overcome the shortcomings of previous methods, we propose a Multi-Constraint (MC) similarity learning method to fully explore the various data relationship among cross-modality samples. In this study, we mainly consider the following three constraints, i.e., Center-to-Center (C2C), Center-to-Instance (C2I), and Instance-to-Instance (I2I). Specifically, the C2C constraint mainly enforces the cross-modality intra-class centers should have higher similarity than inter-class centers, whose primary goal is to bridge the large cross-modality gap. The C2I constraint further enforces that each sample should be close to its corresponding class center of the other modality to reduce the intra-modality variations. Additionally, the I2I constraint force the intermodality intra-class discrepancy between samples should be smaller by pulling them close to each other. These three constraints are jointly utilized to reduce the intra-class discrepancy and enlarge the inter-class distance.

In addition, different informative pairs have different effects during similarity learning, and we devise an Adaptive Weighting Loss (AWL) function to effectively implement the multi-constraint. In the AWL, we first use an adaptive margin to select cross-modality informative pairs and discard those less informative pairs. Then, we exploit a sigmoid variant function to adjust pair weights based on their similarity. By utilizing this pair mining and pair weighting scheme, we will encourage the network to pay more attention to optimize the informative pairs during training.

To sum up, our main contributions are as follows: 1) We propose a Multi-Constraint (MC) similarity learning framework to reduce the inter-and intra-modality discrepancy in the VT-ReID by jointly considering three constraints (I2I, C2I, and C2C). 2) We devise an Adaptive Weighting Loss (AWL) function to leverage the cross-modality informative pairs for model training in a more effective manner. 3) Experiments on two datasets demonstrate the mutual benefits of the aboveproposed components and show superior performance over state-of-the-art methods.

2 RelatedWork

The existing methods of VT-ReID can be mainly divided into three groups. 1) Feature extraction based methods. [Wu et al., 2017] first introduce SYSU-MM01, a visible thermal dataset, and they study a deep zero-padding method for evolving domain-specific nodes in the network. [Ye et al., 2018b] introduce a two streams network to learn crossmodality embedding. To handle the cross-modality discrepancy, [Lu et al., 2020] introduce a cross-modality sharedspecific feature network, [Ye et al., 2020] develop a dynamic dual-attentive aggregation network, and [Yang et al.,

2020a] propose a bi-directional random walk scheme network. These methods mainly focus on exploring a crossmodality network to extract discriminative feature. 2) Metric learning based methods [Dai et al., 2018; Ye et al., 2018a; Hao et al., 2019c; Feng et al., 2019; Hao et al., 2019a; Hao et al., 2019b; Ling et al., 2020; Zhu et al., 2020; Liu and Tan, 2020] are proposed to learn an embedding space that makes the intra-class samples close to each other. [Hao et al., 2019c] and [Feng et al., 2019] only consider the relationship between instances. Hence, [Zhu et al., 2020] and [Liu and Tan, 2020] are proposed to optimize the relationship between class centers of two modalities. However, these methods with a single relationship constraint are not enough to express the rich data relationships. Therefore, we propose a Multi-Constraint (MC), which adopts the relationships of I2I, C2I, and C2C constraints to optimize the feature. 3) Image generation based methods. [Wang et al., 2019c; Wang et al., 2019a; Li et al., 2020] use a variational autoencoder and generative adversarial network to generate fake images to bridge the modality gap. These methods may contain id-unrelated factors, which undermines the performance, so [Choi et al., 2020], [Wang et al., 2020], and [Pu et al., 2020] combine a disentangle representation with VAE for robust cross-modality matching. These models are commonly difficult to train due to complex generative adversarial networks to generate fake images.

3 ProposedMethod

In this section, we will introduce the details of the proposed framework. As shown in Figure 2, the proposed framework mainly includes the base feature extractor, part-based embeddings, and training loss function. The training loss function consists of a multi-constraint based metric learning loss function LMC and the identification loss function LID.

3.1 Base Feature Extractor and Part-Based Embeddings

Due to the large modality discrepancy, we first use two independent modules to compute the modality-specific features at the shallow layers. Then, we use parameter sharing module to learn the modality-shareable feature to embed the two modalities into the same subspace. The independent modules have the same structure as the first convolution block and residual block 1 (Layer1) in ResNet-50 [He et al., 2016], and the share module is the same as the residual block 2-4 (Layer2-4).

Besides, the PCB method [Sun et al., 2018] has demonstrated the effectiveness of using local parts to increase feature discrimination. Therefore, we exploit an efficient module to extract two parallel part features at different scales. Firstly, we divide the feature maps into six non-overlapping parts and three non-overlapping parts along the vertical direction. Then, we compute the feature of each local part by a global pooling and a convolution. The final local feature dimension of the branch with six parts and three parts are 256 and 512, respectively. Finally, the holistic features for these two-part branches are simply concatenating the corresponding local part features. In the testing stage, we concatenate these two holistic features ([0.6fh3, 0.4fh6]) to match the visible and thermal images.

3.2 The Multi-Constraint Loss

For learning a more discriminative feature, we propose a general metric learning by considering multiple cross-modality constraints, i.e., Center-to-Center (C2C), Center-to-Instance (C2I), and Instance-to-Instance (I2I). Specifically, the I2I and C2I constraints are leveraged to optimize the holistic features, the C2C constraint is utilized to constrain the local part features. These three constraints jointly reduce the intermodality discrepancy and intra-class variations, and an illustration of them are shown in Figure 3.

Assume xvi and xti are a sample of class i from visible modality and thermal modality, respectively. Their class center cvi and cti can be computed by:

where Ci is the number of samples of class i for a modality in a mini-batch. f v j and f t j are the feature of xvj and xvj .

1) C2C Constraint. For handling the large inter-modality discrepancy, we use C2C constraint enforces the crossmodality intra-class centers have higher similarity than interclass centers. Given a class center anchor of one modality (i.e., cva ), we select a positive and negative class center form another modality (ctp and ctn ). Then, we reduce the distance between cva and ctp , and enlarge the distance between cva and ctn (Figure 3(a)). Given the selected positive and negative class center pairs, the loss of C2C constraint is defined as:

where the PC2C and NC2C are positive and negative pairs satisfying C2C constraint, respectively. The pair-based loss function L(P, N) will be introduced later.

2) C2I Constraint. We further enforces that each sample should be close to its corresponding class center of the other modality to reduce the intra-modality variations. Given a class center anchor of one modality (i.e., cva ), we select a positive and negative samples form another modality (xtp and xtn ). Then, we force positive samples xtp to approach their own inter-modality class center cva , and enlarge the distance between cva and xtn (Figure 3(b)). The loss of C2I constraint is defined as follows:

where the PC2I and NC2I indicate the positive and negative pairs of the C2I constraint, respectively.

3) I2I Constraint. We utilize the I2I constraint to increase the similarity of two matching samples from the two different modalities. Given a anchor sample of one modality (i.e., xva ), we select a positive and negative samples from another modality (xtp and xtn ). Then, we pull the positive pairs close to each other and push the negative pairs away from each other. The loss of I2I constraint is defined as follows:

where the PI2I and NI2I indicate the positive and negative pairs of the I2I constraint, respectively.

By jonitly considering these three constraints, the equation of MC is defined as follows:

where α, β, and ω are hyper-parameters.

3.3 AdaptiveWeighting Loss

For implementing the L(P,N) used in the multi-constraints, we devise a new adaptive weighting loss based on the MultiSimilarity (MS) Loss [Wang et al., 2019b] for the crossmodality VT-ReID task.

Revisit Multi-Similarity (MS) Loss

The MS loss optimizes the feature space based on informative pairs mined based on three similarities: Self-similarity, Negative relative similarity, and Positive relative similarity. In the MS loss, And they propose a MS loss to optimize sample pairs by considering these three similarities. Set xi as an anchor, a negative and positive pair {xi, xj} are selected if the pair similarity Sij satisfies the following conditions:

where m is a given margin.

The MS loss function can be calculated by:

where C is the number of selected anchors. λp, λn, and λ are hyper-parameters.

Despite the effectiveness of MS loss in the standard single modality task, the MS will suffer the issue of discarding many informative pairs for the cross-modality VT-ReID. As shown in figure 4 (a), the negative pair with maximum crossmodality similarity is the av , nt . The similarity of positive pair av , pt is larger than the av , nt which does not satisfy with the condition in Eq. 6. Then, this informative positive pair will be discarded and not be used for optimization. To deal with this issue, we propose a new pair mining strategy for selecting cross-modality informative pairs. Besides, we further devise a weighting function to adjust the weights of selected pairs.

Pair Mining

For a given visible modality anchor av , the pt and nv are a positive sample and a negative sample from the intermodality, respectively. Then we will decide whether selecting pt or nv to form a positive pair or a negative pair of av based on following conditions.

For the selection of positive pair, we first find the closest negative intra-modality sample nv of pt . If the similarity of {nv , pt } is higher than the {av , pt }, it means that the similarity of {av , pt } is not optimized as shown in Fig.4. We will select {av , pt } as a positive pair. Formally, the condition for the positive pair selection with an addition margin can be denote as,

where m is a margin.

In a similar manner, a inter-modality negative pair {av , nt } is select if S(av , nt ) satisfies following conditions:

where the nv and nt are belong the same label.

Pair Weighting

After selecting the informative positive and negative pairs, we will adaptively assign large weights for more informative pairs, impelling the model to pay more attention to the informative pair during training. In this step, we exploit a sigmoid variant function to adaptively adjusting weighting based on their similarity with the anchor. Specifically, the weights are inversely correlated to positive pairs similarity and positively correlated to negative pairs similarity. The adaptive weighting function for positive pairs and negative pairs are defined as follows:

where δ indicates the sigmoid function.

We then design the Adaptive Weighting Loss (AWL) based on the Eq 7, by considering the selected informative pairs and their adaptive weights. The final AWL is defined as:

where C is the number of selected anchors. λ is a hyperparameter.

3.4 Final Loss Function

Finally, we train our proposed framework by combining the Multi-Constraint (MC) loss LMC and the identity loss LID in an end-to-end manner. The loss is as follows:

where θ represents the parameters of the model, and δ is a hyper-parameter controlling the influence of the two losses.

4 Experiments

4.1 Experimental Settings

Datasets. We evaluate our proposed methods on two publicly available VT-ReID datasets (SYSU-M001 [Wu et al., 2017] and RegDB [Nguyen et al., 2017]). SYSU-M001 contains 287,628 RGB images and 15,729 infrared images captured by four RGB cameras and two thermal cameras. The training set consists of 22,258 RGB images and 11,909 infrared images from 395 identities. RegDB contains 4,120 RGB images and 4,120 infrared images. There are 412 identities, where 206 identities for training and others for testing.

Evaluation metrics. The Cumulative Matching Characteristics (CMC) and mean Average Precision (mAP) are used to evaluate the performance. For the CMC, we only report the rank-1 (R1) accuracy.

Implementation details. The baseline network is constructed based on [Liu and Tan, 2020], which optimizes part features with the identity loss and hc-tri loss. We also evaluate the performance of a baseline framework without part ([Ye et al., 2018b]) to demonstrate the generalization of our proposed. The training batch is set to 96 (48 RGB images and 48 infrared images from 6 person IDs) and 64 (32 RGB images and 32 infrared images from 8 person IDs) for SYSUMM01 and RegDB dataset, respectively. The input images are resized to 288 × 144 × 3 for both RGB and infrared images. We use the SGD optimizer for training with an initial learning rate of 0.01, and train the model for 80 epochs. We divide the learning rate by 10 after every 10 epochs. The weights α, β, and ω in Eq. 5 are set to 0.5, 1.0, and 0.2, respectively. The pair mining margin m in Eq. 8 is set to 0.2. λ in Eq. 11 is set to 0.5. The weights γ in Eq. 12 is set to 1.

4.2 Comparison with State of The Art

We compare our proposed method with recently published state of the arts on SYSU-MM01 and RegDB datasets. The comparison includes: feature extraction based methods (cmSSFT with single query [Lu et al., 2020], DDAG [Ye et al., 2020], SIM [Jia et al., 2020]), metric learning based methods ( DGD [Feng et al., 2019], DFE [Hao et al., 2019a], CML [Ling et al., 2020], HC-TRI [Liu and Tan, 2020]), image generation based methods (D2RL [Wang et al., 2019c], AlignGAN [Wang et al., 2019a], Xmodal [Li et al., 2020], Hi-CMD [Choi et al., 2020], PIG [Wang et al., 2020], DGVAE [Pu et al., 2020]).

The results on SYSU-MM001 and RegDB datasets are shown in Tabel 1 and Tabel 2, respectively. We can see that proposed method outperforms the state-of-the-art methods on both datasets. Specifically, our method achieve rank-1 accuracy = 64.82 and mAP accuracy = 60.81 on SYSU-MM01 for the single-shot setting of the all-search mode, and, rank-1 accuracy = 93.83 and mAP accuracy = 87.55 on RegDB for visible to thermal setting.

4.3 Evaluation

In this section, we evaluate the effectiveness of each component in our proposed method on SYSU-MM01 (single shot setting of all-search mode) and RegDB (thermal to visible setting).

Comparison over the baseline. To evaluate the effectiveness and demonstrate the generalization of our proposed Multi-Constraints (MC), we conduct experiments of ablation study under two settings: the baseline with part (w/ Part) [Liu and Tan, 2020] and without part (w/o Part) [Ye et al., 2018b] on both datasets. The results show in Table 3. We can see that the proposed MC can significantly improve the performance over the baseline on both datasets for two settings. The observation verifies the effectiveness and generalization of our proposed MC for the challenge cross-modality VT-ReID.

Investigation of three constraints in MC. In Table 4, we further evaluate the effectiveness of different constraints in the MC. From the table, we can find that considering every single constraint can significantly improve the performance over the baseline. Moreover, combining two of them can further improve the performance. Additionally, the highest accuracy is obtained by jointly considering the three constraints. These results demonstrate the complementary and mutual benefits of the three constraints in MC.

Evaluation of the components in AWL. To evaluate the effectiveness of the proposed Adaptive Weighting Loss (AWL), we conduct a controlled experiment with MS loss [Wang et al., 2019b]. The results are reported in Table 5. By replacing the pair mining in MS loss with ours (+MS (our mining) vs. +MS (MS mining)) and the pair mining in our AWL with the one in MS (+AWL (MS mining) vs. +AWL), we can observe a significant performance boost for the former case and a notable degeneration for the later. Then, by comparing the AWL without pair weighting to AWL (+AWL (w/o weighting) vs. +AWL), we can find that AWL with pair weighting gain a higher result than AWL without pair weighting. These results demonstrate the effectiveness of the proposed pair mining and pair weighting strategies.

Comparison of the MC with other metric-learning methods. In order to demonstrate the superiority of our proposed MC, we further compare our MC with other various different metric-learning loss functions, including Triplet Loss (Hard) [Hermans et al., 2017] (I2I), MS Loss with crossmodality pair mining (cm-Mining) [Wang et al., 2019b] (I2I), Hc-Tri Loss [Liu and Tan, 2020] (C2C), and CML Loss [Ling et al., 2020] (C2C & C2I). Notice that these loss functions usually consider one or two constraints. By comparing our MC with them, our method outperforms them by a large margin. These results suggest considering more comprehensive data relationships is essential for overcoming the large intermodality discrepancy and intra-and-inter class variations in VT-ReID.

5 Conclusion

In this paper, we propose a Multi-Constraint (MC) for VT-ReID, which jointly considers Instance-to-Instance (I2I), Center-to-Instance (C2I), and Center-to-Center constraints (C2C) to optimize feature similarity to reduce inter-modality discrepancy and intra-modality variations. Furthermore, we implement MC with an Adaptive Weighting Loss (AWL) to promote the model training. Extensive experiments on two VT-ReID datasets demonstrate the superior performance of the proposed over the state of the art methods.

Acknowledgements

This work is supported by the National Nature Science Foundation of China (No. 61876159, No. 61806172, No. 62076116, No. U1705286), the China Postdoctoral Science Foundation Grant (No. 2019M652257).