Base-Detail Feature Learning Framework for Visible-Infrared Person Re-Identification

1 Introduction

Person re-identification (ReID) aims to retrieve a target identity from gallery images captured by different cameras [Liu et al., 2022] and has recently demonstrated significant advancements in the fields of security and public surveillance [Ye et al. ]. However, most existing methods [Cao et al., 2023][Wang et al., 2022][Yan et al., 2021] primarily focus on utilizing RGB images captured by visible (VIS) cameras during the daytime, which are inadequate for accommodating 24-hour scenarios that involve infrared (IR) images captured by IR cameras. To address the substantial cross-modality gap and facilitate operation in all-day scenarios, visible-infrared person re-identification (VIReID) methods [Chen et al., 2022][Park et al., 2021] have been developed, enabling the matching of IR (RGB) images given an interest in a specific RGB (IR) pedestrian image.

Figure 1: Motivation of the proposed BDLF, which focuses on sufficiently mining the modality-shared and modality-specific knowledge simultaneously and are not applicable for additional auxiliary data.

The existing research on VIReID can generally be categorized into two principal methods: extracting distinguishing modality-shared features from VIS and IR modalities[Park et al., 2021][Zhang and Wang, 2023] and compensating for modality-specific or modality-shared features [Zhang et al., 2022a]. As shown in Figure 1(a), the former method aims to reduce cross-modality discrepancies by aligning comprehensive cross-modality features into a common semantic space. However, it neglects to leverage modality-specific and shared cues, which inevitably leads to performance bottlenecks. The latter approach, depicted in Figure 1(b) can be further divided into embedding-level and image-level methods. These methods generate compensatory knowledge in the embedding space and at the pixel level respectively, using auxiliary models(e.g., GANs[Goodfellow et al., 2014], segmentation networks, part alignment networks, etc.). However, these methods typically introduce losses and noise into the generated features or require additional data processing by other models, making them less effective and convenient. Consequently, advancing the development of VIReID to a more comprehensive level remains a significant challenge.

Inspired by the analyses presented above, it is essential to recognize that modality-shared information, such as the contour and movement characteristics of pedestrians, can be considered base features. In contrast, modality-specific information, including the color and texture details of the RGB modality and the thermal characteristics of the IR modality, can be regarded as detail features. Both types of them should be integrated and utilized effectively together. Therefore, in this paper, we propose a novel Base-Detail Feature Learning Framework (BDLF), as shown in Figure 1(c). This framework is designed to extract modality-shared base features and modality-specific detail features from the original images with minimal additional computational costs, while jointly optimizing modality-shared, modality-specific, and comprehensive features.

The proposed BDLF comprises a modality-specific detail feature extraction (DFE) module and a modality-shared base embedding generation (BEG) block, which ultimately combine the optimized features collected. Inspired by [Zhao et al., 2023], we designed the DFE module to mine the modality-specific detail information losslessly. Subsequently, the BEG block derives modality-shared base features. To fully capture both specific and shared information, we proposed a novel specific-shared knowledge distillation(SKD) loss. It encourages the detail (base) features to effectively incorporate modality-specific (modality-shared) knowledge by imposing a constraint on the correlation that the crossmodality detail and base features should exhibit. Specifically, it ensures that the correlations across RGB and IR modalities are indistinct and notable, respectively. Perspectives in [Feng et al., 2023] explain that the independent decomposition of features can maximize the mutual information of subfeatures; therefore, we introduced an independence constraint in the semantic space between the derived detail and base features. This indicates that the base feature exclusively encompasses modality-shared knowledge, while the detail feature contains modality-specific information. In summary, the main contributions of our work are as follows:

• A novel correlation optimization method is proposed that effectively generates both modality-shared and modality-specific features using a non-parametric approach, rather than relying on classifiers.
• We propose an end-to-end Base-Detail Feature Learning Framework (BDLF) for VIReID that integrates extracts of modality-shared base knowledge and modalityspecific detail knowledge.
• Extensive experiments have demonstrated that the proposed BDLF outperforms other state-of-the-art methods for the VIReID task on the SYSU-MM01, RegDB, and LLCM datasets.

2 RelatedWork

The main idea for solution VI-ReID task is decreasing the notable discrepence across VIS and IR modalities, thereby the existing methods consist of aligning the cross-modality features and utilizing the auxiliary data or features generated by other models.

The alignment of feature representation methods seeks to convert cross-modality features into a unified semantic space through either metric learning techniques [Liu et al., 2022] [Park et al., 2021] [Luo et al., 2019] or by enhancing networks with more effective feature extraction components [Zhang and Wang, 2023] [Sarker and Zhao, 2024]. However, these approaches ultimately encounter performance bottlenecks due to the loss of modality-specific information.

The methods for utilizing auxiliary information produced by other models are proposed to enhance identifiable knowledge. GAN-based methods [Zhang et al., 2022a]d[Wang et al., 2020] generate compensatory features at either the image level or the embedding level to simulate features from another modality. XIV [Li et al., 2020] introduces the Xmodality generated by a lightweight auxiliary network to decrease discrepancies between the two modalities. LUPI [Alehdaghi et al., 2022] establishes an intermediate domain between VIS and IR modalities. Furthermore, it generates images that belong to this intermediate domain to guide the network in acquiring more discernible information. SGIEL [Feng et al., 2023] innovatively adopts the shape knowledge of identity generated by segmentation models to enrich supplementary information. TMD [Lu et al., 2024] generates style-aligned images to minimize differences at the image level, subsequently aligning cross-modality features to eliminate discrepancies in feature distribution and instance features. However, this remains a challenging field of research because these methods either inevitably introduce information distortion during the generation process or fail to completely capture modality-specific and modality-shared information.

3 Methodology

3.1 Overall Framework

The pipeline of our proposed method, referred to as BDLF, is illustrated in Figure 2. This method utilizes a single-stream ResNet-50 network[He et al., 2016a] as its backbone. The intermediate features ZM , which pass through a portion of the backbone, are fed into the proposed detail feature extraction (DFE) module to yield detail features Z D . Additionally, the base feature Z B is generated by inputting the output Z from the backbone into the proposed base embedding generation (BEG) block. A novel specific-shared knowledge distillation (SKD) loss is proposed to ensure that the generated detail(base) features contain as much modality-specific (modality-shared) knowledge as possible, thereby effectively leveraging modality-specific and shared information. Furthermore, we construct a modality-shared feature Z F using a cross-modality feature fusion method to optimally supplement the base features. During the inference phase, only the comprehensive feature Z yielded by the backbone is used for performance evaluation. This is because the proposed DFE and BEG modules effectively enhance the comprehensive feature by incorporating additional detail and base information.

Given an identity image from either the visible or infrared modality, VIReID intends to identify the most similar sequence of that identity in another modality. Let the training set {XV ,XI} consist of B identities, with each identity including P samples. Therefore, XV = {xV b,p , b = 1, ..., B; p = 1, ..., P } symbolizes the set of visible images, while XI = {xI b,p , b = 1, ..., B; p = 1, ..., P } denotes the set of infrared images. As illustrated in Figure 2, the VIS and IR images are processed through the backbone network, i.e,

where Efore (·) and Erear (·) are the former and latter blocks of the backbone network, the embeddings ZM ∈ RB×C′×H′×W ′ and Z ∈ RB×C×H×W denote the intermediate and complete outputs from the backbone for the VIS and IR modalities, cat(·) refers to the concatenation operation along the batch dimension.

3.2 Specific-shared Knowledge Distillation

We observe that the similarity of base information, such as contours and movements, between the VIS and IR modalities is noticeable. In contrast, the similarity of detail information including color, texture, and thermal details between the two modalities is suppressed. Inspired by [Zhao et al., 2023], as shown in Figure 3, the base and detail features can be generated by increasing and reducing the correlation between the two modalities respectively. Based on this, we propose a novel specific-shared knowledge distillation (SKD)

loss, which is numerically smoother and easier to optimize, formulated as follows:

in which Z B V/I denotes the base features generated by the proposed BEG block, and Z D V/I denotes the detail features extracted from the proposed DFE module. Corr(·) is the Pearson correlation coefficient operation, while γ represents a constant that ensures the denominator remains non-zero. According to optimize the SKD loss, the correlation between the VIS and IR modalities of both base and detail features(i.e, Corr(Z B V ,Z B I ) and Corr(Z D V ,Z D I ) in formula (2)) is simultaneously increased and decreased. This approach allows the proposed DFE module to extract embeddings rich in detailed knowledge. Consequently, the proposed BEG block is capable of generating base embeddings that contain a greater amount of modality-shared knowledge.

3.3 Detail Feature Extraction

The proposed DFE module aims to acquire detail features that imply modality-specific information from the intermediate embedding ZM V/I by utilizing a series of invertible neural network (INN) blocks[Zhao et al., 2023][Dinh et al., 2017][Zhou et al., 2022], which can effectively preserves detailed characteristics and mitigates information loss during feature extraction by making its input and output embeddings are mutually generated. Taking the VIS case as an example, we obtain the input for the DFE module, ZV (1 : c)M and ZV (c + 1 : C)M ∈ R B2 × C′ 2 ×H′×W ′ by splitting ZV M in half along the channel dimension. The transformations in each block can be denoted as follows:

Here, ZM V,k is the input of the kth (k ∈ 1, ...,K) block, Fi(·)(i ∈ 1, 2, 3) denotes the convolution blocks. The symbol · indicates element-wise multiplication of matrices, LN(·) represents layer normalization in the Lite Transformer[Wu et al., 2020] and cat(·) is the channel concatenation operation. The IR situation can be easily derived by substituting I for the subscript V in the aforementioned formulas.

At the final stage of DFE, we consider the detail features of both modalities integrally, as concatenating the extracted detail embeddings from the two modalities can help reduce computational complexity. Therefore, we feed the extracted detail embeddings into a cross-attention-based transformer to facilitate cross-modality reasoning and information exchange. This process enables the detail feature integration of knowledge from various modalities and allows for a more effective focus on distinguishable information, thereby enhancing the robustness and efficacy of semantic representation. Inspired by[Li et al., 2022][Xu et al., 2024], the transformer can

where GAP(·) is the global average pooling operation, ZM V/I ∈ R B2 ×C′ denotes the embeddings after pooling. Wq, Wk , and Wv are the learnable parameters for DFE, and softmax(·) indicates the calculation of the softmax by row.

Ultimately, the detail feature Z D ∈ RB×C′ produced by the proposed DFE module is obtained by concatenating the VIS and IR detail embeddings along the batch dimension:

With the proposed SKD loss formulated in formula (2), the extracted detail feature Z D can significantly enrich modalityspecific detail knowledge. Thus a private classifier CLSD that is specially designed for the detail feature Z D is constructed, alongside a communal classifier CLSB that processes the base embeddings and the comprehensive feature Z obtained from formula (1), as illustrated in Figure 2. Furthermore, the commonly used id loss[Luo et al., 2019] driven by cross-entropy (ce(p, q) = - ∑ni=1 qilog(pi)) was applied to strengthen the distinguishable information of detail feature Z D , i.e,

Since there are differences in the distribution of classification results between the detail feature Z D and the comprehensive feature Z, this misalignment may impede our goal of enhancing the representation ability of Z leveraging detailed knowledge. Therefore, we constrain the probability distribution predicted from Z D to align with the distribution from Z, ensuring that their semantic representations are consistent. This process can be expressed as follows:

The total loss of the proposed DFE module can be obtained by combining formulas (6) and (7):

3.4 Base Embedding Generation

The proposed BEG block is designed to produce the base embeddings from Z utilizing the acquired detail feature Z D . Take notice that there are significant semantic differences between modality-specific detail information such as color and texture and modality-shared base information, which includes movements, contours, and so on. For this reason, inspired by [Feng et al., 2023], we have developed a method to ensure that the detail(base) features can only contain modality-specific(modality-shared) distinguishable knowledge, thereby maximizing the collection of both modality-specific and modality-shared information. Furthermore, the proposed DFE and BEG blocks can learn these two categories of knowledge simultaneously without interfering with each other. Based on this premise, we consider the detail and base embeddings to be independent of each other, i.e, Z D ⊥ Z B . According to the approach of making ¯ Z D comprehensively converge to Z D and impose the independence restriction between the detail and base embedding, the proposed BEG block can then generate modality-shared base embedding by excluding detailed knowledge from Z in the semantic space,i.e,

in which Z is the output of backbone network, I is the identity matrix, → denotes approximating, ¯ Z D and Z B ∈ RB×C are the gained detail and base embeddings by using a projection matrix P ∈ RC×C to decompose Z into mutually orthogonal subspaces. By the properties of orthogonal projection matrix, P should be a conjugate symmetric idempotent matrix and must satisfy the following constraints in the real number case:

The process of approaching can be divided into three components: approximating in the feature space, semantic representation, and the correlation between Z D and ¯ Z D . In the case of approximating on feature space, we first calculate the distances between all embeddings in a mini-batch for ¯ Z D and Z D respectively, and obtain the difference map M by:

Then we enforce the distance distribution of ¯ Z D to converge to that of Z D by optimizing the following loss:

Furthermore, we aligned the semantic representation between Z D and ¯ Z D by adjusting the predicted probability distribution of ¯ Z D closer to that of Z D . By drawing an analogy with formula (7), we have:

Considering that the detail feature ¯ Z D generated by the BEG block should exhibit the same correlation properties as Z D . As illustrated on the right side of the dashed line in Figure 3, we achieved consistency in correlation between ¯ Z D and Z D by pulling close their cross-modalities correlations denoted as Corr(Z p V ,Z p I ), p ∈ {D, / ¯D} and by reducing the discrepancy in correlation within the same modality, represented as Corr(Z D m , Z ¯D m ),m ∈ {V, I}. This is accomplished by optimizing the follows loss:

Thereby, the total approaching function for ¯ Z is:

After the description provided above, we generated the base feature Z B by eliminating the detail feature Z D from Z. Given that the base information across modalities, such as contours and movements, should exhibit significant similarities, we constructed a cross-modality feature fusion method that integrates the base feature Z B V and Z B I to generate an auxiliary feature Z B F . Inspired by [Li et al., 2022] [Wang et al., 2018], the fusion method can be formulated as follows:

Here, Z B V/I ∈ R B2 ×C represents the cross-modality base embedding, P ,Q are the learnable parameters. The fused Z B F aggregates the base knowledge from VIS and IR modalities, employing attention mechanisms across both channel and batch dimensions. We then enhance the similarity between Z B V and Z B I by aligning themwith Z B F :

This approach ensures that Z B contains only the knowledge shared between the modalities. In addition, we also utilize cross-modality semantic alignment for Z B V/I to strengthen the collection of modality-shared knowledge:

The id loss for both was also employed to enhance the distinguishable information of Z B and Z B F , and the loss for crossmodality feature fusion method is:

Consequently, the total loss for the BEG block can be summarized as follows:

where lorth represents the constraint in formula (10) for parameter P to achieve the decomposition of orthogonal subspaces.

3.5 Optimization

In the preceding section, the proposed DFE module extracted detailed knowledge from the intermediate feature ZM and subsequently produced the detail feature Z D , the proposed BEG block produced the base feature by eliminating the detailed knowledge from the comprehensive feature Z, the proposed SKD loss ensures that both the detail and base features effectively capture modality-specific and shared information. We also incorporated the commonly used id and triplet loss [Hermans et al., 2017] ltri for Z into our method. Similar to (18), we enforce cross-modality consistency for Z by:

Eventually, the total loss of BDLF is defined as:

4 Experiments

In this section, we validate the effectiveness of our BDLF by conducting experiments on the widely recognized SYSUMM01, RegDB and LLCM benchmarks.

4.1 Datasets and Evaluation Protocol

SYSU-MM01 dataset [Wu et al., 2017] comprises 287,628 VIS and 15,792 IR images from 491 identities captured by 4 RGB and 2 IR cameras. It features both All-Search and Indoor-Search modes for evaluation. RegDB [Nguyen et al., 2017] contains 412 identities, each represented by 10 VIS and

10 IR images captured from a pair of cameras. We adhere to the evaluation protocol outlined in [Ye et al., 2022b] to randomly split the identities into training and testing sets of equal size. LLCM [Zhang and Wang, 2023]is a challenging largescale low-light dataset for VI-ReID task, which contains 713 identities with 25,626 VIS and 21,141 IR images, all captured by 9 cameras in both RGB and IR modalities

The Cumulative Matching Characteristic curve (CMC) and mean Average Precision(mAP) are adopted as standard evaluation metrics in our experiments to comprehensively assess the performance of our framework.

4.2 Implementation Details

The entire framework is implemented using PyTorch and runs on a single NVIDIA RTX3090 GPU with 24GB VRAM. We employed a pre-trained ResNet-50[He et al., 2016b] as the backbone network and incorporated INN blocks with affine coupling layers[Dinh et al., 2017][Zhou et al., 2022] to construct the DFE module, setting the number of INN blocks to 6. All images are resized to 3×384×144, and we adopted the Random Channel Exchangeable Augmentation and ChannelLevel Random Erasing techniques proposed in [Ye et al., 2021b] during the training phase. The SGD optimizer was used, with the initial learning rate set to 1×10-2 , which was warmed up to 1 × 10-1 during the first 10 epochs, then we decayed the learning rate to 1×10-2 and 1×10-3 at epochs 20 and 95 for SYSU-MM01, and at epochs 70 and 130 for RegDB and LLCM, respectively. The learning rate was further decayed to 1 × 10-4 at 180 epoch, with a total of 220 epochs. For each mini-batch, we randomly sampled 8 identities, each consisting of 4 VIS and 4 IR images for training. Additionally, the exponential moving average (EMA) model [Ge et al., 2020] also employed in our method.

4.3 Comparison with State-of-the-art Methods

We demonstrate the superiority of our BDLF by comparing performance with several existing state-of-the-art methods on the SYSU-MM01, RegDB, and LLCM datasets. The performance of these methods is presented in Table 1, with optimal performances annotated by underlining.

Comparison on SYSU-MM01 and RegDB. Table 1 presents the results of our BDLF alongside selected outstanding methods, confirming the superiority of our BDLF, which almost outperforms all other state-of-the-art methods. In the All-Search mode of SYSU-MM01, our method achieved a rank-1 accuracy of 76.8% and a mAP of 74.6%, in the IndoorSearch mode, BDLF achieved a rank-1 accuracy of 84.2% and a mAP of 85.8%. On the RegDB dataset, our method achieved a rank-1 accuracy of 94.4% and a mAP of 90.1% for the VIS to IR search, and attained a rank-1 accuracy of 94.5% and a mAP of 89.6% for the IR to VIS search. These results validate the effectiveness of BDLF that independently learns the detail and base information and sufficiently utilizes cross-modalities knowledge.

Comparison on LLCM. According to Table 1, our method outperformed other approaches. Specifically, BDLF achieved a rank-1 accuracy of 67.0% and a mAP of 68.9% in VIS to IR search, as well as a rank-1 accuracy of 58.1% and a mAP of 64.5% in IR to VIS search. It is evident that our BDLF is well-equipped to handle challenging scenarios.

4.4 Ablation Studies

Effectiveness of each component. In this section, we designed an ablation experiment to validate the effectiveness of certain components of BDLF. Specifically, we removed the DFE, lapp, lorth and lskd modules from BDLF, while retaining the backbone with the BEG block as the baseline. All experiments adopted the same training settings, and we evaluated their performance in the All-Search mode of SYSU-MM01. The results are presented in Table 2, Notably, the removal of the DFE module resulted in poor precision, demonstrating the effective detail extraction capability of DFE. The experiments also indicated that the lapp loss enhances the model's distinguishing performance by effectively aiding in the generation of base embeddings, eliminating detailed knowledge from the comprehensive feature. Although the DFE module significantly promotes the mining of detail information, its performance remains suboptimal, as the model cannot extract all modality-specific and shared information without interference each other due to the absence of correlation constraint lcorr and independent constraint lorth .

Effectiveness of how many INN blocks are more favorable for DFE. The proposed DFE module consists of a series of INN blocks with an LN layer to extract detail information non-destructively. We conducted experiments to determine the optimal number of blocks for our framework. As shown in Figure 4, we modified the number of INN blocks and evaluated performance in the All-Search mode of SYSUMM01. The results indicate accuracy gradually improves as the number of INN blocks increases, reaching a plateau when the count is 6. This confirm that a balance exists between accuracy and computational complexity when the number of INN blocks is set to 6.

Effectiveness of which stage of ResNet-50 to combine DFE module. In this section, we implement experiments to assess which stage of ResNet-50 is most suitable for serving as the input to the proposed DFE module. All experiments maintain consistent settings, except for the locations of the DFE module within ResNet-50. The results are presented in Table 3, we observed that connecting the DFE module to stage-3 of ResNet-50 yielded the best accuracy in the AllSearch mode of SYSU-MM01. This can be attributed to the fact that modality-shared information is more prominent in the high-level features produced by stages-4, which impedes the extraction of modality-specific detail information. Furthermore, the low-level features generated by stages 1 and 2 are inadequate for effectively expressing the semantics necessary to distinguish between different identities. These findings elucidate why the best accuracy is achieved when the DFE module is connected to stage-3 of ResNet-50.

4.5 Visualization

To investigate the detail information extraction capabilities of the proposed DFE, we visualize the comprehensive and detailed features of several identities produced by BDLF. As illustrated in Figure 5, a comparison of the images of comprehensive and detailed features reveals that the attention regions of the comprehensive features is broader and more dispersed than that of the detailed features. This observation indicates that the DFE module has the capacity to focus on subtly distinguishable characteristics.

5 Conclusion

In this paper, we propose a novel base-detail feature learning framework(BDLF) that learns detail and base features from a correlation and mutual information maximization for the VIReID task. The proposed BDLF consists of a DFE module and a BEG block. The DFE module non-destructively extracts detail information, while the BEG block generates base features by eliminating detail information from the output of the backbone network, constrained by independence and correlation requirements on the detail and base embeddings. Extensive experiments on the SYSU-MM01, RegDB, and LLCM datasets have demonstrated the superiority of BDLF.

Acknowledgments

This work was partially supported by the Shenzhen Science and Technology Program (KJZD20230923114600002), and the Guangdong Major Project of Basic and Applied Basic Research (2023B0303000010).