Richer Semantics, Better Alignment: Aligning Visual Features with Explicit and Enriched Semantics for Visible-Infrared Person Re-Identification

1 Introduction

Person Re-Identification (ReID) [Ye et al., 2021b; Yan et al., 2023a; Dong et al., 2024b] aims to match images of the same individual across cameras, a critical component of intelligent security with profound research implications. Despite significant advancements [Li et al., 2021; Yan et al., 2022; Gong et al., 2022; Dong et al., 2024a], most existing algorithms focus solely on visible image retrieval, failing to meet the demands of 24-hour surveillance systems, which must also retrieve infrared images captured at night. To overcome this limitation, visible-infrared person ReID (VIReID) [Wu et al., 2017] has been proposed to retrieve visible images that match the identity of a given infrared query, and vice versa.

∗Corresponding author

The primary challenge in VIReID lies in aligning the feature distribution of cross-modality images, for which two main approaches have been developed. The first is generative-based methods [Dai et al., 2018; Choi et al., 2020; Miao et al., 2021], which transfer the style of images to another modality. However, these algorithms often introduce noise during the generation process, compromising feature discriminability. The second approach, generative-free methods [Ling et al., 2021; Ye et al., 2021a; Li et al., 2022], fo-

Comparatively, the latter has demonstrated greater effectiveness and currently stands as the predominant solution. However, the large modality discrepancy makes it challenging to align heterogeneous features into a suitable common space.

To address this limitation, a recent study [Yu et al., 2025] incorporates Contrastive Language-Image Pre-training (CLIP) [Radford et al., 2021] into VIReID, demonstrating that semantics represented by language descriptions of heterogeneous images exhibit no modality gap, thus aligning visual features into the semantic space is beneficial for alleviating their distribution discrepancy. However, pedestrian images typically lack accompanying language descriptions. Learning textual prompts [Zhou et al., 2022] effectively addresses this issue, as illustrated in Figure 1(a), but it still presents several drawbacks: 1) Uncertainty. The set trainable words are unknown, raising questions about what the semantic information they represent; 2) Coarseness. Pedestrian images with the same identity share a common prompt, and only four learnable tokens are allocated for identity depiction, which is insufficient for the cross-view and fine-grained nature of VIReID; 3) Cumbersomeness. Rather than end-toend, the paradigm of learnable prompts requires a meticulously designed two-stage training process.

Recently, LLaVA [Liu et al., 2023], a prominent large language-vision model, has demonstrated exceptional capability in image captioning. As shown in Figure 1(a), it can generate clear and detailed descriptions for pedestrian images, whose explicit semantics, such as age and gender, are able to facilitate the effective alignment of visual features. This inspires us to supplement specific texts with the assistance of LLaVA and align image-text pairs within each modality. Furthermore, the alignment of images to intermodality texts is also necessary as it can further alleviate the distribution discrepancy between visual features. However, descriptions of visible and infrared images may include conflicting semantics, such as color attributes, which makes the direct alignment inappropriate. This motivates us to explore an indirect alignment of images to inter-modality texts. In addition, as shown in Figure 1(b), within each modality, the descriptions corresponding to different images of the same pedestrian contain complementary content. Integrating them to acquire comprehensive semantics and accordingly guide the alignment of visual features is beneficial for further enhancing their modality invariance. This prompts us to enrich pedestrian semantics with multi-view texts.

In this paper, we propose a novel framework termed Richer Semantics, Better Alignment (RSBA), which aligns visual features with explicit and enriched semantics for effective VIReID. As shown in Figure 2, it consists of Explicit Semantics-Guided Feature Alignment (ESFA), Consistent Similarity-Guided Indirect Alignment (CSIA), and Cross-View Semantics Compensation (CVSC). ESFA leverages LLaVA to generate textual descriptions for visible and infrared images, respectively, and maximizes the similarity between visible (infrared) image-text pairs to align crossmodality visual features into the semantic space. CSIA constrains the similarity between intra-modality image-text pairs to be consistent with that between inter-modality text-text pairs, achieving the indirect alignment of visible visual features with infrared semantics as well as infrared visual features with visible semantics. CVSC integrates text features from another view into the current view and accordingly improves the image-text matching in ESFA and CSIA from oneto-one to one-to-many, thereby further advancing their alignment. Our RSBA is trained end-to-end, with only the visual side used to extract cross-modality image features for testing.

Our main contributions are summarized as follows:

• We explore the advantages of explicit semantics in alleviating the modality gap between visible and infrared images, and accordingly propose ESFA to align visual features into the semantic space for effective VIReID.
• We realize the alignment of visual features with intermodality semantics, and accordingly present CSIA to address the challenge of the direct alignment resulting from conflicting semantics.
• We consider the comprehensiveness of multi-view semantics, and develop CVSC to achieve the one-to-many alignment between images and texts, further strengthening the modality invariance of visual features.
• Extensive experiments across three datasets demonstrate that RSBA achieves new state-of-the-art performance, with each component contributing effectively.

2 RelatedWork

2.1 Visible-Infrared Person Re-Identification

VIReID is a challenging task due to the significant modality gap between visible and infrared images. An intuitive approach is to transfer images from one modality to the style of another. For instance, JSIA [Wang et al., 2020] employed feature decoupling and cycle generation to augment crossmodality image pairs. Given the substantial gap between heterogeneous data, MSA [Miao et al., 2021] designed a style similarity constraint to ensure the quality of generated images. To prevent identity information loss during transfer, ACD [Pan et al., 2024] introduced conditional probability density to optimize the generation network. Although generative-based methods are intuitive and effective, they are prone to model collapse and susceptible to introducing noise.

Generative-free methods have recently attracted considerable attention due to they circumvent the limitations of generative-based approaches. These methods primarily focus on aligning cross-modality features by constructing appropriate networks or metric functions. For instance, ZeroPadding [Wu et al., 2017] evaluated the suitability of four networks for VIReID and proposed a one-stream structure with a zero-padding strategy. AGW [Ye et al., 2021b] devised a weighted regularization triplet loss to optimize the relative distance between positive and negative pairs in both intra-modality and inter-modality. To learn informative representations, DEEN [Zhang and Wang, 2023] designed an embedding expansion network to extract diverse features. However, the large modality discrepancy still makes the feature alignment challenging. In this paper, we explore a semanticguided approach to effectively align visual features.

2.2 Large Language-VisionModels

Large language-vision models have emerged as a significant research topic, bridging computer vision and natural language processing. CLIP, a representative model, excels in learning visual content with high-level semantic information, showcasing exceptional potential across various downstream vision tasks [Wang et al., 2022; Zhao et al., 2022; Yan et al., 2023b; Tang et al., 2024; Yan et al., 2024;

Shen et al., 2025]. In the field of ReID, CLIP-ReID [Li et al., 2023] first introduced CLIP to advance this community. To tackle the occlusion problem, RGANet [He et al., 2024] employed CLIP to generate local textual prototypes for mining discriminative part features. In VIReID, CSDN [Yu et al., 2025] incorporated trainable textual prompts to acquire implicit pedestrian descriptions, aligning visual features of visible and infrared images into the semantic space. However, the semantics learned by CSDN are unknown and coarse, limiting its alignment ability. In this paper, we propose ESFA to address this limitation, and further develop CSIA and CVSC to improve our RSBA framework for more efficient VIReID.

3 Methodology

3.1 Preliminaries

Formally, we define the visible and infrared image sets as {xvis i }Nv i=1 and {xir i }Nr i=1 , whereNv andNr represent the sizes of these two heterogeneous data, respectively. The label set is denoted as {yi}Np i=1 , with Np indicates the number of identities. In each mini-batch, N paired cross-modality images {xvis i , xir i }Ni=1 are randomly sampled and their visual features {f vis i , f ir i }Ni=1 ∈ RN×d are extracted, where d is the dimension of features. We employ identity loss and modality-shared enhancement loss [Lu et al., 2023] to optimize the network:

where qi is the one-hot vector of identity label yi. pvis i and pir i represent classification results of f vis i and f ir i , respectively.

The modality-shared enhancement loss constrains the average distance between positive pairs across modalities to be equal to that between positive pairs under the intra-modality:

where P and K denote P identities and K visible and K infrared images of each identity randomly sampled in each mini-batch. ∥·∥2 represents the Euclidean distance.

3.2 Explicit Semantics-Guided Feature Alignment

Most existing frameworks treat VIReID as a pure vision task, lacking the ability to capture pedestrian semantics that is beneficial for modality alignment. Although CSDN introduces CLIP and CoOP to address this limitation, the uncertainty and coarseness of implicit semantics hinder the alignment of visual features into the semantic space. To this end, we propose ESFA, which leverages LLaVA to generate explicit textual descriptions and aligns cross-modality images with them.

As illustrated in Figure 2, given a pedestrian image, we send the request command 'Please describe the characteristics of the pedestrian in the image' to LLaVa. It responds with a natural language description 'The pedestrian in the image is a young woman wearing a blue skit, carrying a shoulder bag and holding a parasol'. This description provides clearer and more detailed explicit semantics, such as age, gender, and clothing, compared to the learnable textual prompt 'A photo of a [X1][X2][X3][X4] person' in CSDN. Notably, LLaVA operates without requiring training and delivers responses quickly, taking approximately 1.2 seconds per image.

Suppose the generated language bases for visible and infrared images are {lvis i }Nv i=1 and {lir i }Nr i=1 . In eachmini-batch, we sample {lvis i , lir i }Ni=1 corresponding to {xvis i , xir i }Ni=1 and input them into the textual encoder to extract features {tvis i , tir i }Ni=1 ∈ RN×d . To align {f vis i , f ir i }Ni=1 with {tvis i , tir i }Ni=1 , we maximize the similarity between them:

where

where s(·) represents the cosine similarity.

3.3 Consistent Similarity-Guided Indirect Alignment

ESFA achieves the alignment of images and texts in each modality; however, it ignores the alignment of images and inter-modality texts. A straightforward approach is to maximize the similarity between them similar to the above process. However, cross-modality texts describe the same object with conflicting attributes due to visual ambiguity. For example, the clothing exhibits 'blue' in the visible text while being seen as 'gray' in the infrared one. Forcing the maximization of similarity between images and inter-modality texts may disrupt the expressiveness of semantics. To this end, we develop CSIA to explore the indirect alignment between them.

As illustrated in Figure 2, for the visible visual feature f vis i , CSIA constrains its similarity with the visible text feature tvis i to be equal to the similarity between the infrared text feature tir i and visible text feature tvis i , thereby indirectly establishing the alignment relationship between f vis i and tir i . Similarly, infrared visual features f ir i and visible text features tvis i are indirectly aligned by constraining the similarity between f ir i and tir i to be consistent with that between tvis i and tir i :

This similarity consistency loss not only achieves the alignment of images with inter-modality texts but also indirectly maximizes the similarity between infrared and visible texts, which helps alleviate cross-modality semantic discrepancy, thus facilitating more effective alignment of visual features.

3.4 Cross-View Semantics Compensation

The above two alignments are based on the one-to-one matching between image and text. However, within each modality, variations in camera views result in descriptions for different images of the same pedestrian emphasizing distinct objects. For example, the description for a front-facing image may highlight age and gender, while that for a rear-facing image is more likely to focus on hairstyle and backpack. As a result, semantics derived solely from single-view text are one-sided and contribute limited to the robustness of visual features. To address this limitation, we design CVSC to explore the oneto-many correspondence between images and texts.

As illustrated in Figure 2, we introduce an attention fusion module to integrate information in the textual feature from another view into the textual feature of the current view. Specifically, for the visible textual feature tvis i , we randomly select a textual feature tvis a that shares the same identity with tvis i while from different views. We compute the similarity between tvis i and tvis a to derive the attention weight avis i :

where Wq and Wk are two linear projection layers. We multiply avis i and tvis a to determine the contribution of tvis a , and add the the resulting weighted feature to tvis i :

where Wa and Wv are also linear projection layers. tvis m,i represents the multi-view textual feature corresponding to lvis i , which contains richer pedestrian semantics as it compensates for the missing cross-view information in tvis i . Similarly, we can acquire the multi-view infrared textual feature tirm,i .

We reformulate Equations (5), (6), and (7) as the following Equations (11), (12), and (13), which maximize the similarities between f vis i and tvis m,i , as well as between f ir i and tirm,i :

This process achieves the one-to-many alignment between images and texts within each modality. In addition, we also redefine the Equation (8) to the following Equation (14), indirectly aligning images with multi-view inter-modality texts:

3.5 Training and Inference

The proposed RSBA is trained in an end-to-end manner, with the total loss expressed as:

where λ1 and λ2 are two hyper-parameters used to balance the relative importance of Lo2m con and Lo2m sc , respectively.

Notably, the generation of language descriptions is only performed in the training phase, ensuring the practicality of our framework. During inference, the textual encoder and attention fusion module are not required, reducing the model complexity and inference time of our framework.

4 Experiments

4.1 Datasets and EvaluationMetrics

Datasets. SYSU-MM01 [Wu et al., 2017] contains 30,071 visible images captured by 4 RGB cameras and 15,792 infrared images captured by 2 IR cameras. The training set includes 22,258 visible images and 11,909 infrared images corresponding to 395 identities. The testing set comprises 3,803 infrared images of 96 identities and either 301 or 3,010 randomly sampled visible images for single-shot or multishot settings, respectively. RegDB [Nguyen et al., 2017] is a small-scale VIReID dataset with 4,120 visible images and 4,120 infrared images from 412 pedestrians. Following the standard protocol, 2,060 visible and 2,060 infrared images of 206 identities are allocated for training, while the remaining images are used for testing. LLCM [Zhang and Wang, 2023] is a recently released challenging VIReID dataset collected under low-light conditions. Its training set includes 16,946 visible images and 13,975 infrared images of 713 identities, and its testing set consists of 8,680 visible images and 7,166 infrared images corresponding to 351 identities.

Evaluation Metrics. We assess the retrieval performance using the general indicators named mean Average Precision (mAP) and Cumulative Matching Characteristics (CMC).

4.2 Implementation Details

We conduct experiments using the PyTorch library on a single RTX 4090 GPU. The proposed RSBA framework incorporates a training-free LLaVA, a CLIP model comprising a visual encoder and a textual encoder, with ResNet50 [He et al., 2016] serving as the backbone for the visual encoder, and an attention fusion module consisting of four linear projection layers. Following AGW [Ye et al., 2021b], we train two parallel first convolutional layers of ResNet50 for each modality while sharing the parameters of the subsequent four blocks. During training, we randomly sample 8 identities, each with 4 visible and 4 infrared images. All input images are resized to 288×144 and subjected to data augmentation techniques, including random padding, cropping, and flipping. The training process spans 120 epochs, with the initial learning rate set to 3e-4 for the visual encoder and 1e-6 for the textual encoder and attention fusion module, decaying by a factor of 0.1 at the 40th and 70th epochs, respectively. The hyper-parameters are configured as λ1 = 0.25 and λ2 = 0.2.

4.3 Comparison with State-of-the-Art Methods

SYSU-MM01. Table 1 presents the comparison results with the state-of-the-art methods on the SYSU-MM01 dataset, showing that RSBA consistently outperforms them across all settings. Specifically, in the all-search testing mode, our Rank-1 accuracy and mAP surpass those of the best generative-based method, ACD, by 4.0% (4.2%) and 3.7% (3.1%), respectively, while in the indoor-search mode, the improvements are 8.8% (6.3%) and 5.1% (5.2%). These gains are attributed to our approach aligning modalities at the feature level, which circumvents performance limitations imposed by the generated low-quality images. Compared to generative-free methods, under the single-shot mode, our Rank-1 accuracy exceeds that of CycleTrans by 1.9% (0.5%), and our mAP surpasses HOS-Net by 0.6% (2.1%). This advantage arises from the proposed RSBA aligns visual features with the semantic space, which are beneficial for alleviating the modality gap. Furthermore, our RSBA also outperforms CSDN across all settings, benefiting from its ability to capture clear, detailed, and rich semantics, as opposed to the coarse and ambiguous semantics learned by CSDN.

RegDB. We further evaluate the performance of RSBA on the RegDB dataset, with the quantitative results summarized in Table 2. Our method achieves superior recognition rates compared to existing generative-based approaches. For instance, in the visible-to-infrared testing mode, RSBA outperforms TSME in Rank-1 accuracy by 7.9% and surpasses ACD in mAP by 7.6%. Similarly, our method exhibits significant performance advantages over state-of-the-art generative-free methods, such as MBCE and HOS-Net. In comparison with CSDN, RSBA improves the Rank-1 and mAP by 2.1% and 4.0% in the visible-to-infrared testing mode.

LLCM. We also evaluate the proposed RSBA on the challenging LLCM dataset to provide a comprehensive assessment. As detailed in Table 3, in the visible-to-infrared testing mode, RSBA achieves a Rank-1 accuracy and mAP that are 1.1% and 0.9% higher, respectively, than those of the stateof-the-art HOS-Net. Similarly, in the infrared-to-visible testing mode, RSBA outperforms HOS-Net with improvements of 0.9% in Rank-1 accuracy and 1.0% in mAP. These results further highlight the superiority of our approach.

4.4 Ablation Studies

We evaluate the effectiveness of each component in our proposed RSBA, with the results presented in Table 4. The Rank1 and mAP of Baseline ('0') are 71.9% and 67.6% under the single-shot and 80.0% and 61.9% under the multi-shot.

Effectiveness of ESFA. ESFA aims to introduce explicit semantics to guide the alignment of cross-modality visual features. As shown in Table 4, it improves the Rank-1 and mAP by 4.4% and 5.0% under the single-shot mode, which validates that aligning visual features into the semantic space is reasonable and effective for mitigating the modality gap.

Effectiveness of CSIA. CSIA constrains the consistent similarity between intra-modality image-text pairs and intermodality text-text pairs to establish the correspondence between images and cross-modality texts. As detailed in Table 4, under the single-shot test mode, it improves the Rank-1 accuracy from 76.3% to 77.4%, which indicates that the alignment of inter-modality image-text pairs plays a positive role in the further effective alignment of visual features.

Effectiveness of CVSC. CVSC integrates multi-view texts to capture comprehensive semantics that are beneficial for improving the alignment in ESFA and CSIA. As illustrated in Table 4, when it is equipped with ESFA, the Rank-1 accuracy is improved by 1.3% and 1.0% under the two test modes, respectively. In addition, when incorporating it with both ESFA

and CSIA, the recognition performance reaches a peak. These results fully demonstrate the reasonableness of motivation behind CVSC and the effectiveness of its technology.

4.5 Parameters Analysis

We introduce the hyper-parameters λ1 and λ2 to regulate the relative importance of the loss terms Lo2m con and Lo2m sc . The former optimizes the model to align image-text pairs within each modality, while the latter drives the model to mine the correspondence between image-text pairs across modalities.

As shown in Figure 3, the optimal values for λ1 and λ2 are 0.25 and 0.2. In addition, excessively large values diminish the contributions of the identity loss and modality-shared enhancement loss, while overly small values result in insufficient alignment, both leading to performance degradation.

4.6 Further Discussions

In this section, we further discuss each proposed module, with the experimental results presented in Table 5.

The superiority of explicit semantics

Different from the implicit semantics in CSDN [Yu et al., 2025], our ESFA acquires explicit pedestrian semantics to align cross-modality visual features. In contrast, the latter is more representative and thus guides the more efficient alignment. As shown in Table 5, the recognition performance achieved by ESFA (2) is higher than that achieved based on implicit semantic alignment (1), with improvements of 2.1% in Rank-1 accuracy and 2.7% in mAP under the single-shot test mode. Notably, we observe that the descriptions generated by LLaVA follow the fixed sentence structure of 'The pedestrian in the image is a [age group] [gender] wearing [clothing], carrying [accessory]', which may cause the model to overfit to the non-differentiated semantic pattern, limiting the effect of alignment. This motivates us to explore acquiring diverse pedestrian semantics in the future.

The advantage of indirect alignment

(1) Why align visual features with inter-modality semantics? The proposed ESFA achieves alignment of visible visual features and visible semantics, as well as infrared visual features and infrared semantics. If we further align visible visual features and infrared semantics, as well as infrared visual features and visible semantics, the distribution discrepancy between visual features of visible and infrared can be further reduced. (2) Why indirectly align them? Different from image-text pairs within each modality, which naturally correspond to each other, images and inter-modality texts are not completely matched. Therefore, aligning them directly by maximizing the similarity between them may destroy the expressiveness of semantics, thereby weakening the alignment between intra-modality image-text pairs. As shown in Table 5, the direct alignment (1) reduces the Rank-1 accuracy and mAP of ESFA from 76.3% to 75.6% and from 72.6% to 72.1%. In contrast, our designed indirect alignment strategy (2) improves the Rank-1 and mAP by 1.1% and 0.6%. This proves the rationality and effectiveness of the approach.

The number of cross-view texts

The proposed CVSC aims to enrich pedestrian semantics with multi-view texts, and we achieve this by integrating text with that from an additional view. It is also feasible to integrate it with texts from multiple additional views. However, we observe that the recognition performance degrades as the number of views increases (2 and 3). This is because the generated descriptions may contain some inaccurate content, amplifying the noisy semantics during the information integration. In addition, CVSC is achieved through the attention fusion network, which requires more parameters as the number of views increases, making model optimization challenging.

4.7 Limitations

This paper acquires explicit and enriched semantics to effectively alleviate the modality gap between visible and infrared pedestrian images. However, as we discussed above, on the one hand, the rigid semantic pattern weakens the effect of alignment. On the other hand, this paper initially explores the enrichment of pedestrian semantics with multi-view texts, while we ignore the quality of texts, the number of cross-view texts, and the strategy of text fusion, which all affect the richness of the semantics. These limitations motivate us to explore the semantics of diversity and richness more deeply.

5 Conclusion

In this paper, we propose a novel Richer Semantics, Better Alignment (RSBA) framework for effective VIReID. It focuses on aligning visible and infrared visual features with explicit and enriched semantics and achieves this through Explicit Semantics-Guided Feature Alignment (ESFA), Consistent Similarity-Guided Indirect Alignment (CSIA), and Cross-View Semantics Compensation (CVSC). ESFA supplements language descriptions for pedestrian images and builds the correspondence of image-text pairs, aligning visual features into the semantic space. CSIA introduces the similarity consistency constraint to indirectly align visual features with inter-modality semantics, further alleviating the distribution discrepancy of visual features. CVSC mines comprehensiveness semantics to further facilitate ESFA and CSIA. Experimental results highlight the advancements RSBA achieves over state-of-the-art methods. In the future, we will further explore the assistance of semantic information for VIReID.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 62172212 and Grant 62427819, the Natural Science Foundation of Jiangsu Province under Grant BK20230031, the Jiangsu Provincial Science and Technology Major Project under Grant BG2024042.