Enhancing Cross-modal Completion and Alignment for Unsupervised Incomplete Text-to-Image Person Retrieval

1 Introduction

The goal of the person re-identification (ReID) task is to match images of individuals who share the same person identity across various camera viewpoints. Categorized based on the query object's data type, ReID can be segmented into three primary categories: image-based ReID [Sun et al., 2018; Xuan and Zhang, 2021; Yu et al., 2019; Gong et al., 2023a], text-based person search [Liu et al., 2019; Chen et al., 2018; Jing et al., 2020b; Gong et al., 2023b] and video-based ReID [Hou et al., 2021; Bai et al., 2022;

∗Corresponding Author

Hou et al., 2019]. Text-based person ReID is a cross-modal fine-grained retrieval task, and its objective is to explore the fine-grained information shared between visual and linguistic domains while simultaneously establishing their tighter finegrained alignment. In recent years, numerous effective textimage representation learning methods [Ding et al., 2021; Shao et al., 2022] have made remarkable advancements. These studies adhere to a similar scheme: 1) They employ the cross-modal alignment loss to align visual and textual representations into a shared embedding space. 2) Text-based person ReID models are trained on fully matched and labeled image-text pairs. These approaches heavily depend on completely matched and labeled image-text pairs, as depicted in Figure 1 (a). Indeed, this assumption is idealistic and constrained by an array of inevitable practical factors, e.g., privacy protection [Zhang et al., 2022; Dou et al., 2022;

Tang et al., 2016; Li et al., 2018], data missing [Xiang et al., 2023], and data corruption [Xian et al., 2023]. Therefore, previous approaches that relied on complete and labeled modality data to construct ranking loss for exploring text and image alignment do not perform effectively in these scenarios. As depicted in Figures 1 (b) and (c), real-world applications frequently confront challenges involving unlabeled and incomplete multimodal data. In this work, we first propose a more robust and practical setting referred to as unsupervised incomplete text-based person ReID, where person images and text descriptions are not fully matched and lack the supervision of identity labels during the training phase.

Certainly, unsupervised incomplete text-image ReID must address two key challenges: (1) How to effectively handle incomplete multimodal training data? (2) How to establish alignment between distinct fine-grained features across images and texts in the absence of true label supervision, and devise cross-modal alignment loss functions? To tackle the aforementioned concerns, we propose a novel Enhancing Cross-modal Completion and Alignment (ECCA) method, as shown in Figure 2, which comprises four key modules: cross-modal nearest neighbors construction with high semantic similarity, cross-modal feature-level completion for missing data, weight inter-instance granularity alignment, and enhanced prototype-wise granularity alignment. Specifically, we propose the high semantic similarity neighbor generation method, in which a new Jaccard distance metric is proposed to calculate the distance between two nearest neighbor samples and select the most reliably k-reciprocal nearest neighbors from cross-modality and self-modality. Relational graphs for missing features are then built using the nearest neighbors with high semantic similarity to the incomplete modality data that is reconstructed by weighting the neighbors. In addition, to address the challenge of cross-modal matching ambiguity as shown in Figure 1 (d), we propose weighted inter-instance granularity alignment as well as enhanced prototype-wise granularity alignment modules that can encourage the model to map semantically similar imagetext pairs more compact in the common embedding space.

Our key contributions can be summarized three-fold: (1) We pioneer a new unsupervised incomplete text-image ReID task, aiming to improve the robustness and generalization of text-based ReID. (2) We put forward cross-modal neighbor construction with high semantic similarity and featurelevel missing modality completion modeling to achieve reliable missing modal feature completion. (3) We propose the weighted inter-instance granularity alignment and enhanced prototype-wise granularity alignment modules, which can reduce the effect of cross-modal matching ambiguities.

2 RelatedWork

2.1 Text-Based Person Re-Identification

The existing text-based person ReID methods can essentially be classified into two categories: cross-modal interactionbased and cross-modal interaction-free methods. The former [Niu et al., 2020; Gao et al., 2021; Ding et al., 2021; Wang et al., 2020] mainly utilizes various attention schemes to establish word-patch [Ding et al., 2021; Chen et al., 2018;

Li et al., 2017a; Li et al., 2017b] or phrase-region [Jing et al., 2020b; Niu et al., 2020] multi-granularity alignment relations and predict the matching score for image-text pairs. The latter [Gao et al., 2021; Niu et al., 2020] primarily focuses on learning global features without interactive attention mechanisms for global alignment. Such methods often employ different model structures and optimizing functions [Zhang and Lu, 2018] to align the image and text embeddings in a shared latent feature space. Recently, some works applied image and text modal pre-training of CLIP [Li et al., 2022; Shao et al., 2022] and achieved significant improvement.

2.2 Unsupervised Text-Image Retrieval

Research on unsupervised text-based person ReID tasks is scarce. There are only a few studies on unsupervised imagetext cross-modal retrieval. Patel et al. [Patel et al., 2019] propose an unsupervised cross-modal retrieval framework that leverages a latent Dirichlet allocation topic modeling framework to supervise the training of deep CNN. Liu et al. [Liu et al., 2022] propose an unsupervised deep cross-modal method that exploits unsupervised contrastive learning to model the relationship among intra- and inter-modality instances. Different from general unsupervised text-image retrieval, the text-based ReID task explores more fine-grained cross-modal semantic alignment. Therefore, we utilize text-IoU guided weights to facilitate cross-modal instance discriminate learning, and leverage unified prototypes to predict soft prototype assignments to minimize intra-class variations and maximize inter-class variations between different modalities.

2.3 Incomplete Cross-modal Retrieval

There is currently no work on the unsupervised incomplete text-based ReID research. Most related to our unsupervised incomplete text-based person ReID task is the traditional incomplete image-text retrieval task. Guo et al. [Guo and Zhu, 2019] propose a collective affinity learning method (CLAM) to recover the missing adjacency information. Jiang et al. [Jing et al., 2020a] exploit the dual-aligned variational autoencoders (DAVAE) to generate completion features. Zeng et al. [Zeng et al., 2021] investigate a prototype-based adaptive network (PAN) to reconstruct the completion samples by prototype propagation scheme. Our unsupervised incomplete text-based ReID method is fundamentally different from them in the following aspects: (1) CLAM is based on hashing to cope with partial cross-modal problems in hash space. Our method focuses more on improving accuracy in text-based ReID. (2) DAVAE and PAN are supervised incomplete or imbalanced image-text retrieval methods, which cannot effectively learn modality alignment representations without labels to generate complete representations.

3 Methodology

In the unsupervised incomplete text-based person ReID task, the fully matched training dataset is defined as X = {(Ii, Ti)}K1 i=1 , where Ii represents the i-th image instance, Ti is the i-th corresponding text description for that image Ii, and K1 denotes the total number of fully matched imagetext pairs. Incomplete multi-view data comprises missing vi-

𝒕

Figure 2: Illustration of the enhancing cross-modal completion and alignment (ECCA) framework for unsupervised incomplete text-image person retrieval. FMC, HSNG, WIGA and EPGA represent feature-level missing modality completion, high semantic similarity neighbor generation, weighted inter-instance granularity alignment and enhanced prototype-wise granularity alignment. The orange is shared prototypes across the image and text modalities for modal interaction and semantic alignment.

sual modality data Xt only with text modality data and missing text modality data Xv only with image modality data, where Xt = {˜ Im, Tm}K2 m=1 , Xv = {In, ˜ Tn}K3 n=1 , ˜ Im and ˜ Tn is missing (unavailable, inaccessible, incomplete) data, and Tm, In as well as (Ii, Ti) are available (accessible) data during training. Here, K2 and K3 represent the total number of missing visual data and missing text data, respectively. K1 +K2 +K3 = N denotes the total number of samples.

3.1 Feature-level Cross-modal Completion

Firstly, we introduce the feature extraction networks for both the visual and textual modalities. For each image instance Ii and text instance Ti, the initial visual embedding zv i and the initial textual embedding zt i can be generated using the visual encoder f v (Ii, θv ) with trainable parameters θv , as well as textual encoder f t (Ti, θt ) with trainable parameters θt . To project features from distinct modalities into a joint embedding space that ensures substantial modality interaction and semantic alignment at the feature level, we adopt the shared prototypes across images and texts for local fine-grained implicit alignment. In specific terms, we define the shared prototypes as D ∈ Rs×d across the image and text modalities. Here, s signifies the number of prototypes, while d denotes the dimension of features. The prototypes are randomly initialized, and the prototypes and common representations are jointly learned in the subsequent training process. For the fusion of image (text) representation, the shared prototype D serves as the query Q, while the original image (text) representation zv i (zt i ) is employed as the key K and value V in the transformer's cross-attention operation. Hence, the fused visual and textual feature representations by,

here, vi and ti represent the reconstructed visual and textual contextualized features. The operation MHA(.) refers to a transformer block, which comprises multi-head crossattention and a feed-forward network [Vaswani et al., 2017].

High Semantic Similarity Neighbor Generation. We propose the high semantic similarity neighbor generation (HSNG) method, in which a new Jaccard distance metric is proposed to calculate the distance between two nearest neighbor samples and selects the most reliably k-reciprocal nearest neighbors from cross-modality and self-modality. Specifically, for missing image feature ˜ vm shown in Figure 3, we can acquire the corresponding textual embedding feature tm. Additionally, we calculate cross-modal cosine similarity between tm and all existing image features {vi}K1+K3 i=1 . By utilizing the k-nearest neighbor algorithm, we rank and identify the k most similar image embeddings to the textual representation tm, denoted by,

Further, for vl ∈ Nk(tm), we calculate cross-modal cosine similarity between vl and all existing text representations, and can obtain the k-nearest neighbor set for vl as Nk(vl) = {t1, t2, ..., tk}. Accordingly, the cross-modality kreciprocal nearest neighbors Rk(tm) for tm are formulated,

Furthermore, for vi, vj ∈ Nk(tm), we can calculate intramodal cosine similarity between vi, vj and all existing image representations, and can obtain the k-nearest neighbor set for vi and vj defined as Nk(vi) and Nk(vj). Accordingly, we define the self-modality k-reciprocal nearest neighbor as,

Considering both cross-modality and self-modality kreciprocal nearest neighbors, a new Jaccard distance metric is given by,

Under such constraints, we can find reliable cross-modal kreciprocal nearest neighbors to improve the reliability of the nearest neighbor generation. Finally, we can obtain the high semantic similarity neighbor generation set formulated as,

The same applies to the missing text features as well.

Feature-level Missing Modal Completion. To efficiently complete missing modality embeddings, we introduce a feature-level missing modality completion (FMC) method. Specifically, for missing image feature ˜ vm and missing text feature ˜ tn, we can build the most relevant nearest neighbor sets of cross-modal features by the aforementioned neighbor generation method with high semantic similarity, defined as Nk′ (tm) = {v1, v2, ..., vk′ } and Nk′ (vn) = {t1, t2, ..., tk′ }. The reconstructed visual representation ˜ vm of ˜ Im and textual representation ˜ tn of ˜ Tn are formulated as,

where Av denotes the affinity matrix of [tm,Nk′ (tm)] = [tm, v1, v2, ..., vk′ ] = [g1, g2, ..., gk′ +1 ], and At denotes the affinity matrix of [vn,Nk′ (vn)] = [vn, t1, t2, ..., tk′ ]. Each value of the affinity matrixes Av and At represents the degree of semantic similarity between two instances, formulated as,

where Z -1 represents the normalized Laplacian matrix of S, and each element Sij ∈ S is calculated by,

here ⟨, ⟩ denotes the cosine similarity between two instances. Similarly, this calculation process also applies to At. This approach is essentially equivalent to constructing graph relationships, enabling information to be transmitted across different samples based on the graph, thus enhancing the features of completion. Here, the affinity matrix Av is the edges and the feature [tm,Nk′ (tm)] is the nodes. To mitigate the modal discrepancy between the generated representations and the original corresponding representations, the feature-level missing modality completion (FMC) is formulated as,

3.2 Weighted Instance Granularity Alignment

To address the challenge of cross-modal matching ambiguity caused by the absence of true label supervision as shown in Figure 1 (d), we propose a weighted inter-instance granularity alignment module (WIGA). which adaptively applies different weights according to the matching probability between different instances, and adaptively adjusts the alignment of texts and images in the shared space. Our study is based on an empirical observation that noun phrases within two textual descriptions originating from the same pedestrian identity consistently exhibit either the same or synonymous attributes. For the provided textual description Ti, we employ NLTK [Loper and Bird, 2002] to extract relevant noun phrases from the text Ti, which are represented as,

where P denotes the noun phrase extractor and l represents the count of noun phrases. Next, the Intersection over Union (IoU) based on the textual descriptions is defined as,

here, |Ri ∩ Rj| represents the count of synonymous noun phrases shared between Ri and Rj. |Ri ∪ Rj| indicates the number of noun phrases in the union between Ri and Rj . The matching probability weights between different instances can be obtained by,

The WIGA dynamically adjusts the alignment of different instances by adding different similarity weights as,

where ⟨, ⟩ represents the cosine similarity, and τ represents an temperature factor. N is the total number of image-text pairs, and α ∈ [0, 1] is the prior probability that image vi is matched with its paired text tj. When α = 1, we should use the one-hot labels Iij for contrastive learning. However, to better align unpaired text feature tj with image feature vi, αIij provides supervision for paired image-text samples, while (1-α)Wij supervises the unpaired samples. The overall objective of our WIGA loss is computed as,

3.3 Enhanced Prototype-wise Alignment

Besides, we propose the enhanced prototype-wise granularity alignment (EPGA) module that can achieve a more effective alignment of global visual and textual embeddings by utilizing cross-modal unified prototypes for both image and text modality as intermediaries. To begin, we establish trainable unified prototypes for both image and text modalities, denoted as C = {c1, c2, ..., cK}, where K signifies the count of trainable prototype vectors. More precisely, for each image-text embedding pair (vi, ti), we assign vi and ti to K unified prototypes in C, and obtain two soft prototype assignment codes qv,i ∈ RK and qt,i ∈ RK by using the Sinkhorn Knopp algorithm [Cuturi, 2013]. Following this, we compute the visual and textual softmax probabilities, pv,i ∈ RK and pt,i ∈ RK , respectively. Here pv,i and pt,i can be acquired by applying the softmax function to the cosine similarities between vi and all cross-modal unified prototypes in C, as well as between ti and all cross-modal unified prototypes in C as,

where τ denotes a cluster-level temperature factor, and k represents the k-th vector in unified prototypes C. EPGA can be achieved by optimizing the cross-entropy loss as,

where the EPGA is executed by utilizing the soft text prototype assignment qt,i as the "pseudo-label" for training the visual embedding vi, while the soft image assignment qv,i is employed as the "pseudo-label" for training the textual embedding ti. The overall LEPGA objective is calculated by,

By employing the losses LWIGA, LEPGA and LFMC, our model is trained by minimizing the loss as,

4 Experiments

4.1 Experimental Setup

Datasets. CUHK-PEDES [Li et al., 2017b] comprises 40,206 pedestrian images along with 80,412 text descriptions corresponding to 13,003 distinct pedestrian identities. Each individual image is accompanied by a minimum of two corresponding text descriptions. The training set includes 34,054 images, 68,108 textual descriptions, and 11,003 person identities. The test set contains 3,074 images and 6,156 textual descriptions, with 1,000 distinct person identities. ICFGPEDES [Ding et al., 2021] comprises 54,522 images with 4,102 distinct identities. Each person's image includes a corresponding textual description. The training set encompasses 34674 image-text pairs for 3102 different person identities. The test set consists of 19,848 image-text pairs.

Challenging Data Partitions. We define three distinct settings to represent varying levels of difficulty. For the easy setting, we use 50% of the training set as the complete imagetext pair data, 25% as missing image data, and 25% as missing text data, denoted as (50%, 25%, 25%). Similarly, we establish the medium setting, defined as (30%, 35%, 35%), and the hard setting as (10%, 45%, 45%) to elevate the training complexity. We employ Rank-k (where k = 1, 5, 10), a commonly used metric in text-image person retrieval.

Implementation Details. In our experiments, we adopt the image encoder and text encoder components of the Clip [Radford et al., 2021] model to serve as the feature extractors. During training, image data augmentation is applied through the incorporation of random horizontal flipping, random cropping, and random erasing techniques. All images are resized to 384 × 128 pixels. For the text modality, the maximum length of text tokens is set to 80. The model is optimized via the Adam optimizer [Kingma and Ba, 2014] with a 0.0001 learning ratio. The batch size is set to 64, and the training process spans across a total of 60 epochs. The temperature parameter τ (Equations 19 and 24) is set to 0.02.

4.2 Comparison with State-of-the-Art Methods

Comparisons on Incomplete Modal Data. We initially evaluate the proposed ECCA method on the widely-used CUHKPEDES and ICFG-PEDES using unsupervised incomplete modal data. As shown in Tables 1 and 2, our ECCA outperforms SOTA text-image person retrieval approaches in three distinct settings, including unsupervised IRRA and supervised AXM-Net, LGUR, SSAN, ViTAA, SCAN, MIA and CMPM/C methods. More specifically, our ECCA improves the unsupervised IRRA method (same feature extractor as ours) by 1.43%, 4.61% and 5.96% Rank-1 accuracy on CUHK-PEDES, by 2.53%, 3.68% and 4.47% Rank-1 accuracy on ICFG-PEDES under three different settings, respectively. It can be observed that these methods suffer significant performance degradation when encountering incomplete data. Therefore, the performance improvements on the easy setting are not as prominent as on the hard setting. Under the hard setting, our method achieves 56.38% and 42.08% Rank1 accuracy on CUHK-PEDES and ICFG-PEDES, which fully demonstrates that our method can effectively deal with incomplete data and improve the robustness of the model.

Comparisons on Complete Modal Data. To further verify the more robust advantage of our ECCA model for finegrained cross-modal semantic alignment, we compare our model with several SOTA text-image person retrieval methods. In Table 3, for CUHK-PEDES with full modality data, our model surpasses unsupervised CMMT, MM-TIM and fully supervised AXM-Net, LGUR, CAIBC, IVT, TextReID and SSAN methods, and achieves 68.13% on Rank-1, 87.26% on Rank-5 and 91.88% on Rank-10. These experimental results fully demonstrate that our method can be applied to more realistic scenarios, such as the lack of identity labels.

4.3 Ablation Studies

Analysis of Feature-level Missing Modality Completion (FMC). As illustrated in Table 4, to verify the effectiveness of our feature-level missing modality completion (FMC), our method is trained under the medium setting on four distinct training sets, including 1) only complete modal data, 2) missing visual data, 3) missing textual data and 4) complete all missing data. It can be observed from these experimental results that the accuracy of Rank-1, 5, 10 using only the complete data is the worst. As we reconstruct incomplete visual data or incomplete text data, experimental performance is gradually improved. The experimental performance reaches a maximum until all incomplete data are completed. The ECCA with complete all missing data improves ECCA with only complete modal data by 5.23% Rank-1 accuracy and 4.25% Rank-5 accuracy on CUHK-PEDES dataset under the medium setting. These results demonstrate that our featurelevel missing modality completion can reduce the impact of performance degradation caused by incomplete data.

Ablations on High Semantic Similarity Neighbor Generation. In Table 5, to verify the effectiveness of our featurelevel missing modality completion (FMC) with high semantic similarity neighbor generation (HSNG) in our ECCA, we conduct ablation experiments on CUHK-PEDES dataset under the easy setting. WIGA + EPGA represents weighted inter-instance granularity alignment and enhanced prototypewise granularity alignment. WIGA + EPGA adding HSNG

achieve 61.68 % on Rank-1 and 81.91 % on Rank-5, and WIGA + EPGA adding FMC with general nearest neighbor completion achieve 62.14% on Rank-1 and 82.56% on Rank5. However, our WIGA + EPGA adding FMC with HSNG consistently surpasses both cases. This verifies that the FMC with HSNG can effectively recover the missing features, and fully mine the side information of the missing data.

Performance comparisons of different losses. To demonstrate the effectiveness of our proposed weighted inter-instance granularity alignment (WIGA) and enhanced prototype-wise granularity alignment (EPGA), as shown in Table 6, we compare experimental results of the commonly used CMPM loss [Zhang and Lu, 2018], Ranking loss [Faghri et al., 2017], InfoNC loss [Oord et al., 2018] with our proposed WIGA and EPGA loss on CUHK-PEDES dataset under the full multimodal data. Specifically, our WIGA + EPGA achieves 68.13% on Rank-1, 87.26 % on Rank-5 and 91.88% on Rank-10, and consistently outperforms CMPM loss, Ranking loss, and InfoNC loss by 8.96%, 6.37% and 5.74 % Rank-1 accuracy, respectively. This is because our WIGA and EPGA effectively handle the cross-modal matching ambiguity caused by the absence of true label supervision, which encourages the model to map semantically similar image-text pairs more compactly. This fully validates that our WIGA and EPGA can achieve tighter fine-grained crossmodal semantic alignment in unsupervised scenarios.

4.4 Parameter Analysis

In this section, we conduct hyperparameter analysis experiments on the CUHK-PEDES dataset for the temperature τ, the number of the shared prototypes s in D of Equations (1)

k

Figure 4: Parameter analysis with different values of temperature τ , number of prototypes s under full data and nearest neighbors k and k′ under the Easy setting on CUHK-PEDES.

and (2) under the full data, the mutual neighbor values k and k′ in Equations (5) and (7) under the easy settings. We transform the values of τ, s, k and k′ in a certain range, report the corresponding Rank-1 and mAP values, respectively, and then perform experimental analysis. As illustrated in Figure 4, a) experimental results show that as the value of τ rises, the Rank-1 and mAP accuracy initially increase and then decrease more. Our method achieves a more stable performance when τ is set to 0.02. b) When s is approximately equal to 400, the model reaches the optimal value, which indicates that the number of prototypes is sufficient for learning shared features between images and texts. c, d) The peak performance is achieved when k = 6 and k′ = 4 on the CUHK-PEDES dataset. This trend underscores that excessively large values of k and k′ will increase the probability of false neighbors belonging to distinct person identities, leading to a reduction on Rank-1 and mAP.

5 Conclusions

In this paper, we propose a novel enhancing cross-modal completion and alignment (ECCA) framework for unsupervised incomplete text-image person retrieval task. Specifically, we introduce a feature-level cross-modal completion technique tailored for incomplete data. In addition, we achieve a tighter semantic fine-grained alignment between images and texts by integrating weighted inter-instance granularity alignment and enhanced prototype-wise granularity alignment. Extensive experimental results on public datasets fully demonstrate the effectiveness of our method in the face of substantial missing data.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grant 62172212, in part by the Natural Science Foundation of Jiangsu Province under Grant BK20230031.