Cross-modal Generation and Alignment via Attribute-guided Prompt for Unsupervised Text-based Person Retrieval

1 Introduction

Text-based Person Retrieval involves searching for a person of interest from a large-scale image gallery according to a provided textual query [Li et al., 2017; Farooq et al., 2022; Li et al., 2023]. As a sub-task of person re-identification

∗Corresponding author

(Re-ID)[He et al., 2022; Wang et al., 2022a], the textbased person retrieval solely adopts the language description to locate the target person, which offers greater accessibility and closely mirrors real-world scenarios. However, this task presents challenges arising from the imperative need for feature alignment in two distinct modalities. Therefore, a large number of approaches [Zhu et al., 2021; Shao et al., 2022; Farooq et al., 2022; Jiang and Ye, 2023; Jing et al., 2024] have been developed to attain modalityinvariant features by aligning fine-grained visual and textual attributes, leading to impressive results. However, these methods demand the presence of both textual descriptions and identity labels, thus leading to significant investments in terms of time and resources.

In response to the challenges of costly labeling, researchers have directed their attention toward mitigating the expenses associated with annotating images from multiple non-overlapping cameras with captions and identity labels. Specifically, MAN [Jing et al., 2020] proposes a moment alignment network to address the cross-domain text-based person search task, where pairwise text-image identity labels are absent in the target domain. Similarly, CMMT [Zhao et al., 2021] introduces a Cross-Modal Mutual Training framework for weakly supervised person search, where solely image-text pairs are available without any identity annotations. GTR [Bai et al., 2023b] proposes a generationthen-retrieval framework without parallel Image-Text data, liberating it from the dependence on manually labeled text descriptions. However, the assignment of one hot positive and negative label fails to accurately capture the non-strict mutual exclusion relations between image and text pairs, particularly when the generated caption is not sufficiently precise.

Therefore, to eliminate the cost associated with manual annotation, we aim at the Unsupervised Text-based Person Retrieval, as depicted in Figure 1(c), which involves solely image data without any additional information. To solve this challenging task, we leverage the potential of the visionlanguage pre-training model and propose a Cross-modal Generation and Alignment via Attribute-guided Prompt (GAAP) framework, encompassing two key processes: Attributeguided Prompt Caption Generation and Attribute-guided Cross-modal Alignment. GAAP employs attribute-based prompts to generate pseudo text and subsequently aligns the coarse textual information with finely detailed attribute prompts during the finetuning process. The overall architecture of GAAP is illustrated in Figure 2.

To alleviate the issue of missing image-text pairs, Attribute-guided Prompt Caption Generation is proposed to generate pseudo captions. Specifically, prompts for various person attributes are established for fine-grained captions. These prompts are then combined with the images and input into the text and image encoder within the pre-trained vision-language model BLIP[Li et al., 2022]. By calculating the similarity between the image and attribute prompt embeddings, the corresponding attributes can be determined by choosing the optimal matching. All identified attributes are then aggregated to form appropriate captions. Furthermore, to enhance sentence coherence and style diversity, we leverage Large Language Model (LLM) to reconstruct the generated captions, resulting in more comprehensive and varied sentences. Additionally, the image and its newly constructed caption are sent to the ITM head, facilitating the computation of a matching score. This matching score serves as the basis for sample selection, wherein only reliable pairs are utilized for the subsequent fine-tuning process.

Leveraging the generated image-text pairs, we present an Attribute-guided Cross-modal Alignment module to align cross-modal features, which is mainly comprised of three sub-modules: Cross-Modal Center Alignment (CMCA), Attribute-guided Image-Text Contrastive learning (AITC), and Attribute-guided Image-Text Matching (AITM). In the CMCA module, we assign the image-text pairs with the same attribute set with identical identity labels. And the imagebased and text-based class memories are established by computing average features from identical identities. The pairs with notably significant disparities in class similarities are subsequently omitted. Considering the potential overlap in local attributes across different image-text pairs, we leverage the local attribute information to guide the cross-modalities interaction. Specifically, in the AITC module, the attributeimage similarity is employed to guide the text and image relations, weakening the strict cross-modal constraint. Similarly, the AITM loss is proposed to align image caption matching score with attribute-based similarity by introducing soft targets rather than original strict positive or negative labels. This approach can mitigate the adverse impact of the noise pairs and the unpaired samples exhibiting certain overlaps.

The main contribution of our proposed framework can be summarized as follows:

1. We propose a Cross-modal Generation and Alignment via Attribute-guided Prompt framework for unsupervised text-based person retrieval. By leveraging the attributebased prompt, GAAP effectively generates text captions and aligns the cross-modal relation.
2. We propose an Attribute-guided Prompt Caption Generation module for caption generation. Incorporating with LLM and BLIP, we generate image captions and select reliable image-text pair for unsupervised finetuning.
3. We propose an Attribute-guided Cross-modal Alignment module to align the cross-modal relation by employing fine-grained attribute-image similarity as softened targets. Extensive experiments demonstrate the effectiveness of our proposed GAAP framework, which brings significant improvements on several text-based person retrieval datasets.

2 RelatedWork

2.1 Pre-TrainingMethods

Pre-trained models have demonstrated impressive performance across a wide array of tasks and can be broadly classified into three main categories: Vision pre-training Models, Vision-Language pre-training Models, and Language pretraining Models. Vision pre-training Model usually performs self-supervised [He et al., 2020] on large image datasets, while Vision-Language pre-training Model effectively learns modality-invariance features from large image-text datasets, with CLIP [Radford et al., 2021] as a prominent example. Subsequent to CLIP, SoftCLIP [Gao et al., 2023] and PyramidCLIP [Gao et al., 2022] employ a softened target to achieve a soft cross-modal alignment. Moreover, BLIP [Li et al., 2022] leverages CapFilt to generate image descriptions and eliminate noisy text effectively. In the text-based person search field, PLIP [Zuo et al., 2023] generates attributes and stylish textual captions based on these attributes. Similarly, APTM [Yang et al., 2023] generates pedestrian images and corresponding captions based on attributes, and regularizes the model training by leveraging both the attribute recognition task and the text-based person retrieval task.

Early Language pre-training Models like BERT [Devlin et al., 2018] and GPT-2 [Radford et al., 2019] play a foundational role in the development of the Large Language Models. Subsequently, GPT-3 [Brown et al., 2020] emerges as a revolutionary model capable of achieving remarkable performance across diverse tasks, eliminating the need for gradient updates or fine-tuning. GLM [Du et al., 2022] operates on the principle of autoregressive blank infilling. On the other hand, ChatGLM2-6B showcases the dual advantages of being lightweight and high-performing.

In our paper, we explore the potential of employing the pre-training models for fully unsupervised text-based person search. Subsequently, ChatGLM2-6B was employed to generate diverse styles of texts, while BLIP was adopted for caption generation and retrieval.

2.2 Text-Based Person Retrieval

Li et al. [Li et al., 2017] first propose the text-based person retrieval task and introduce a cross-modal dataset with imagetext pairs, CUHK-PEDES. Recent methods can be categorized into three groups:global-based [Zhang and Lu, 2018; Zhu et al., 2021; Jiang and Ye, 2023], local-based [Shen et al., 2023; Farooq et al., 2022; Yan et al., 2022a; Jing et al., 2023], and attribute-based [Aggarwal et al., 2020; Wang et al., 2020; Zuo et al., 2023; Yang et al., 2023] approaches. In the global-based methods, IRRA [Jiang and Ye, 2023] presents a groundbreaking use of the complete CLIP [Radford et al., 2021] for text-based person retrieval. In the local-based methods, CFine [Yan et al., 2022a] skillfully leverages CLIP's image encoder to capture rich multimodal information. For the attribute-based methods, CMAAM [Aggarwal et al., 2020] extracts attributes from the text in the training set as annotations, while ViTAA [Wang et al., 2020] employs semantic segmentation to categorize pedestrians into different body parts.

Although these supervised methods can achieve satisfactory performance, they require a substantial amount of textimage labels. Therefore, researchers are devoted to exploring weakly supervised methods. MAN [Jing et al., 2020] pioneers the field with a cross-domain moment alignment network for cross-domain text-based person retrieval. In contrast, CMMT [Zhao et al., 2021] addresses the absence of ID labels by generating pseudo labels through clustering. However, it is important to note that all these approaches necessitate the use of text data, resulting in a substantial requirement for manual annotation efforts. In light of this, GTR [Bai et al., 2023b] presents an alternative approach by generating textual descriptions corresponding to person images through VQA. Nonetheless, it falls short in considering the local similarity between different image-text pairs, which is different from our work.

3 Methodology

3.1 Overview

In the unsupervised text-based person search, a training dataset denoted as I = {xi}Ni=1 with only image data is given, where xi represents the i-th image and N is the total number of images. The principal aim of this task is to learn discriminative multi-modal features with image-only data.

Our proposal Unsupervised framework, namely Crossmodal Generation and Alignment via Attribute-guided Prompt (GAAP), is composed of two principal components: Attribute-guided Prompt Caption Generation and Attributeguided Cross-modal Alignment, as depicted in Figure 2. The former component leverages the attribute-based prompts in conjunction with a pre-trained vision-language model to generate pseudo captions. On the other hand, the latter component aims to establish alignment between image-caption relationships at a fine-grained attribute level, which is achieved through the incorporation of three distinct sub-modules: Cross-modality Center Alignment (CMCA), Attribute-guided Image-Text Contrastive Learning (AITC), and Attributeguided Image-Text Matching (AITM).

3.2 Attribute-Guided Prompt Caption Generation

Given the image data within the realm of unsupervised textbased person search, the initial step involves generating pseudo captions for each individual's images. To facilitate this, we formulate distinct attribute text prompts denoted as A = {a1, a2, ..., an}, where n signifies the count of binary attributes. We feed the image into the visual encoder, yielding the image embedding denoted as Eimg(xi). Concurrently, the attribute prompts P are directed to the text encoder, producing prompt embeddings {Etext(a1), ...,Etext(an)}. Subsequently, we compute the cosine similarity between the image and each attribute, and the softmax is used to normalize the similarities of attributes belonging to the same category:

where S(aj) denotes the attribute set containing attributes of the same category with aj, e.g., color set: yellow, red,...blue. Furthermore, the image and attribute prompts are sent into the cross-modal encoder to obtain the matching score, denoted as m(ai, x). Then the attribute-image similarity can be established by averaging the matching score and cosine similarity, expressed as ˆ s(x, ai) = m(x, ai) + p (x, ai).

The prediction for each attribute set is realized by a systematic selection of the attribute-image score that boasts the highest value within each individual attribute set. This approach results in the acquisition of an attribute label set associated with the image xi, denoted as Ai = {a1, ..., aAi }. Subsequently, these attributes are fused into a template with masked sentences to construct the caption ti for image xi.

Subsequent to obtaining corresponding pseudo captions according to the predicted attribute, we leverage LLM to generate distinct stylistic captions by giving the prompts:" I will give you a sentence, please return a sentence with different styles with the same semantics as if it is spoken by different people: [CAPTION] ". Then the template-based captions and the rephrased captions combined with the images are fed into the cross-modal encoder, resulting in matching scores m(t, x) between the captions and images. The text-image matching scores representing reliability are used to filter out noisy image-text pairs that could potentially impair performance. Consequently, we set the sample selection threshold th ∈ [0, 1], whereby solely the image-text pairs with matching scores of m(t, x) > th are preserved, while other samples are considered outliers.

3.3 Attribute-Guided Image-Text Alignment

Cross-Modal Center Alignment. After the acquisition of the pseudo image-text pairs, the absence of identity labels for these image-text pairs also presents a notable challenge. To address this issue, we adopt a strategy of assigning identical identity labels to image-text pairs that share a common attribute set. However, employing the ID labels solely based on the generated attribute set can potentially result in visual dissimilarity among images due to potential noise introduced during the generation process. To mitigate this issue, we employ the similarity between samples and class centers as a metric for assessing cluster reliability, enabling the filtration of samples with lower ID similarity. Specifically, we define the cluster reliability as the cosine similarity between the image feature and the corresponding center feature, expressed as R(xi) = cos(Eimg(xi),CI[yi]), where yi is the assigned identities of the image, and CI[yi] represents the image class center obtained by averaging the image features belonging to the same identities. Consequently, images with lower ID similarity with R(xi) < th2 are regarded as cluster outliers. This selection operates on the class level by measuring the sample-class similarity, while sample selection is performed on the pair level.

After assigning the pseudo-identity label Y , we construct image and caption class-level memory by averaging the features belonging to the same identities, which can be represented as CI = {c1i , ..., cNc i } and CT = {c1t , ..., cNc t }. Therefore the image class-level contrastive loss can be denoted as a nonparametric Softmax loss[Xiao et al., 2017]:

where N represents the total number of images, yi is the identity label of image xi and τ is temperature hyper-parameter. During each iteration, the memory bank is updated by:

where m is the momentum parameter for updating the features in class memory.

Moreover, to learn the class semantic cross-modal relations, the text class-level memory is used to guide the image features by a text center alignment loss:

where CT is the text class memory. Similar to the image modality, the text is also supervised by both image and text memory to learn the cross-modal class relations. Therefore, the cross-modal center alignment loss can be formulated as:

Attribute-guided Image-Text Contrastive Learning. As a cross-modal retrieval task, our approach involves the ImageText contrastive loss (ITC) to impose the cross-modal alignment. The ITC loss facilitates the learning of the relationship between the positive and negative pairs by pulling the matched pair and pushing the unpaired apart. Given a batch of NB image-text pairs, we first calculate the normalized similarity between the images and text in a batch. The image-totext similarity pi2t i (I) = {pi2t i,j (I)}Nj=1 can be denoted as:

where τ is a learnable temperature parameter, sim(·) represents the cosine similarity, and ti denotes the pseudo captions for i-th sample. Similarly, the text-to-image similarity pt2i i (T) can also be obtained. And the one-hot label yi = {yij}NB j=1 with the image-text paris with the same pseudo identity label as 1 and others as 0. Therefore the ITC loss can be denoted as:

where NB is the batch size and H(·, ·) represents the cross entropy loss.

As aforementioned, such ITC loss may neglect some local attribute similarity between the image-text pairs in a minibatch. To explore more fine-grained relations between images and text, we use the image attribute similarity to provide the implicit local relation knowledge. Formally, the text Ti consists of several corresponding attributes, Ai = {a0, ...aNAi }. Here, Ai corresponds to the predicted attribute for image xi. Therefore, the image-attribute contrastive loss can be denoted as:

where S(aj) is the attribute set with the same category with attribute aj . The image and the attribute prompts are fed into the image and text encoder, obtaining the attribute and image embeddings to calculate the image-attribute similarity:

where xi,Aj is the image and attribute set of i-th and j-th samples in a batch. And NAj is the number of matched attributes for j-th sample.

Therefore the normalized image-attribute similarity, denoted as pi2a i (I) = {pi2a i,j (I)}Nj=1 can be calculated by:

The image-attribute similarity is used to smooth the hard target label for solving the non-strict relation between image and text, the attribute-guided image-text label can be formulated as:

Similarly, the attribute-image similarity pa2i i (A) = {pa2i i,j (A)}Nj=1 and attribute-guided text-image label ˜ yt2i i can also be obtained through Eq 10 and 11. The attribute-guided image-text alignment loss can be calculated by incorporating local similarity with attribute-guided labels, thereby mitigating the effects of the strict regularization:

The attribute-guided image-text contrastive loss can be computed by aggregating the image-attribute contrastive loss and the attribute-guided image-text alignment loss:

Attribute-guided Image-Text Matching. The primary objective of the ITM loss is to predict the positivity or negativity of the given image-text pair. The image embedding and text are sent into the cross-modal encoder M, following a fully-connected layer to predict the matching score pitm. Therefore the ITM can be formulated as:

where yitm represents the ground-truth label for ITM, 1 when image text matched, 0 otherwise.

The image-text positive pairs may not always be matched since the generated caption may contain noise, while the negative pairs may also contain some local similarity information. However, the one-hot GT label assumes there are strictly matched positive pairs, neglecting the local attribute relations. Therefore, attribute-guided Image-Text Matching aims to soften the hard GT labels with the image-attribute matching score, which can be denoted as mi(I,A) = 1 NAj ∑ak∈Aj M (xi, ak), where M is the cross-modal encoder. This image-attribute matching score is used to soften the one hot image-text matching label, which can be denoted as:

where α2 is the weighing parameter. The attribute-guided image-text matching loss can be denoted as:

Given the above objectiveness, the final optimization function can be formulated as:

where λ1, λ2, λ3 are loss weight.

4 Experiments

4.1 Datasets and Protocol

Our approach is meticulously evaluated across three widely used datasets: CUHK-PEDES [Li et al., 2017], ICFGPEDES [Ding et al., 2021], and RSTPReid [Zhu et al., 2021]. To ensure adherence to an unsupervised approach, only image data is employed during the training process to generate captions. In the testing phase, we utilize the captions from the dataset for retrieval.

The CUHK-PEDES dataset, pioneering in text-based person retrieval, comprises 34,054/68,108 images/sentences of 11,003 identities in the training set. The validation/test set contains 3,078/3,074 images. The ICFG-PEDES dataset consists of 54,522 images of 4,102 individuals, with one caption per image. The training/testing set comprises 34,674/19,848 image-text pairs of 3,102/1,000 identities. The RSTPReid dataset encompasses 20,505 images of 4,101 identities, with each image accompanied by two textual descriptions. The training/validation/testing sets include 3,701/200/200 identities, respectively.

The widely adopted Rank-K metric serves as one of the fundamental evaluation measures. Specifically, it involves identifying the most relevant one/five/ten image(s) based on the similarities between text and images. Furthermore, we also incorporate the mean average precision (mAP) as an additional comprehensive assessment.

4.2 Implementation Details

The Image Encoder, Text Encoder, and Image-grounded Text Encoder in BLIP [Li et al., 2022] have been incorporated into our methodology. In the Attribute-guided Prompt Caption Generation module, we commence the process by generating the initial caption through 8 distinct prompts, encompassing 47 attributes, and another stylistic caption is created using ChatGLM2-6B. During this process, the dimension of images is set to 256×256, while the th stands at 0.9. In the Attribute-guided Cross-modal Alignment phase, the batch size is set to 32. The optimization process employs the AdamW [Loshchilov and Hutter, 2017] optimizer, with a decay rate of 0.05 and an initial learning rate of 1e-5. α1 and α2 have been assigned values of 0.5 and 0.2, respectively, while the parameter m in Equation 3 has been set to 0.2. The threshold th2 for label assignment has been established as 0.84. Image augmentation contains random horizontal flipping and RandAugment [Cubuk et al., 2020] techniques.

4.3 Comparison with the State-of-the-Art

To verify the superiority of our proposed framework, We conducted experiments in three widely used text-based person search datasets. Our baseline is built on the basis of BLIP [Li et al., 2022], using our generated captions to train the model.

CUHK-PEDES. We compare our framework with SOTA methods on CUHK-PEDES, as shown in Table 1. Our method achieves 47.64% Rank-1 and 41.28% mAP, which is 6.77% and 5.56% higher than the previous method. Besides, the performance has a remarkable improvement of 8.28% at Rank-1 and 6.93% at mAP compared with baseline. Moreover, our method even outperforms some supervised methods.

ICFG-PEDES. Our framework achieves competitive performance on the ICFG-PEDES dataset, as shown in Table 2. ICFG-PEDES is the most challenging dataset with a large query in the test set. GAAP set the new SOTA Rank-1 of 27.12% and mAP of 11.43%, surpassing the baseline by 4.61% Rank-1 and 0.58% mAP.

RSTPReid. As shown in Table 3, the baseline achieves 39.60% Rank-1 and 27.61% mAP accuracy on RSTPReid while GAAP reaches 44.45% Rank-1 and 31.21% mAP.

4.4 Ablation Study

A series of ablation experiments are performed to analyze the effectiveness of components within the GAAP framework on the CUHK-PEDES dataset, as shown in Table 4.

Ablations on the components. We conduct experiments to investigate the efficacy of the Sample Selection and LLM rephrase module in the caption generation module. The sample selection (SS) is used to select clean image-text pairs for training. As shown in Table 4, SS contributes significantly, bringing an improvement of 4.9% and 3.2% on rank-1 and mAP, This result validate the effectiveness of removing noisy pairs, thereby enhancing overall performance. When adding the LLM Rephrase module, our method gains a commendable 1.1% for Rank-1 and 1.0% for mAP. It validates that integrating more stylish captions can bolster performance.

In the Attribute-guided Cross-modal Alignment module, CMCA is proposed to narrow the gap between text and image samples with cross-modal center features, which improves the performance by 1.3% and 1.6% on Rank-1 and mAP. Furthermore, the discernible efficacy of AITM becomes evident, culminating in performance gains of approximately 0.6% and 0.7%, as shown in Model 3. Besides, AITM gains a 0.5% improvement on mAP. As mentioned before, AITM and AITC are different from vanilla ITM and ITC, which are designed for attribute-based work.

Parameters analysis. The Sample Selection threshold, denoted as th, serves the purpose of choosing reliable pairs whose matching score surpasses this threshold. The observations in Figure 3 elucidate that an increase in the value of th corresponds to an improvement in performance. This phenomenon arises from the utilization of cleaner samples during training. The weighting parameter α1 aids in harmonizing the ground truth label and the image-attribute score. Consequently, as α1 ranges from 0 to 0.4, a discernible enhancement in performance is observed. This can be attributed to the contribution of the attribute-guided score, effectively relieving the strict regulation. However, as α1 continues to increase, the performance starts to decline, stemming from the diminishing contribution of the GT labels. This observation validates of the efficacy of our proposed approaches.

Retrieval results comparison. We additionally present a comparative analysis of the top-10 retrieval results between the baseline method and the proposed GAAP method. The integration of attribute guidance in our approach enhances its capability to discern finer details and information during the retrieval process.

5 Conclusion

In this paper, we introduce a Cross-modal Generation and Alignment via Attribute-guided Prompt framework (GAAP) for unsupervised text-based person search. Our approach leverages a pre-trained vision-language model to generate person attributes using a diverse set of attribute prompts with a sample selection module to identify reliable training samples. Subsequently, we propose an Attribute-based Crossmodal Alignment module to effectively align image and text features with attribute-guided assistance. Through extensive experimental validation, we demonstrate the efficacy of our method for unsupervised text-based person retrieval.

Contribution Statement

Zongyi Li and Jianbo Li made equal contributions. All the authors participated in designing research, performing research, analyzing data, and writing the paper.

Acknowledgements

This work was supported in part by the Natural Science Foundation of China under Grant 61972169,62302186 and 62372203, in part by China Postdoctoral Science Foundation 2022M711251, in part by the National key research and development program of China(2022YFB2601802), in part by the Major Scientific and Technological Project of Hubei Province (2022BAA046, 2022BAA042), in part by the Knowledge Innovation Program of Wuhan-Basic Research.