RaSa: Relation and Sensitivity Aware Representation Learning for Text-based Person Search

1 Introduction

Text-based person search [Li et al., 2017; Wang et al., 2021] aims at retrieving the person images in a large-scale person image pool given a query of textual description about that person. This task is related to person re-identification [Ji et al., 2021; Wang et al., 2022a] and text-image retrieval [Cao et al., 2022; Li et al., 2021], which have been very active research topics in recent years. It, however, exhibits unique characteristics and challenges. Compared to person re-identification with image queries, text-based person search with more accessible open-form text queries provides a more user-friendly searching procedure while embracing greater challenges due to the cross-modal search. In addition, compared to general image-text retrieval, text-based person search focuses on cross-modal retrieval specific for the person with more finegrained details, tending to larger intra-class variance as well as smaller inter-class variance, which toughly bottlenecks the retrieval performance.

∗Corresponding author

Targeting learning powerful feature representation and achieving cross-modal alignment for text-based person search, researchers have developed a batch of technologies over the past few years [Wu et al., 2021; Shao et al., 2022]. It has been proved that the model armed with reasonable tasks tends to learn better representation. In this paper, we propose a representation learning method, namely RaSa, with two novel tasks: relation-aware learning and sensitivity-aware learning for text-based person search.

Relation-aware Learning. In existingmethods [Han et al., 2021; Li et al., 2022], the de facto optimization objective is to bring image and text representations of the same identity (i.e., positive pairs) together and repel representations of different identities (i.e., negative pairs) away. However, it tends to encounter the following issue. Normally, a textual description is generated by annotating a particular single image in the text-based person search dataset. The text strongly matches the annotated image without a doubt, whereas it is not always well-aligned to other positive images of the same person at the semantic level due to intra-class variation in the image. As shown in Figure 1 (a), the images and texts depict the same person, leading to a positive relation for each imagetext pair. However, there exist two different types of positive relations. text1 (resp. text2) is the exact description of image1 (resp. image2), where they are completely matched and form a strong positive pair. Nevertheless, image1 and text2 (resp. image2 and text1) constitute a weak positive pair with the noise interference. For instance, "white t-shirt" and "blue shorts" in text1 correspond to non-existent objects in image2 due to the occlusion. Existing methods endow the strong and weak positive pairs with equal weight in learning representations, regardless of the noise problem from the weak pairs, eventually leading to overfitting learning.

In order to mitigate the impacts of the noise interference from weak positive pairs, we propose a Relation-Aware learning (RA) task, which is composed of a probabilistic ImageText Matching (p-ITM) task and a Positive Relation Detection (PRD) task. p-ITM is a variant of the commonly-used ITM, aiming to distinguish negative and positive pairs with a probabilistic strong or weak positive inputting, while PRD is designed to explicitly makes a distinction between the strong and weak positive pairs. Therein, p-ITM emphasizes the consistency between strong and weak positive pairs, whereas PRD highlights their difference and can be regarded as the regularization of p-ITM. The model armed with RA can not only learn valuable information from weak positive pairs by p-ITM but also alleviate noise interference from them by PRD, eventually reaching a trade-off.

Sensitivity-aware Learning. Learning invariant representations under a set of manually chosen transformations (also called insensitive transformations in this context) is a general practice for improving the robustness of representation in the existing methods [Chen and He, 2021]. We recognize it but there is more. Inspired by the recent success of equivariant contrastive learning [Dangovski et al., 2022], we explore the sensitive transformation that would hurt performance when applied to learn transformation-invariant representations. Rather than keeping invariance under insensitive transformation, we encourage the learned representations to have the ability to be aware of the sensitive transformation.

Towards this end, we propose a Sensitivity-Aware learning (SA) task. We adopt the word replacement as the sensitive transformation and develop a Momentum-based Replaced Token Detection (m-RTD) pretext task to detect whether a token comes from the original textual description or the replacement, as shown in Figure 1 (b). The closer the replaced word is to the original one (i.e., more confusing word), the more difficult this detection task is. When the model is trained to well solve such a detection task, it is expected to have the ability to learn better representation. With these in mind, we use Masked Language Modeling (MLM) to perform the word replacement, which utilizes the image and the text contextual tokens to predict the masked tokens. Furthermore, considering that the momentum model, a slow-moving average of the online model, can learn more stable representations than the current online model [Grill et al., 2020] to generate more confusing words, we employ MLM from the momentum model to carry out the word replacement. Overall, MLM and m-RTD together form a Sensitivity-Aware learning (SA), which offers powerful surrogate supervision for representation learning.

Our contributions can be summarized as follows:

• We differentiate between strong and weak positive image-text pairs in learning representation and propose a relation-aware learning task.
• We pioneer the idea of learning representation under the sensitive transformation to the text-based person search and develop a sensitivity-aware learning task.
• Extensive experiments demonstrate RaSa outperforms existing state-of-the-art methods by 6.94%, 4.45% and 15.35% in terms of Rank@1 metric on CUHK-PEDES, ICFG-PEDES and RSTPReid datasets, respectively.

2 RelatedWork

2.1 Text-based Person Search

Li et al. [2017] first introduce the text-based person search task and publish a challenging dataset CUHK-PEDES. Following this, a series of methods are proposed to solve this task. Part of methods [Zheng et al., 2020a; Wang et al., 2021] focus on designing a reasonable cross-modal alignment strategy, while others [Zhang and Lu, 2018; Shao et al., 2022] concentrate on learning powerful feature representation. For cross-modal alignment, it begins with global alignment [Zheng et al., 2020b] or local correspondences (e.g., patch-word or region-phrase correspondences) [Chen et al., 2022; Niu et al., 2020], and evolves into self-adaptively learning semantic alignment across different granularity [Li et al., 2022; Gao et al., 2021]. Beyond that, some works [Wang et al., 2020; Zhu et al., 2021] utilize external technologies (e.g., human segmentation, pose estimation or attributes prediction) to assist with the cross-modal alignment. For representation learning, Wu et al. [2021] propose two color-related tasks based on the observation that color plays a key role in textbased person search. Zeng et al. [2021] develop three auxiliary reasoning tasks with gender classification, appearance similarity and image-to-text generation. Ding et al. [2021] firstly notice the noise interference from weak positive pairs and propose to keep the difference between strong and weak positive pairs by manually assigning different margins in the triplet loss. More recently, some works [Han et al., 2021; Shu et al., 2022; Yan et al., 2022] resort to vision-language pretraining models to learn better representations. In this paper, we design two novel tasks: RA and SA. RA detects the type of the positive pair to weaken noise from weak positive pairs, differently from the method [Ding et al., 2021] with the sophisticated trick. SA focuses on representation learning by detecting sensitive transformation, which is under-explored in the previous methods.

2.2 Equivariant Contrastive Learning

Different from contrastive learning [He et al., 2020] that aims to learn transformation-insensitive representations, equivariant contrastive learning [Dangovski et al., 2022] is recently proposed by additionally encouraging the learned representations to have the ability to be aware of sensitive transformations. Mathematically, the notions of insensitivity and sensitivity can be inductively summarized as: f(T(x)) = T ′ (f(x)) where T denotes a group of transformations of an input instance x, and f is an encoder to compute the representation of x. When T ′ is the identity transformation, it can be said that f is trained to be insensitive to T; otherwise, f is sensitive to T. Equivariant contrastive learning has shown its successful application in the fields of computer vision (CV) [Dangovski et al., 2022] and natural language processing (NLP) [Chuang et al., 2022], which inspires us to explore sensitive transformations for learning high-quality representations in the cross-modal retrieval task. In this paper, we develop a sensitivity-aware learning with MLM-based word replacement as the sensitive transformation to encourage the model to perceive the replaced words, thus obtaining more informative and discriminative representations.

3 Method

In this section, we take ALBEF [Li et al., 2021] as the backbone 1 and elaborate on the proposed method RaSa by introducing the modal architecture in Section 3.1 and the optimization objectives involving the proposed RA and SA tasks in Section 3.2.

1 More experiments on other backbones are shown in Appendix.

3.1 Model Architecture

As illustrated in Figure 2, the proposed RaSa consists of two unimodal encoders and a cross-modal encoder. We adopt 12layer and 6-layer transformer blocks for the image and text encoders, respectively. The cross-modal encoder comprises 6-layer transformer blocks, where a cross-attention module is added after the self-attention module in each block. Considering that the textual description usually covers a part of the information in the corresponding image, we employ a textguided asymmetric cross-attention module in the cross-modal encoder, i.e., using the textual representation as query and the visual one as key and value. Simultaneously, we maintain a momentum version of the online model via Exponential Moving Average (EMA). Specifically, EMA is formulated as ˆ θ = mˆ θ + (1 -m)θ, where ˆ θ and θ are the parameters of the momentum and online models, respectively, and m ∈ [0, 1] is a momentum coefficient. The momentum model presents a delayed and more stable version of the online model and is used to guide the online model to learn better representations.

Given an image-text pair (I, T), we first feed the image I into the image encoder to obtain a sequence of visual representations {vcls, v1, · · · , vM} with vcls being the global visual representation and vi (i = 1, · · · ,M) being the patch representation. Similarly, we obtain a sequence of textual representations {tcls, t1, · · · , tN} by feeding the text T into the text encoder, where tcls is the global textual representation and ti (i = 1, · · · ,N) is the token representation. The visual and textual representations are then fed to the cross-modal encoder to obtain a sequence of multi-modal representations {fcls, f1, · · · , fN}, where fcls denotes the joint representation of I and T , and fi (i = 1, · · · , N) can be regarded as the joint representation of the image I and the i-th token in the text T . Simultaneously, the momentummodel is employed to obtain a sequence of momentum representations.

3.2 Optimization Objectives

Relation-aware Learning

The vanilla widely-used ITM predicts whether an inputted image-text pair is positive or negative, defined as:

where H represents a cross-entropy function, yitm is a 2dimension one-hot vector representing the ground-truth label (i.e., [0, 1]⊤ for the positive pair, and [1, 0]⊤ for the negative pair), and ϕitm(I, T ) is the predicted matching probability of the pair that is computed by feeding fcls into a binary classifier, a fully-connected layer followed by a softmax function.

However, it is unreasonable to directly adopt the vanilla ITM in text-based person search. On the one hand, there exists noise interference from weak positive pairs, which would hamper the representation learning. On the other hand, the weak positive pairs contain certain valuable alignment information that can facilitate representation learning. As a result, to reach a balance, we retain a proportion of weak positive pairs in ITM by introducing the probabilistic inputting. Specifically, we input the weak positive pair with a small probability of pw and the strong positive pair with a probability of 1 - pw . To distinguish with the vanilla ITM, we denote the proposed probabilistic ITM as p-ITM.

Furthermore, we continue to alleviate the noise effect of the weak pairs. We propose a Positive Relation Detection (PRD) pretext task to detect the type of the positive pair (i.e., strong or weak), which is formulated as:

where (I, T p ) denotes a positive pair, yprd is the ground truth label (i.e., [1, 0]⊤ for the strong positive pair and [0, 1]⊤ for the weak pair), and ϕprd (I, T p ) is the predicted probability of the pair which is computed by appending a binary classifier to the joint representation fcls of the pair.

Taken together, we define the Relation-Aware learning (RA) task as:

where the weight λ1 is a hyper-parameter.

During the process of the optimization, p-ITM focuses on the consistency between strong and weak positive pairs, while PRD highlights their difference. In essence, PRD plays a role of a regularized compensation for p-ITM. As a whole, RA achieves a trade-off between the benefits of the weak pair and the risk of its side effects.

Sensitivity-aware Learning

Learning invariant representations under the insensitive transformation of data is a common way to enhance the robustness of the learned representations. We go beyond it and propose to learn representations that are aware of the sensitive transformation. Specifically, we adopt the MLM-based word replacement as the sensitive transformation and propose a Momentum-based Replaced Token Detection (m-RTD) pretext task to detect (i.e., being aware of) the replacement.

Given a strong positive pair (I, T s ), MLM loss is formu-

where T msk is a masked text in which each token in the input text T s is randomly masked with a probability of pm, ymlm is a one-hot vector denoting the ground truth of the masked token and ϕmlm(I, T msk ) is the predicted probability for the masked token based on the information of the contextual text T msk and the paired image I.

We use the result of MLM from the momentum model as the word replacement, denoted as T rep . The momentum model is a slow-moving of the online model and can learn more stable representations. Therefore, the momentum model is expected to generate more confusing tokens. As m-RTD detects such challenging tokens well, the model is motivated to learn more informative representations to distinguish the tiny differences. Remarkably, besides serving as a generator for the word replacement, MLM also plays a role of token-level optimization, promoting fine-grained representation learning.

Next, m-RTD performs a detection of the MLM-based token replacement. Specifically, the pair (I, T rep ) is inputted to the model to obtain a sequence of multi-modal representations {fcls, f1, ..., fN}, and a binary classifier works on {f1, ..., fN} to predict whether the i-th token is replaced or not. m-RTD minimizes a cross-entropy loss:

where ym-rtd is a one-hot vector denoting the ground truth of the replaced token and ϕm-rtd (I, T rep ) is the predicted replacement probability. We illustrate the pipeline of m-RTD in Figure 3 for clarity.

Overall, Sensitivity-Aware learning (SA) loss is defined as:

where the weight λ2 is a hyper-parameter.

In conclusion, RA works on the global representation fcls and mainly focuses on the correlation between the image and text, which can be regarded as a coarse-grained optimization. As a complement, SA acts on the token representations {f1, ..., fN} and pays more attention to the interaction between the image and textual tokens, exhibiting a fine-grained optimization. The two complementary tasks effectively facilitate representation learning.

Contrastive Learning

The proposed RA and SA are directly applied on the multimodal representations from the cross-modal encoder. Furthermore, we introduce an intermediate Contrastive Learning task (CL) on the representations from the unimodal encoders, so as to make the subsequent cross-modal fusion easier to performmulti-modal representation learning.

Given an image-text pair (I, T), we feed it into the unimodal encoders and obtain the global visual and textual representations vcls and tcls. Then a linear layer is applied to project them to lower-dimensional representations v′cls and t′cls . Meanwhile, we obtain the output of momentum unimodal encoders, denoted as ˆ v′cls and ˆ t′cls . We maintain an image queue ˆQv and a text queue ˆQt to store the recent R projected representations ˆ v′cls and ˆ t′cls , similarly to MoCo [He et al., 2020]. The introduction of the queues implicitly enlarges the batch size, and a larger batch will provide more negative samples, thereby facilitating representation learning.

In CL, the general form of InfoNCE loss is formulated as:

where τ is a learnable temperature parameter, Q denotes a maintained queue, and s(x, x+) = xTx+/∥x∥∥x+∥ measures the cosine similarity between x and x+.

Beyond the widely-used cross-modal image-text contrastive learning (ITC) [Li et al., 2021; Radford et al., 2021], denoted as:

we additionally explore the intra-modal contrastive learning (IMC). The representations of the same person are supposed to stay closer than those of different persons within each modality. IMC loss is formulated as:

Taken together, we define the overall loss for CL as:

Joint Learning

Overall, we formulate the joint optimization objective as:

where λ3 is a hyper-parameter.

During inference, given a query text and a large-scale image pool, we use the predicted matching probability from pITM to rank all images. Considering the inefficiency of the cross-modal encoder with quadratic interaction operation, we refer to ALBEF [Li et al., 2021] and exclude a large number of irrelevant image candidates prior to the cross-modal encoder, thereby speeding up the inference. Specifically, we first calculate each pair's similarity s(tcls, vcls) via the unimodal encoders, and then select the first 128 images with the highest similarities to send them to the cross-modal encoder and compute the p-ITMmatching probabilities for ranking.

4 Experiments

We conduct experiments on three text-based person search datasets: CUHK-PEDES [Li et al., 2017], ICFGPEDES [Ding et al., 2021] and RSTPReid [Zhu et al., 2021]. The introduction of each dataset and the implementation details of the proposed method are shown in Appendix.

4.1 Evaluation Protocol

We adopt the widely-used Rank@K (R@K for short, K=1, 5, 10) metric to evaluate the performance of the proposed method. Specifically, given a query text, we rank all the test images via the similarity with the text and the search is deemed to be successful if top-K images contain any corresponding identity. R@K is the percentage of successful searches. We also adopt the mean average precision (mAP) as a complementary metric.

4.2 Backbones

Most text-based person search methods [Li et al., 2022; Shao et al., 2022] rely on two feature extractors pre-trained on unaligned images and texts separately, such as ResNet [He et al., 2016] or ViT [Dosovitskiy et al., 2020] for the visual extractor, Bi-LSTM [Hochreiter and Schmidhuber, 1997] or BERT [Devlin et al., 2018] for the textual extractor. Recently, some works [Shu et al., 2022; Yan et al., 2022] have applied vision-language pretraining (VLP) to text-based person search and obtained impressive results. Following this, we adopt VLP models as the backbone.

The proposed RaSa can be plugged into various backbones. To adequately verify the effectiveness, we conduct RaSa on three VLP models: ALBEFF [Li et al., 2021], TCL [Yang et al., 2022] and CLIP [Radford et al., 2021]. We use ALBEF as the backbone by default in the following experiments, which is pre-trained on 14M image-text pairs and adopts ITC and ITM tasks for image-text retrieval. The details and experiments on TCL and CLIP are shown in Appendix.

4.3 Comparison with State-of-the-art Methods

We compare the proposed RaSa with the existing text-based person search methods on CUHK-PEDES, ICFG-PEDES and RSTPReid, as shown in Table 1, 2 and 3, respectively. RaSa achieves the highest performance in terms of all metrics, outperforming existing state-of-the-art methods by a large margin. Specifically, compared with the current best-performing method CFine [Yan et al., 2022], RaSa gains a significant R@1 improvement of 6.94%, 4.45% and 15.35% on the three datasets, respectively. The comparison clearly demonstrates the effectiveness of RaSa in text-based person search.

4.4 Ablation Study

We analyze the effectiveness and contribution of each optimization objective in RaSa by conducting a series of ablation experiments on CUHK-PEDES, as shown in Table 4.

Effectiveness of Optimization Objectives

RaSa consists of three optimization objectives. CL provides an explicit alignment before the cross-modal fusion. RA implements the deep fusion by the cross-modal encoder with an alleviation of noise interference. And SA encourages the learned representations to be sensitive to the MLM-based token replacement.

We can see from Table 4, (1) RaSa with a single CL achieves a modest performance of 61.35% and 59.44% in Table 4: Comparison of RaSa with different settings on CUHKPEDES. ITM learns from all positive pairs without a probabilistic inputting. s-ITM learns from only strong positive pairs and discards all weak positive pairs. p-ITM uses a probabilistic inputting of strong and weak positive pairs. f-RTD adopts DistilBERT [Sanh et al., 2019] as a fixed generator to produce the replaced tokens. oRTD uses the online model as the generator, while m-RTD is based on the momentum model.

terms of R@1 and mAP, respectively. On account of the modality gap between the image and text and the fine-grained intra-class variation, CL contributes a coarse alignment with a lack of deep interaction across modalities, which is not enough to handle such a challenging retrieval task. (2) When adding RA(p-ITM + PRD), the performance has a remarkable improvement of 12.85% at R@1 and 8.67% at mAP, effectively demonstrating that deep cross-modal fusion with RA is extraordinarily significant to text-based person search. And (3) with the aid of SA(MLM + m-RTD), RaSa achieves the best performance of 76.51% at R@1 and 69.38% at mAP. SA utilizes the visual information and the contextual token information of the corresponding text to detect whether a token has been replaced or not. In order to handle such a challenging detection task, the learned representations are encouraged to be powerful enough to distinguish the tiny difference between the original token and the replaced one.

Analysis of RA

RA contains p-ITM and PRD, where the former focuses on the consistency between the strong and weak positive pairs, while the latter highlights their difference, serving as a regularization of p-ITM.

The vanilla ITM learns from all positive pairs without the probabilistic inputting. However, there exists too much noise interference from weak positive pairs. Intuitively, we can discard all weak positives to get rid of the noise. s-ITM only uses the strong positive pairs and gains a boost of 2.23% at R@1 compared to the vanilla ITM. Nevertheless, such a straightforward way ignores the weak supervision from the weak positives which is also beneficial to representation learning. To reach a trade-off between the benefits of the weak supervision and the risk of side effects, p-ITM resorts to the probabilistic inputting and retains a small proportion of the weak positives. Compared with the vanilla ITM and s-ITM, p-ITM achieves an intermediate performance. Not surprisingly at all, the more noise there exists, the more it affects the retrieval performance. In order to alleviate the impact of the noise, we further propose PRD to perform an explicit distinction between the strong and weak positives, which serve as a regularization for p-ITM. Significantly, no matter whether adding PRD to the vanilla ITM or p-ITM, PRD can obtain consistent performance improvement, which powerfully demonstrates its effectiveness.

Analysis of SA

SA includes MLM and m-RTD. MLM not only plays the role of generating the text with word replacement but also performs a token-level optimization. m-RTD detects the replaced tokens by virtue of the visual information and the contextual token information.

Based on CL and RA, adding a single MLM without the replacement detection task brings a slight boost of 0.61% at R@1. Furthermore, we introduce the detection task and use the momentummodel as the generator to produce the replaced tokens. In order to adequately investigate the effectiveness of the generator, we compare three different variants. (1) Following DiffCSE [Chuang et al., 2022], we use DistilBERT [Sanh et al., 2019] as a fixed generator for the word replacement, which is denoted as f -RTD. From Table 4, RaSa with f-RTD gains a modest performance of 75.13% at R@1. We argue that the generated tokens from a fixed generator can be easily detected as the training advances and thus provides a limited effect on learning representation. (2) oRTD adopts the online model as the generator. RaSa with o-RTD achieves a better performance of 75.99% at R@1. Compared with f-RTD, o-RTD resorts to a dynamic generator which is optimized constantly during the whole training process and can produce more confusing tokens with the proceeding of the model's training, effectively increasing the difficulty of replaced tokens detection and facilitating representation learning. And (3) m-RTD adopts the momentum model as the generator and reaches the best performance of 76.51% at R@1. The momentum model is a slow-moving of the online model and can obtain more stable representations. As the training goes ahead, the momentum model iteratively bootstraps MLM to generate more challenging tokens for detection, which encourages the learned representations to be powerful enough to distinguish the tiny difference and substantially improve results.

Hyper-parameters

In Section 3.2, we use the inputting probability pw to retain a small proportion of weak positive pairs to alleviate the noise, the masking ratio pm to randomly mask tokens to perform the replaced token detection, and the loss weights λ1, λ2, λ3 to make a trade-off. We show how these hyper-parameters impact the performance of RaSa in Figure 4. (1) The best result is achieved at pw = 0.1. The inputting probability pw in RA is introduced to seek a balance between the useful information and the noise from weak positives. A larger pw may introduce too much noise, while a smaller pw hinders the model frommaking full use of the useful information. (2) RaSa performs best at pm = 0.3. A larger pm brings more perturbations to the text, making the detection task too difficult to be carried out. In contrast, when pm goes smaller, SA will contribute less to representation learning. And (3) for the loss weights λ1, λ2 and λ3, they present an overall trend of first increasing and then decreasing. Empirical results show that RaSa performs best when they are set as 0.5.

4.5 Extended Experiments and Visualization

To go a step further and validate the effectiveness of RaSa, we perform extended experiments on two coarse-grained imagetext retrieval datasets (Flickr30K [Plummer et al., 2015] and COCO [Lin et al., 2014]), as well as two fine-grained datasets (CUB [Reed et al., 2016] and Flowers [Reed et al., 2016]). The experimental results are shown in Appendix. Besides, we conduct a series of domain generalization experiments following LGUR [Shao et al., 2022] in Appendix to verify the generalization ability of RaSa. These results clearly demonstrate the effectiveness and the generalization ability of RaSa.

For a qualitative analysis, we also present the retrieval visualization in Appendix, vividly showing the excellent retrieval ability of RaSa.

5 Conclusion

In this paper, we propose a Relation and Sensitivity aware representation learning method (RaSa) for text-based person search, which contains two novel tasks, RA and SA, to learn powerful multi-modal representations. Given that the noise from the weak positive pairs tends to result in overfitting learning, the proposed RA utilizes an explicit detection between strong and weak positive pairs to highlight the difference, serving as a regularization of p-ITM that focuses on their consistency. Beyond learning transformationinsensitive representations, SA encourages the sensitivity to MLM-based token replacement. Extensive experiments on multiple benchmarks demonstrate the effectiveness of RaSa.

Acknowledgments

This work is supported by the National Science Foundation of China under Grant NSFC 62002252, and is also partially supported by the National Science Foundation of China under Grant NSFC 62106165.