Fast One-Stage Unsupervised Domain Adaptive Person Search

1 Introduction

The objective of the pedestrian search task is to identify and locate specific individuals within panoramic images. Initially, all pedestrians present in the image are detected, and subsequently, they are compared to the target pedestrian image for recognition. In the context of the intelligent security field, it becomes increasingly difficult and costly to complete person search tasks in a supervised manner due to the massive volume of videos and pictures generated by numerous cameras. Therefore, unsupervised domain adaptive methods should receive more attention.

∗Corresponding author

The main challenges currently in unsupervised pedestrian search are the lack of bounding box annotations and the lack of pedestrian identity label annotations. Since it is very difficult to directly train unsupervised detection and re-identification, a series of weakly supervised research works[Yan et al., 2022; Han et al., 2021; Jia et al., 2023] have been proposed to address the challenge of lacking identity annotations, which have employed clustering techniques to assign pseudo labels. As shown in Fig.1(a), clustering approaches typically require multiple iterations of calculating sample similarity, which can be time-consuming and resource-intensive. Moreover, these methods still demand significant time and effort to annotate ground-truth bounding boxes, and their model generalization capability is limited, rendering them unsuitable for real-world scenarios.

Therefore, unsupervised person search was emerged as the times require. Considering the difficulty and impracticality of directly training unsupervised detection and re-identification, current research methods rely on powerful pre-trained models. Li et al.[4] introduced a domain adaptive method to address the challenge of unsupervised pedestrian search by training on a labeled dataset from the source domain. Specifically, Li et al. designed a domain alignment module to alleviate the difference between unsupervised detection and reidentification subtasks, and then proposed a dynamic clustering strategy to generate pseudo labels on the target domain. We can observe that the clustering method, which is widely employed in both weakly supervised and unsupervised person re-identification tasks, offers certain advantages. It is capable of withstanding noisy labels and generating dependable labels, thus effectively training a model with robust generalization capabilities. However, the computational resources required for this method are quite demanding. The clustering method involves calculating pairwise similarities and multiple iterations to optimize results, which can result in an impractical computational burden. Consequently, these methods face challenges when applied to smart security scenarios involving a massive amount of video footage.

In order to solve this problem, this paper proposes a fast and efficient unsupervised domain adaptation framework for the person search task, which can learn a model with strong generalization ability at a lower cost. As illustrated in Fig.1(b), different from the clustering methods of existing models, FOUS gives up generating relatively accurate labels through complex clustering algorithms, and instead tries to learn potential correlations from soft labels with greater noise and fewer resources. First, in order to quickly assign soft labels, we use the prototype correlation labeling method to calculate the similarity between partial samples only once. To reduce computational resources, these prototypes are used as annotators to soft-label unlabeled samples based on nearest neighbor correlations. Secondly, considering the importance of mitigating the impact of generated noisy labels, we follow the domain alignment module in DAPS and introduce a new attention mechanism, which can amplify global-dimensional interaction features while reducing information dispersion. Effectively reduce the number of noisy labels generated by low-quality candidate frames in unsupervised detection. And FOUS also proposes an adaptive selection strategy to perform label-flexible training of the network to gradually refine the rough labels.

Our contributions can be summarized as follows:

1. We introduce a fast unsupervised domain-adaptive pedestrian search framework based on a prototype correlation labeling method instead of the prevalent clustering algorithm. Compared with multiple iterations of the clustering algorithm, FOUS only calculates the similarity between some samples once and directly assigns soft label groups.
2. In order to reduce the noise in the soft label group, we first introduce a new attention mechanism to alleviate the differences between unsupervised detection and re-identification tasks, and also propose an adaptive selection strategy to gradually refine the coarse labels to find reliable and similar target images.
3. Without any auxiliary labels in the target domain, our method achieves good performance on both pedestrian search benchmark datasets, which not only reduces the amount of calculation and improves the inference speed but also has good generalization ability.

2 Method

2.1 Framework Overview

Fig.2 highlights the design architecture of the proposed FOUS framework. Given the input images from both the source and the target domain, we utilize a pre-trained model on the source domain annotation dataset to produce the source-prototype vector, and the image-level feature maps are extracted with a backbone network. Next, these features are fed into the Region Proposal Network (RPN) to generate proposals, which are passed through the attention module to create instance-level feature maps. To bridge the domain gaps and ensure effective detection and ReID (ReIdentification) tasks, we introduce an attention-based domain alignment module (ADAM). The ADAM is designed to align both image-level and instance-level features across different domains. It helps harmonize the representations and close the gap between domains.

To tackle the issue of computational cost in unsupervised reID tasks, a new prototype-guided labeling method is proposed to assign and optimize soft labels instead of computationally complex clustering methods, which frees up the process of label generation, including prototypes-guided labeling phase and label-flexible training phase. We first annotate the unlabeled samples with coarse labels in a simple way, which selects the prototype vectors generated above and softly annotates unlabeled samples via the relevance between these prototypes and samples. Additionally, FOUS trains its models using a combination of cluster-level contrastive loss and instance-level loss, which ensures that the models adhere to both the constraints imposed by the two types of losses. It aims to refine the coarse labels by incorporating the proposed adaptive selection strategy. Above all, except for the initial iteration, the labeled samples from the source domain are not utilized in our FOUS.

2.2 Attention-based Domain Alignment Module

Attention Mechanism. To mitigate the negative effects of low-quality proposals that arise from unsupervised detection, a novel attention module is designed, which can amplify global-dimensional interactive features while reducing information dispersion. Fig.2 emphasizes that the attention module primarily consists of information aggregation and information interaction.

Input feature x ∈ RH×W×C is initially processed into feature xc ∈ RH×W×C and feature xs ∈ RH×W×C through channel-level information aggregation and spatial-level information aggregation respectively. And these two features are subsequently cross-connected in the channel dimension to yield features xcs ∈ RH×W×2C . The information aggregation is formulated as:

where fclia, fslia and Concat represent channel-level information aggregation, spatial-level information aggregation and cross-concatenation, respectively. Channel-level information aggregation can capture channel dependency information, channel structure information, and spatial global information through cross-channel global covariance pooling and channel-level global average pooling. Spatial-level information aggregation can capture spatial dependency information, spatial structure information, and channel global information through cross-spatial global covariance pooling and spatiallevel global average pooling. Cross-concatenation plays an important role in organizing various pieces of information, enabling effective interaction and integration of subsequent information.

Fig.3 illustrates the specific structure of channel-level information aggregation and spatial-level information aggregation in ADAM, which can simultaneously achieve the perception of multi-dimensional dependency information, multidimensional global information and multi-dimensional structural information in information aggregation. The channellevel information aggregation is formulated as follows:

where CcGCP, CwGAP, MM and Up represent crosschannel global covariance pooling, channel-wise global average pooling, matrix multiplication and upsampling operations, respectively. Moreover, the spatial-level information aggregation is formulated as follows:

where SwGAP, CsGCP and ∆ represent spatial-wise global average pooling, cross-spatial global covariance pooling and reshape operations, respectively.

Similarly, feature xcs ∈ RH×W×2C is separated into feature xc′ s ∈ RH×W×C and feature xcs′ ∈ RH×W×C through channel-driven information interaction and spatial-driven information interaction respectively. And these two features are subsequently adaptively fused into features xc′ s′ ∈ RH×W×C by allocating learnable parameters α ∈ R1×1×C and β ∈ R1×1×C . The information interaction is formulated as:

where fcdii and fsdii represent channel-driven information interaction and space-driven information interaction, respectively. Channel-driven interaction focuses on perceiving diversity in channels by assigning unique parameters along the channel dimension while sharing the same parameters in the spatial dimension. On the other hand, spatial-driven interaction aims to perceive diversity in space by assigning distinct parameters along the spatial dimension while sharing the same parameters in the channel dimension.

Fig.4 shows the specific structure of channel-level information interaction and spatial-level information interaction in ADAM, which concurrently achieves the perception of multidimensional diverse information during information interaction. ADAM stimulates intrinsic information potential through a more comprehensive proactive interaction. The channel-level information interaction is formulated as follows:

where GConv represents the combination of group convolution, batch normalization, and the ReLU function. The information interaction between different groups is undisturbed. Channel-driven information interaction, achieved by assigning different parameters in the channel dimension and sharing the same information parameters in the spatial dimension, not only facilitates the interaction and fusion of multiple information in channel-level and spatial-level information collection but also further perceives channel diversity information.

The spatial-level information interaction is formulated as follows:

where Sum represents the row-wise summation. Reshaping and splitting operations are used to adjust the feature shape for subsequent specific information interactions. The spatialdriven information interaction, achieved by assigning different parameters in the spatial dimension and sharing the same parameters in the channel dimension, not only facilitates the interaction and fusion of multiple information from channellevel and spatial-level information collection but also further perceives spatial diversity information.

Finally, feature xc′ s′ ∈ RH×W×C is activated into the attention map by the sigmoid function. The attention map is employed to modulate input sub-feature x ∈ RH×W×C , resulting in the output sub-feature x′ ∈ RH×W×C :

The sigmoid function distinguishes the importance by activating eigenvalues within the range of 0 to 1. Importantly, the input sub-features are transformed into output sub-features across all branches. This multi-branch structure is advantageous in activating different attention mechanisms, allowing for the selective perception of valuable feature information on each branch in a targeted manner.

Task-sensitive Domain Alignment. As studied by [Chen et al., 2018b; You et al., 2019; Xu et al., 2020; Wang et al., 2024], reducing domain disparities is advantageous for both subtasks within person search, and can enhance the model's capacity to learn domain-invariant representations. Given the swift advancement of domain-adaptive detectors[Khodabandeh et al., 2019; Wang, 2021; Vibashan et al., 2023; Zhang et al., 2023], FOUS proposes a task-sensitive domain alignment module. As shown in Fig.2, ADAM uses a patchbased domain classifier to predict the domain from which the input features come, and uses a minmax formula to mislead the domain classifier and encourage domain-invariant representation learning. Assuming we possess N training images {I1, ..., IN} along with their respective domain labels, where 0 and 1 represent the source domain and target domain respectively. Then denote the domain classifier as D, and further denote the domain prediction result of the input feature Ii as pi. We incorporate a cross-entropy loss for domain alignment through an adversarial training approach:

Based on the characteristics of the upstream and downstream tasks, we suggest achieving balance in the tasksensitive alignment module by equitably adjusting the alignment weights of instance-level features in both sub-tasks. The instance-level loss can be formulated as:

where j ∈ {1, ..., K1} and k ∈ {1, ..., K2}. The source domain comprises Ns images, while the target domain includes Nt images. The balancing factor λ is set as:

where σ is Sigmoid function employed for domain normalization. What's more, we introduce an L2-norm regularizer to maintain consistency between image-level and instance-level classifiers.

2.3 Unlabeled Target Domain Training

FOUS employs a prototype-relevance labeling method as a replacement for complex clustering algorithms, leading to a significant reduction in computational costs. In FOUS, the prototype vector needs to be initially selected or calculated, followed by the computation of the correlation between the prototype vector and all the samples. Then assign the label of the nearest prototype vector to the sample as a pseudo label. Therefore, the selection of the prototype vector is a critical aspect. And FOUS design two types of prototypes, namely the source-prototype vector and the random-prototype vector. Initially, the source-prototype vector is computed using welllabeled data from the source domain. Subsequently, to enhance FOUS's adaptability in the target domain, features are randomly selected from the target domain to create a randomprototype vector.

Prototype vector initialization. We utilize the network Es(·), pre-trained on the source domain, to extract features from the source domain samples. Subsequently, based on the ground-truth labels, the source-prototype vector can be computed as:

where F s k represents a set of features with the same labels k. fi is the feature of each instance. Nk is the number of samples with the label k. The source-prototype vector P s is composed of {P s 1 , ..., P sK }, k ∈ [1,K]. Particularly, K means the number of identity categories in the source domain. However, models trained solely on pseudo-labels generated from source-prototype vectors tend to be sensitive to variations in the target domain and are susceptible to introducing a considerable amount of noise. Hence, FOUS sets up random prototype vectors in the target domain to enhance diversity. Similar to the source-prototype, Es(·) is employed to extract features from target domain samples. Subsequently, Nt vectors are randomly selected to form the random-prototype P t = {P t 1 , ..., P tNt }.

Labeling. Once the source and randomprototype vector are calculated, the labels for other sample instances in the target domain are determined by the k-Nearest Neighbor algorithm,

where dist(a, b) means the Euclidean distance between a and b. fi is the characteristic of the i-th instance. Eq.12 signifies assigning the instance with the label of the vector in P s that is closest to fi. All the labeled images collectively constitute a new dataset Ds . Similar to Ds , according to the randomprototype P t , we can also derive a group of labels,

The assignment of pseudo labels through prototypes results in the creation of two datasets, which are Ds = {(xi, ls1 , ..., lsn )}Ni=1 and Dt = {(xi, lt1 , ..., ltn )}Ni=1 , n represents the number of instances in xi. Then, the sourceprototype is updated by the target datasets with pseudo labels,

where F t j means the features belonging to the cluster j. Therefore, FOUS does not use images from the source domain except for the first iteration.

Label-flexible Training. To improve the training of models with flexible labels, FOUS constructs distinct memory structures to store features based on different label groups. Based on the source prototype P s and random prototype P t , the pseudo labels Ls and Lt can be calculated. To mitigate the influence of noise, we designed two cluster-level memory structures to store the features at the training stage. Taking the Ls as an example, the features {c1, ...cK} of each cluster are stored in a memory-based feature dictionary. K means the number of clusters. After clustering with prototypes, the features in the cluster are utilized to initialize the memory dictionary, which could be expressed as,

where Fk represents the feature set of instances labeled as k, comprising all feature vectors in cluster k. Nk is the number of features in Fk. When employing random-guided memory, we are provided with a query instance q and perform feature comparisons with all features within the respective memory{mt1 , ...mtK } structure utilizing the InfoNCE loss,

where mt+ denotes the features that share the same identity as f within the random-guided memory. τ is the proportional coefficient balancing factor. Similar to Eq.16, the loss Lsc of the source-guided memory also could be calculated.

In addition, FOUS employs invariant learning between images to bolster the model's generalization capability. An instance-level memory bank {ml1 , ...mlN } is consequently established to store all features, and soft labels are estimated by evaluating the correlation between images.

Due to the absence of genuine labels, neighborhoods displaying high correlation play a role in diminishing the distance between similar images. Consequently, FOUS introduces an adaptive selection strategy. Taking inspiration from [Ding et al., 2020; Peng et al., 2020; Wang et al., 2022], a similarity threshold is defined. Only when the distance between the adaptively chosen neighbors and the provided image falls below this threshold, these samples are presumed to share the same label as the target image. Therefore, these samples should exhibit closer proximity to the target image. The corresponding loss can be formulated as:

where vj ∈ {0, 1}N . If vij = 1, it means the i-th image is a reliable neighborhood of the j-th image. If vij = 0, the i-th image is abandoned.

Final loss. The final loss is composed of detection loss and re-id loss, which is formulated as,

where Lins is calculated by Eq.9. Ltc and Lsc are calculated by Eq.16. Let and Les are calculated by Eq.17.

3 Experiments

3.1 Datasets and EvaluationMetrics

Datasets. The proposed method is evaluated on two largescale benchmark datasets, CUHK-SYSU[Xiao et al., 2017] and PRW[Zheng et al., 2017]. CUHK-SYSU consists of 12,490 images captured by real surveillance cameras and 5,694 images from movies and TV shows. The training set comprises 11,206 images and 5,532 query persons, while the test set includes 6,978 images and 2,900 query persons. The widely utilized PRW dataset comprises a total of 11,816 video frames captured by 6 cameras and 2,057 queries with 932 identities.

Evaluation metrics. In the dataset configuration, annotations for the source domain dataset are available, while the ground truth boxes and identity information for the target domain dataset are not accessible. We employ widely used metrics such as mean average precision(mAP) and cumulative matching characteristic(CMC) top-1 accuracy for evaluating the re-identification subtask. For the detection task, average precision(AP) and recall rate are adopted as the evaluation metrics.

3.2 Implementation Details

Our experiments are implemented on NVIDIA GeForce GTX 3090Ti GPU with Pytorch[Paszke et al., 2017]. FOUS utilize ResNet50[He et al., 2016] pre-trained on ImageNet-1k[Deng et al., 2009] as the default backbone network. During the training process, the dimensions of input images are adjusted to 1500 × 900, and random horizontal flipping is applied for data augmentation. We employ the Stochastic Gradient Descent(SGD) method to optimize the model, with a mini-batch size set to 4 and a learning rate of 0.003. The momentum and weight decay are set to 0.9 and 5 × 10-4 , respectively. For the k-nearest neighbor algorithm, the number of neighbors is set to 1. When PRW is chosen as the target domain, we pretrain on the source domain CUHK-SYSU for 8 epochs and subsequently engage in joint training for 10 epochs. Similarly, when CUHK-SYSU is chosen as the target domain, we pre-train on the source domain PRW for 2 epochs and subsequently engage in joint training for 14 epochs.

3.3 Comparison with State-of-the-Art Methods

As there is currently limited interest in unsupervised pedestrian search methods, we begin by comparing FOUS with fully supervised methods in Table.1, encompassing both twostage and end-to-end methods. For the first time, we eliminate the need for computationally expensive clustering algorithms in assigning labels for weakly supervised and unsupervised pedestrian search tasks. Surprisingly, FOUS outperforms some supervised methods. The experimental results indicate a substantial gap between FOUS and state-of-the-art supervised methods. We aspire that our approach may serve as a catalyst for subsequent explorations in this field.

We present a comparison of FOUS with state-of-the-art weakly supervised and unsupervised methods in Table.2. Surprisingly, FOUS exhibits superior performance on the PRW dataset compared to all weakly supervised and unsupervised methods. For the CUHK-SYSU dataset, FOUS surpasses the existing unsupervised method DAPS by 1% in mAP and 1.2% in top-1, although it has not yet reached the level of performance achieved by weakly supervised methods. The reason may be as mentioned in DAPS. Since the number of images and identities in PRW is significantly less than in CUHKSYSU, the detection performance of CUHK-SYSU as the target domain is poor, further leading to lower accuracy in the search task. The proposed attention-based domain alignment module enhances the quality of proposals, further bolstering the performance of detection in FOUS. Consequently, FOUS achieves a balance between reducing computational complexity, accelerating processing speed, and maintaining stable accuracy.

3.4 Ablation Study

In this subsection, we perform ablation experiments to validate the significance of each component in FOUS. In Table.3, we compare the combination of baseline methods with different proposed modules and report the results on both CUHK- SYSU and PRW datasets. The baseline model achieves 50.4% mAP and 52.8% Top-1 accuracy on the CUHK-SYSU dataset. FOUS implemented three components and achieved a 28.2% mAP and 28.0% top-1 improvement over the baseline. We find that if only use the prototype-guided labeling method to generate coarse label groups without utilizing the label-flexible network to refine the labels, the accuracy will decrease significantly by 10% on CUHK-SYSU and 7% on PRW. If the attention mechanism is abandoned to enhance proposals, the accuracy drops even further to 14.3% on CUHK-SYSU. Furthermore, neglecting the domain difference between the two subtasks leads to a reduction in accuracy by 6.3% on CUHK. Hence, the three components we proposed effectively enhance the performance and generalization ability of the model.

Furthermore, to verify the effectiveness of the designed attention module, we compared it with state-of-the-art attention methods. As shown in Table.5, our proposed attention module outperforms significantly, achieving an average accuracy 1.8 higher than CBAM, ECAM, SGEM, and GCT methods on CUHK-SYSU. We can confidently infer that global information aggregation and interaction with proposals are beneficial for model learning.

Last but not least, we compare with state-of-the-art weakly supervised and unsupervised methods in terms of network parameter amount and computational effort. Table.4 presents the comparative results concerning computational requirements, parameter count, training time per iteration, and accuracy during the process. As FOUS computes the similarity of source domain samples only once, it reduces computational costs by nearly half compared to DAPS, which utilizes clustering algorithms. This reduction is achieved at a minor cost in terms of parameters. Furthermore, training time is relatively reduced by 25%, and there is a remarkable improvement in accuracy rather than a decline. We have abandoned the complex clustering method and, instead, proposed a prototype correlation labeling algorithm, resulting in a significant reduction in the number of parameters and computational workload.

3.5 Parameter analysis

Setting several parameters in FOUS may affect the performance of the model. Therefore, we evaluate these parameters in our paper, such as the number of random-prototype vectors and the pre-trained epochs during the training stage. The results on CUHK-SYSU and PRW under different settings are shown in Fig.5.

Analysis of the random selected prototype number. As shown in Fig.5(a), we vary the number of random prototype vectors Nt to 400, 600, 800, 1000, and 1200 on CUHKSYSU, respectively. And Nt is set to 100, 200, 300, 400, 500 on PRW, respectively. In detail, maintaining the parameters of batch size and pretraining rounds unchanged, as illustrated in Figure 3, the model performance varies with the different settings of Nt. This underscores the significance of randomly selecting the number of prototypes for generating pseudo-labels. What's more, Fig.5(a) also shows that Nt = 600 and Nt = 300 can achieve the best accuracy on CUHKSYSU and PRW, respectively.

Analysis of the pre-trained epochs. In this part, the paper explores the relations between the pre-trained epochs and the whole model performance. As observed in Fig.5(b), when CUHK-SYSU is embraced as the target domain, the performance peaks at a pre-trained epoch of 10, while with epoch = 4 for PRW. Our analysis of this result is that in smaller source domains, even limited additional target information may contribute to cross-domain generalization. In contrast, for larger source datasets, unreliable target suggestions may have a negative impact on mitigating domain differences.

4 Conclusion

In this paper, we present the FOUS framework designed for the domain adaptive pedestrian search task. FOUS introduces two components, one is a prototype-guided labeling algorithm, which replaces the commonly used complex clustering algorithms in this domain, significantly reducing computational costs at the expense of accuracy. The other is an attention-based domain alignment module, alleviating the impact of low-quality candidate boxes on re-identification tasks in unsupervised detection. Together, these components complement each other, reducing computational overhead while enhancing the model's performance. Judging from the results, experiments on two foundational datasets demonstrate the performance of FOUS and the effectiveness of the introduced modules and achieve state-of-the-art performance.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China Grant 62172136, U21A20470, 72188101, 62002041 and 62201401, Institute of Advanced Medicine and Frontier Technology under Grant 2023 IHM01080, Dalian Science and Technology Innovation Fund 2022JJ12GX019. And we are grateful for technical support from the High Performance Computing Center of Dalian Maritime University.