Self-Guided Hard Negative Generation for Unsupervised Person Re-Identification

1 Introduction

Person re-identification (re-ID) aims to retrieve images of a specific person from galleries captured by different cameras. With increasing numbers of large-scale person re-ID datasets, supervised methods [Sun et al., 2018; Wang et al., 2018] have achieved great success. Nevertheless, annotating person reID datasets across cameras is difficult and expensive. Therefore, unsupervised person re-ID methods have attracted much attention in recent years.

Without manually labeled annotations, most recent unsupervised person re-ID methods [Zeng et al., 2020; Ge et al.,

∗Corresponding author

2020; Chen et al., 2021b] employ clustering algorithms to generate pseudo labels as supervision signals. However, they all suffer from the label noise caused by clustering procedure. Figure 1 illustrates such label noise, from which we can see that clustering algorithms, e.g. DBSCAN [Ester et al., 1996], are very likely to aggregate different identities with similar appearances, i.e. the hard negative samples for each other. Consequently, for several identities, the truly useful hard negative samples are reduced significantly. The key target of training a re-ID model is to distinguish different identities, where easy negative samples satisfy various of losses easily, making them contribute little to the training procedure. By contrast, the hard negatives are more crucial. The decrease of hard negative samples will inevitably degrade the discrimination ability of a re-ID model.

Several General Adversarial Network (GAN) based methods [Zhong et al., 2018; Zhai et al., 2020; Chen et al., 2021b] attempt to enhance the intra-cluster diversity either by style transfer across cameras or by view generation for a specific person. These methods can be viewed as special data augmentations by which more positive samples are generated for anchor images. Although the lack of positives are alleviated in these methods, the hard negative samples are still lacking.

In addition, most re-ID methods adopt hard sample mining approaches [Hermans et al., 2017; Yu et al., 2018] to accelerate metric learning during training. While hard sample mining selects samples from the existing data which may not be challenging enough. In Figure 2, it is clear that the similarity between anchor images and selected hard negatives continually decreases as the training progresses. Correspondingly, these hard negatives easily meet re-ID constraints, making loss decline rapidly. These observations indicate that the selected hard negatives may not constantly provide effective supervisions to improve the re-ID model, especially under the unsupervised setting.

To alleviate the aforementioned problems, we design a self-guided hard negative generation method for unsupervised person re-ID. Given a certain image as the anchor, to enrich its hard negatives, we propose to constantly generate hard samples rather than selecting from the existing data. To fulfill this goal, a joint framework is developed which incorporates a hard negative generation network (HNGN) and a re-ID network. These two networks are trained alternately to promote each other. Specifically, we design a novel adversarial scheme using contrastive learning, where the re-ID network guides HNGN to generate hard negatives by pulling anchor images and synthetics close in the re-ID embedding space, but conversely push them away during the training stage of the re-ID network. As for the structure of HNGN, a GAN is employed for implementation which follows an encoder-decoder pipeline. To constrain the synthetics to have proper hardness, i.e. the generated samples should be similar to both the anchor and the negative images, a part-level feature fusion strategy is designed to incorporate information from both of them. After then, the decoder generates hard negatives by the fused feature. As can be seen from Figure 2, the loss and anchor-negative similarity of the proposed hard negative generation method are maintained at a higher level, which can continually improve the discrimination ability of the re-ID model.

In this paper, we mainly focus on the fully unsupervised manner. There is thus no extra data or techniques, e.g. ID labels, attributes or pose estimation, for either hard negative generation or person re-ID training. During inference, the HNGN is discarded and the performance of the re-ID model can be improved without introducing any extra parameters.

The contributions of this paper can be summarized as follows. (1) We propose a self-guided hard negative generation concept for unsupervised person re-ID. To our knowledge, this is the first work to study hard negative generation in the unsupervised re-ID field. (2) We develop a joint framework where a hard negative generation network and a re-ID network can improve each other in an adversarial manner. (3) We propose a part-level feature fusion strategy to generate better hard negative samples.

2 RelatedWork

2.1 Unsupervised Person Re-ID

Unsupervised person re-ID methods can be mainly divided into two categories. The first category is fully unsupervised which does not use any ID labels. [Lin et al., 2020] propose a SoftSim method to discard clustering procedure and use similarity-based soft labels to train the re-ID network. SpCL [Ge et al., 2020] utilizes self-paced learning to progressively select reliable samples into clusters. Besides, [Chen et al., 2021b] use 3D meshes as auxiliary information to generate more views of anchor images, which enhances the diversity of positives. [Zhang et al., 2022] generate implicit samples to provide more complementary information to alleviate the clustering noise. However, they all ignore the lack of hard negative samples in the unsupervised re-ID task.

Another category is unsupervised domain adaptation (UDA) methods which aim to transfer knowledge from labeled source data to unlabeled target data. [Zhang et al., 2019] provide a self-training with progressive augmentation strategy. [Bai et al., 2021] introduce multiple source domains for re-ID learning. These UDA methods also do not lay emphasis on the hard-negative problem.

2.2 Hard Sample Mining and Generation

Hard sample mining methods have been studied for many years, which aim to boost machine learning performance by selecting and training hard samples. Given an anchor image, TriHard [Hermans et al., 2017] selects hardest positive sample and negative sample within a batch to constitute a triplet for metric learning. HAPS [Yu et al., 2018] develops a soft hard-mining scheme which will assign greater weights to harder samples adaptively. All these methods rely on the hardness of the existing data. HDML [Zheng et al., 2019] presents a hardness-aware framework to generate hard samples in embedding space. Different from our work, it is designed for supervised methods and trained in a nonadversarial manner.

2.3 Re-IDRelated GANs

In re-ID field, many methods employ GANs for domain adaptation or data augmentation. SPGAN [Deng et al., 2018] proposes a similarity-preserved GAN for style transfer between two domains. PTGAN [Wei et al., 2018] uses semantic segmentation in GAN to retain the consistency of object regions during style transfer. JGCL [Chen et al., 2021b] tries to disentangle appearance and structure of image, and reconstructs a synthetic one by fusing different components. None of them attempts to generate hard negative samples for re-ID.

3 Method

3.1 Baseline Revisit andMethod Overview

Due to the lack of annotations, most unsupervised person reID methods adopt clustering algorithms to produce pseudo labels as supervisions before each epoch. The re-ID model is optimized iteratively with such a self-training scheme.

The proposed method also follows this basic pipeline as illustrated in Figure 3. While such a pipeline is difficult to distinguish different identities with similar appearances, thus causing the lack of hard negative samples. To alleviate this problem, we propose a re-ID guided generator (HNGN) to constantly synthesize hard negatives as a compensation or even enhancement. The re-ID model is then enforced to push hard negatives away from anchor images by contrastive learning. The generator and re-ID model can be optimized jointly in an adversarial manner to promote each other.

3.2 Hard Negative Generation Network

Let X = {xi}Ni=1 denote a batch of training images, where the superscript i indicates the index and N is number of images in a batch. In Figure 3, the notation xanc means the anchor image, xneg denotes a randomly selected negative sample, fanc and fneg are latent embeddings of xanc and xneg. xaug is the augmented view of the anchor. ˜ xanc and xgen denote the reconstructed anchor and the generated hard negative, respectively. Y = {yi}Ni=1 , yanc, yneg, yaug, ˜ yanc and ygen indicate corresponding pseudo labels. In this paper, we usually use subscripts to indicate a type or a specific image.

The proposed hard negative generation network (generator) follows the encoder-decoder structure, where the encoder GE maps an input image into a latent embedding and the decoder GD is responsible for generating a synthetic image. We first use a reconstruction loss to constrain the generator can extract useful information and reconstruct the original image.

where ˜ xianc denotes the reconstructed image.

Then we consider that a good hard negative sample should meets two requirements. (1) It has similar appearance to the anchor. (2) It also has some negative's characters to control the hardness to be appropriate. Therefore, we attempt to feed the fused feature h(fanc,fneg) to the decoder GD to generate desired hard negatives. h(·, ·) denotes a feature fusion strategy, e.g. concatenation or linear combination.

Specifically, in this paper, we propose a part-level feature fusion strategy for generating hard negatives. Given fneg ∈ RC×H×W , as illustrated in Figure 4, we extract part features via a sliding window with size (1/2H, W) and stride d (d = 2 in our experiments). fanc and all part features of fneg are squeezed into feature vectors ganc and {g jneg }Mj=1 by global average pooling, where M indicates the number of parts. Inspired by [Jetley et al., 2018], we use cosine similarity sim(ganc,g jneg ) between a global feature vector and a part feature vector to measure which part of the negative is most similar to the anchor. Let [h1 : h2, 0 : W] denotes the selected part, where h1 and h2 are the start and end vertical coordinates and W is the width of the feature map. The new feature is obtained by linearly combining part features.

where fnew denotes the fused feature, λ ∈ [0, 1] is a weight parameter. Other parts are fused by weights (1 - λ) and λ conversely. We illustrate the fusion of the part [0 : h1, 0 : W] as an example in Eq.(3), and the remaining part [h2 : H, 0 : W] shares the same fusion format.

Intuitively, this strategy tries to generate hard negatives by remaining more anchor's characters on the similar parts and incorporating more negative information from other parts. We also verify the effectiveness of this strategy. See experiments for more details. For simplicity, we do not employ extra techniques, e.g. pose estimation, for strict feature alignment, yet the proposed strategy is also effective.

Subsequently, to enforce the synthetic image to be similar to the anchor, a hard-negative-generation loss Lhng is developed. The contrastive learning paradigm is employed to pull anchor image and synthetic image close in the feature space.

where τ denotes a temperature parameter. P i and N i are the positive set and negative set for the image xianc , where p and q indicate the image p in P i and q in N i , respectively. v denotes a feature vector extracted by the re-ID network E, e.g. v i = E(xianc ), vp = G(p). When each image acts as an anchor, there will be an augmented image and a generated image correspondingly. Thus, we get an augmented batch Xaug and a generated batch Xgen corresponding to the original batch X. Then, P i and N i can be defined as follows.

where Xgen - P i means the difference set of Xgen and P i . Note that, we treat the synthetic image of xigen as positive, and treat the whole batch X and Xaug as negative set in Lhng. This can be viewed as a re-ID guided strong constraint to make the synthetic image similar to the anchor.

Like many GAN-based methods, we employ a discriminator D to distinguish between real images and the synthetics, and use an adversarial loss Ladv to constrain the distribution of synthetics to be close to the real data as follows:

where D(.) measures the probability that an input image belongs to the real data.

3.3 Re-IDNetwork

After generating hard negative samples, the re-ID model is enforced to identify them with a contrastive loss.

where P i con and N i con are the positive set and negative set of image xianc . They are defined as follows.

where + denotes the merge operation between two sets. Xpa indicates a batch of synthetic image parts selected in Eq. 2. The contrastive learning between anchors and generated hard negatives (including negative parts) will constantly encourage the re-ID model to learn more discriminative features.

3.4 Optimization of theWhole Framework

The model is jointly trained with the following overall loss L:

In addition, the proposed method is Plug-and-Play. It can be applied to most unsupervised person re-ID works, and the original losses can also be retained.

Although this is a joint framework, we find that it is difficult to convergence when trained from scratch. This is because the two networks, i.e. the generator and the re-ID model, may confuse each other when they do not have any capability. Thus, we use a warm-up stage to improve their performance individually before joint training. During warmup, the re-IDmodel is trained under the standard setting, and the generator is trained only using Lrec to make it have the basic reconstruction ability.

4 Experiment

4.1 Datasets and Evaluation Protocols

We conduct experiments on three widely-used person re-ID datasets, i.e. Market-1501 [Zheng et al., 2015], DukeMTMCReID [Zheng et al., 2017] and MSMT17 [Wei et al., 2018] to validate our method. Following most re-ID methods, we use Cumulative Matching Characteristic (CMC) top-k accuracy and mean average precision (mAP) as evaluation metrics.

Market-1501 consists of 32,668 images of 1501 persons captured by 6 cameras, where 12,936 images of 751 persons constitute the training set. The remaining images are for test.

DukeMTMC-ReID contains 36,411 images of 1,404 persons from 8 cameras in total. Each person is captured by at least 2 cameras. The training set is comprised of 16,522 images of 702 persons. Other images constitute the test set.

MSMT17 is a challenging dataset containing 126,441 images of 4,101 persons. These images are captured by 15 cameras including 32,621 training images of 1,041 persons and 93,820 test images of 3,060 persons.

4.2 Implementation Details

We implement our method mainly based on the open-source codes released by [Ge et al., 2020] and [Chen et al., 2021a].

ResNet50 is adopted as the backbone of the re-ID network, which is initialized by ImageNet pre-trained parameters. The original fully connected layer is removed and a batch normalization (BN) layer is stacked onto the backbone to obtain normalized features. The encoder of our generator stacks 3 conv-norm-relu layers and 3 residual blocks sequentially. The detailed structure of a residual block is the same as CycleGAN [Zhu et al., 2017]. The decoder stacks 3 residual blocks, 2 deconv-norm-relu layers and 1 convolution layer sequentially. D in this paper is a PatchGAN [Isola et al., 2017] discriminator with 3 conv-norm-leakyrelu layers.

During training, images are resized to 256 × 128. Random flip, random erasing and image padding are adopted as data augmentations. Following most re-ID methods, we use identity-based sampling strategy in our training, which first randomly selects 16 identities and further randomly selects 4 instances for each identity to constitute a mini-batch of size 64. We use Adam optimizer to train our generator and discriminator. The optimizers for re-ID model follow the original settings of [Ge et al., 2020] and [Chen et al., 2021a]. The learning rate is initialized as 1×10-4 for Adam, and the weight decay is set as 5×10-4 . As aforementioned, the proposed method has two training stages, i.e. warm-up stage and joint fine-tune stage. For the former, we train the re-IDmodel following the settings of a basic unsupervised re-ID method. Besides, we train the generator with only reconstruction loss for 50 epochs, where the learning rate is multiplied by 0.1 after 20 epochs. For the joint fine-tune stage, the whole framework is fine-tuned for 20 epochs, where the learning rate is multiplied by 0.1 after 10 epochs. The extra training procedure can be completed in a few hours by modern GPUs, e.g. NVIDIA P40. The temperature τ in Eq. 4 and Eq. 7 is set to 0.05. The widely-used DBSCAN is employed as the clustering algorithm in our method.

4.3 Comparison with state-of-the-arts

To verify the effect of our method, we first integrate our method into a strong baseline provided by [Ge et al., 2020] and compare with many recent works. The results are shown in Table 1. We can see that when equipped with our method, the performance of baseline is improved remarkably (DukeMTMC-ReID mAP +10.7%, top-1 +7.2%; Market-1501 mAP +5.4%, top-1 +1.4%; MSMT17 mAP +7.6%, top-1 +11.7%). It has already achieved comparable results to many recent methods. When applying our method to more powerful approaches [Ge et al., 2020] and [Chen et al., 2021a], we get better results, and outperform all the comparison unsupervised person re-ID works. This indicates the effectiveness and robustness of our method.

4.4 Ablation Studies

To further validate several components proposed in this paper, extensive ablation experiments are conducted.

The Effectiveness of Hard Negative Generation Strategy. We propose to fuse latent embeddings of the anchor and the negative image. The decoder then takes the fused feature as input and outputs a hard negative sample. To validate the effectiveness of this pipeline, we compare it with several feasible generation strategies. The results are shown in Table 2. Mixup means linearly fusing the re-ID feature vectors of the anchor and a negative sample, which is then employed as a new negative sample for unsupervised contrastive learning. Without constraint on the fused feature, the performance of mixup strategy declines drastically. Random noise denotes that we add Gaussian random noise to the latent embedding of an anchor and feed it to the decoder to generate hard negative samples. Under our joint framework, random noise strategy has already outperforms the baseline by a large margin. We also explore the concatenation feature fusion strategy where the latent embeddings of two images are concatenated along the channel axis. This strategy achieves a better result than random noise. The global-fusion and part-fusion strategies can control how much information of anchors and negative references are used, thus generating better hard negatives and further improving the re-ID performance.

The Necessity of the Warm-up Stage. Although the proposed method can be trained jointly, the training procedure is difficult to convergence when initializing the re-ID model and generator from scratch. Under this situation, the generator cannot synthesize samples with high quality, and the re-ID model cannot benefit from the hard negative generation. To address this problem, we use a warm-up stage to make the reID model and generator have a better initialization. Ablation study experiments are carried out on DukeMTMC-ReID and Market-1501 datasets to verify the effectiveness of the warmup stage. Results are listed in Table 3. We can see that if both two networks do not have a warm-up stage, only an inferior result can be achieved. In this case, the generator synthesize some low-quality samples, which may harm the performance of a re-ID model. We gain the best result when two networks all go through the warm-up stage, where they can promote each other based on a better initialization state.

5 Conclusion

In this paper, we propose a self-guided hard negative generation method for unsupervised person re-ID to alleviate the lack of hard negative samples. The proposed method does not rely on any extra data or techniques. We further develop a joint framework which consists of a generator and a re-ID model. These two networks are trained in an adversarial manner to promote each other. The re-ID model guides the generator to synthesize hard negatives, and the hard negatives enforce the re-ID model to learn more discriminative features. Moreover, we propose a part-level feature fusion strategy to make a hard negative similar to both the anchor image and the negative reference. Extensive experiments demonstrate the effectiveness and robustness of our method.

Acknowledgements

This work was supported in part by the Major Project for New Generation of AI (No.2018AAA0100400), the National Natural Science Foundation of China (No. 61836014, No. U21B2042, No. 62072457, No. 62006231).