Color-Sensitive Person Re-Identification

1 Introduction

Person re-identification (Re-ID) aims to automatically match the underlying identities of person images from nonoverlapping camera views [Gong et al., 2014]. As an essential task in video surveillance of distributed multi-cameras, Re-ID is very important for individual-specific long-term behavior analysis. Due to variations of view angles, poses and illuminations in different cameras, it's very challenging to tackle this task.

In the Re-ID community, many models have been proposed, which mainly focus on three parts: hand-crafted descriptor design, metric learning and deep Re-ID models. Hand-crafted person descriptors [Satta, 2013; Zheng et al., 2015a] try to design features that are robust to different view angles, poses and illuminations. Metric learning methods [Farenzena et al., 2010] learn a discriminative space where the similarity of same person images is higher and that of different person images is lower. Deep Re-ID models use deep neural networks to straightly learn robust and discriminative features in an end-to-end manner. For example, [Zheng et al., 2016; Hermans et al., 2017] try to learn a global features by using the identity classification loss, triplet loss or quadruplet loss. [Sun et al., 2018; Li et al., 2018] learn part-level features based on a partition strategy or an attention mechanism. However, most of them fail to take full advantage of intelligible color information, which is one of the most important cues for person Re-ID [Kviatkovsky et al., 2013; Yang et al., 2014].

∗ corresponding author

In this paper, we explicitly and effectively use the color information and propose a novel Color-Sensitive Re-ID model. Firstly, we learn from real and fake images by minimizing identity-based classification and triplet losses. By doing so, we can explore more color information while preserving the identity of a person. Secondly, we further take the advantage of color information by distinguishing images of the same person with different clothing colors. Specifically, we establish triplet images (xa, xp, xn) from the same person. xa and xp own the same clothing color while xn and xp have different clothing colors. Thus, by minimizing the corresponding triplet loss, the learned features can focus on the color difference in regions.

To generate fake images with specified colors, we also propose a novel Color Translation GAN (CTGAN), which includes a translator of generating and reconstructing images, a discriminator of distinguishing images sources and classifying clothing colors as well as a feature extractor of guaranteeing identity consistency among fake images of the same person with the same color. As shown in Figure 1, our CTGAN stably translates images to specified colors meanwhile maintains other contents unchanged.

Our main contributions can be summarized as below:

1. We propose a novel Color-Sensitive Re-ID model to explore the color information fully and explicitly. On one hand, based on the classification and triplet losses of real and fake images, more color information is exploited and the identity of a person is preserved. On the other hand, the learned features are sensitive to color changes by distinguishing images of the same person with different clothing colors.
2. We propose a novel CTGAN model to learn mappings between different clothing colors. It can stably translate person images with expected clothing colors.
3. Experiments on two publically available benchmark datasets: Market-1501 and DukeMTMC-reID demonstrate the effectiveness of our proposed Color-Sensitive Re-ID model and CTGAN. Based on them, our approach achieves new state-of-the-art results.

2 RelatedWorks

2.1 Color Pedestrian Descriptors

In person Re-ID task, most Re-ID researchers explore color information from the person appearance and have got a series of achievements. For example, Kviatkovsky et al. [Kviatkovsky et al., 2013] extract color features in the logchromaticity space. Yang et al [Yang et al., 2014] propose a novel salient color names based color descriptor to describe colors for person re-identification. Those methods prove the effectiveness of color information for person ReID. However, those descriptors are manually designed and hardly comparable with recent high-level semantic features of deep learning. Different from them, our model exploits color information with deep learning model.

2.2 Deep Person Re-IDModels

Deep Re-ID models automatically learn the features via deep neural networks in an end-to-end manner supervised by identity labels. Some deep Re-ID models straightly learn global feature. For example, IDE [Zheng et al., 2016] learns identity-discriminative features by fine-tuning a pre-trained CNN to minimizing the identity classification loss. TriNet [Hermans et al., 2017] straightly learns embeddings in a metric space by minimizing a triplet loss with batch hard. Apart from those global features Re-ID models, several part-level features Re-ID models have been proposed to learn finegrained features via partition strategy or attention mechanism. PCB [Sun et al., 2018] proposes a uniform partition on the conv-layer for learning part-level features and adaptive pooling for accurate region locations. HA-CNN [Li et al., 2018] proposes a Harmonious Attention CNN to learn soft pixel attention and hard regional attention along with simultaneous optimization of the features. Different from those ReID models, which mainly learn abstract high-level features and fail to take full advantage of intelligible color information, our Color-Sensitive Re-ID model explicitly and effectively utilizes color information and achieves state-of-the-art performance.

Recently, some Re-ID models with GAN are also proposed. [Zheng et al., 2017] show that unlabeled samples generated by GAN improve Re-ID baseline. [Zhong et al., 2018] propose to learn camera-invariant descriptors with camerastyle transferred images generated by StarGAN [Choi et al., 2018]. Different from them, ours utilizes the idea of GAN to promote Re-ID by learning both identity-aware and colorsensitive features.

2.3 Image-to-Image Translation

Image-to-image translation is a class of vision and graphics problems where the goal is to learn mappings between(among) two(several) domains. Recently, by using a GAN loss to make the generated images and real images indistinguishable, some GAN based models [Choi et al., 2018] have shown remarkable results in various computer vision tasks including image-to-image translation. Different from them, our CTGAN is specifically designed for clothing color translation, which considers an extra constraint on identity consistency of translated images from the same person and thus generating stable color translated images of a person.

3 Color-Sensitive Re-IDModel

3.1 Learning fromReal Images (Baseline)

The classification loss and the triplet loss have been proven to be efficient in learning identity-discriminative features for Re-ID. Inspired by [Zheng et al., 2016; Hermans et al., 2017], we use a CNN model to learn the feature map M of an image and then pool it to a feature vector V . Given real images Xr , we optimize their feature vectors with the classification loss Lreal cls of a classifier C and the triplet loss Lreal trl of an embedder E.

where p(·) is the predicted probability of the input belonging to the ground-truth. Ltrl(·, ·, ·) is defined in Eq.(2), where xa and xp are a positive pair belonging to the same person, xa and xn are a negative pair belonging to different persons, Dx1,x2 is the cosine distance between x1 and x2 in embedding space of the embedder E, and m is a margin parameter, [x]+ = max(0, x).

3.2 Extension of Baseline to Fake Images

In Eq. (1), features learned from real images do not explicitly take full advantage of color information. To solve it, we generate more fake images with different colors by the CTGAN

(please see Section 4) and add them to train our baseline ReID model under a triplet loss. Then, the newly added triplets are constructed as follows: positive pairs are from real (or fake) images of the same person while the negative sample correspondingly comes from the fake (or real) image of a different person with the anchor. Considering that fake images may have noise, we exclude the triplet using all fake images. Then, our extension of baseline to fake images can be formulated as:

Compared with Eq. (1), Eq. (3) further explores more color information among existing persons. It brings the following advantages: (1) we have extra positive pairs with different clothing colors, thus making the model be adaptive with different colors; (2) we have extra negative pairs and may make it easier to distinguish different person with similar clothing color; (3) the augmented data is generated without extra annotations and it can avoid over-fitting during training.

3.3 Learning from the Same Person with Different Colors

Although we can learn more color information with Lreal+fake in Eq.(3) than Lbasel. in Eq.(1), color information is not fully exploited because it does not consider the case with different clothing colors of the same person. To that end, we propose a novel color-sensitive branch where triplets are constructed with real and fake images of the same person. Specifically, for a person, we choose images with the same clothing color as positive pairs and treat those with different clothing colors as the negative pairs. Then, for each person i, the corresponding loss can be formulated as below:

where Xr i is all real images of the ith person and Xf i denotes all fake images of the ith person. The final loss of our colorsensitive branch can be formulated as below:

Based on Eq. (5), we can take full advantage of color information and make the learned features focus on clothing color changes.

3.4 Overall Objective Function

Finally, the overall objective function of our Color-Sensitive Re-ID model is formulated in Eq.(6). Our whole ColorSensitive Re-ID model can be optimized by scholastic gradient descent in an end-to-end way.

3.5 Discussion

Although both Lreal+fake in Eq.(3) and Lcolor in Eq.(5) use fake images, they are total different. The former constructs triplets with different persons, which contributes to exploring more color information. The latter constructs a triplet using images of the same person, which makes the learned features focus on clothing color changes. The experimental results in Section 5.3 show the effectiveness of both contributions.

4 Color Translation GAN (CTGAN)

To generate fake images with specified clothing colors, we propose a novel Color Translation Generative Adversarial Network (CTGAN). As shown in Figure 3, the CTGAN consists of three components, i.e. a translator T, a discriminator D and a feature extractor F. The module T is to translate a x to a fake image x′ with color c′ , i.e. x′ = T(x, c′ ). The module D is to distinguish images source (real or fake images) Dsrc(·) and classify images color Dcls(·). The module F extracts features to compute the similarity between any two images.

Specifically, the objective function of our CTGAN includes an adversarial loss Ladv, a color classification loss Lcls, a reconstruction loss Lrec, and an identity consistency loss Licl. Ladv is to make fake images as realistic as possible by making fake images indistinguishable from real ones. Lcls is to translate images to a specified color. This is implemented by making Dcls(·) classify a real image x to its ground truth color c, and T translates an image x to x′ that can be classified as a specified color c′ . Lrec guarantees that the fake image x′ preserves the contents of a person by making the x′ reconstruct corresponding real ones x. Thus, Ladv, Lcls and Lrec can be formulated in Eq.(7), Eq.(8) and Eq.(9).

Although the above-mentioned three losses learn multidomain mappings between different clothing colors, it cannot generate stable clothing color translated images of a person. This will be shown in Section 5.4. To solve the problem, we propose a novel identity-consistency loss in Eq.(10).

where x1 and x2 are the fake images of the same person with the same clothing color, Dx1,x2 means the cosine distance between two images in the feature space learned by the module F. Note that the module F is a pre-trained feature learning part for person Re-ID and is fixed when training our CTGAN. With it, we can guarantee that the fake images of a person with the same clothing color stay close in the feature space and thus translating person images with specified colors steadily.

The overall objective function of our proposed CTGAN model can be written in Eq.(11), where λcls, λrec, λicl are trade-off weights of Lcls, Lrec and Licl, respectively. We set λcls = 1, λrec = 10 as in [Choi et al., 2018] and λicl = 1 empirically. Our CTGAN can be learned by alternatively optimizing LD and LT .

5 Experiment

5.1 Datasets

Market-1501 [Zheng et al., 2015b] contains 32,669 annotated images of 1,501 identities from 6 cameras. Each identity is annotated 8 colors of upper-body clothing and 9 colors of lower-body clothing. DukeMTMC-reID [Ristani et al., 2016] includes 16,522 training images of 702 identities. Each identity is annotated 8 colors of upper-body clothing and 7 colors of lower-body clothing. The color attributes of both datasets are annotated by [Lin et al., 2017].

5.2 Implementation Details

For the CNN part, we adopt ResNet-50 [He et al., 2016] pre-trained on ImageNet [Deng et al., 2009] as our CNN backbone, and use the Pool5 layer as our feature map M. For the classification loss, a classifier includes a 256-dim FC layer as the middle layer followed by batch normalization and dropout, and an FC layer with identity number logits as output layer. For the triplet loss, an embedder is an FC layer which maps a part-level feature vector to 256-dimensions embeddings.

Recent works [Li et al., 2018; Sun et al., 2018] show that part-level features own more fine-grained information, which contributes to accurate Re-ID. As in [Sun et al., 2018], we uniformly partition the feature map M into 6 horizontal stripes, pooling them into 6 part-level vectors {Vi}6i=1 , and optimize them with non-shared classifiers and embedders. Considering that some parts of negative pairs may be unchanged after CTGAN, from which we can hardly learn color information, we abandon those parts of negative pairs with cosine distance larger than a constant (we set 0.6 throughout the paper). During the test stage, we use the concatenation of normalized feature vectors {Vi}6i=1 as person features and the cosine distance as similarity.

We implement our approach with Pytorch. The training images are augmented with horizontal flip, random cropping, random erasing [Zhong et al., 2017] and normalization. The batch size of real images is set to 192 (24 persons and 8 images for a person), and that of fake images is set to 112 (4 persons, randomly select 7 fake images for a person). We initialize the learning rates of the CNN part at 0.05 and the other parts (classifiers and embedders) at 0.5. The learning rates are decayed by 0.1 every 4, 000 iterations, and the model is trained for 15, 000 iterations in total. We set margin m = 0.1 empirically.

5.3 Ablation Study

To evaluate each component of our color-sensitive Re-ID model, we conduct two variants trained under different settings. Firstly, we train the model under Eq.(1) as a baseline. Secondly, we evaluate the effectiveness of the learning from both real and fake images under Eq.(3). Thirdly, by comparing with our whole model under Eq.(6) with the first two variants, we can verify the effectiveness of learning by distinguishing the same person with different colors. We mark them as Lbasel., Lreal+fake and Ours(Lall), respectively.

As shown in Table 1, Lbasel. achieves 80.5% and 68.5% mAP scores and 93.1% and 82.3% Rank-1 scores on Market1501 and DukeMTMC-reID, respectively. This demonstrates that the identity-based classification and triplet losses of real images are able to learn identity-discriminative features for person Re-ID. Secondly, Lreal+fake achieves 82.6% and 69.9% mAP scores and 94.5% and 83.9% Rank-1 scores on Market-1501 and DukeMTMC-reID, respectively. Compared with Lbasel., the fake images improve the results by 2.1% and 1.4% mAP scores and 1.4% and 1.6% Rank-1 scores on Market-1501 and DukeMTMC-reID, respectively. This demonstrates that the fake images generated by our CTGAN are meaningful and contribute to learn color-sensitive features. Finally, our whole Color-Sensitive Re-ID model further improves Lreal+fake by 4.5% and 4.0% mAP scores and 2.5% and 2.8% Rank-1 scores on Market-1501 and DukeMTMC-reID, respectively. This significant improvement indicates that color information is critical to improve the discriminative ability and distinguishing the same person with different color is an effective way to make the features more discriminative. In summary, Ours(Lall) significantly improves Lbasel. by 6.6% and 5.4% mAP scores and 3.9% and 4.4% Rank-1 scores on Market-1501 and DukeMTMCreID, respectively. This verifies the effectiveness of our proposed approach by using extra color information.

5.4 CTGAN Evaluation

Fig.1 displays the fake images generated by our Color Translation GAN (CTGAN). As we can see that, our CTGAN stably translates real images of a person by using expected clothing colors and keeps other contents unchanged.

We also compare fake images generated by our CTGAN with and without identity-consistency loss Licl in Eq.(10). We mark the CTGAN without Licl as CTGAN-. As is seen, our CTGAN generates stable images while the CTGAN- can not. For example, Fig.5(a) shows that the color of fake images of a person generated by CTGAN is relatively stable, but those generated by CTGAN- are biased. Fig.5(b) demonstrates that our CTGAN- accurately finds clothing regions and changes the color, but the CTGAN- fails to.

The unstable fake images may import noise to the following Color-Sensitive Re-ID model. We compare the results of our Color-Sensitive Re-ID model with fake images generated by CTGAN and CTGAN- in Fig.4. We can see that, when using CTGAN-, the performance drops a lot. Specifically, there are declines of 3.1% and 2.7% Rank-1 scores and 5.5% and 4.4% mAP scores on Market-1501 and DukeMTMCreID, respectively. This phenomenon demonstrates the effectiveness of our CTGAN from a point of quantitative view.

5.5 Effect of Different Clothing Regions

To analyze the effect of different clothing regions, we also report the experimental results on Market-1501 when using fake images generated by different upper or lower clothing colors. We represent them as Ours(Llower) and Ours(Lupper). As we can see in Table 2, when the fake images with different upper or lower clothing colors are used, better results can be achieved than baseline. In addition, the upper part seems more important than lower one and we can obtain the best results by using both.

5.6 Performance with Different CNN Backbones

As we all know, the architecture of CNN has a large effect on various machine learning tasks [Krizhevsky et al., 2012; Simonyan and Zisserman, 2015; He et al., 2016]. To verify the robustness of our approach with different CNN architectures, we evaluate our approach with three different CNN backbones, including AlexNet [Krizhevsky et al., 2012], VGG16 [Simonyan and Zisserman, 2015] and ResNet-50 [He et al., 2016]. The experimental results on Market-1501 are shown in Table 3. As we can see, Ours(Lall) improves Lbasel. by 8.6%, 6.8% and 6.6% mAP scores on Market-1501 with AlexNet, VGG16 and ResNet-50 respectively. This verifies the robustness of our proposed approach to different CNN backbones.

5.7 Comparison with State-of-the-art

We compare our Color-Sensitive Re-ID model with state-ofthe-art, which can be grouped into three groups, i.e. handcrafted descriptors, deep Re-ID models of global features, deep Re-ID models of local-level features. All deep Re-ID models are based on ResNet-50.

The experimental results on Market-1501 and DukeMTMC-reID are shown in Table 4 and Table 5. Firstly, the hand-crafted descriptors contribute to re-identification but achieve an unsatisfactory performance. This is because hand-crafted descriptors are usually designed manually without identity labels. Secondly, deep Re-ID models with global features significantly outperform the hand-crafted descriptors, which demonstrates the effectiveness of deep learning. Thirdly, deep Re-ID models with part-level features further improve those of global features, which demonstrates that part-level features acquire fine-grained information and can be in favor of accurate re-identification. Finally, Ours outperforms all those methods achieving new state-of-theart. The experimental results verify the superiority of our Color-Sensitive Re-ID model over existing methods.

6 Conclusion

In this paper, we propose a novel approach to exploit colorsensitive features for person Re-ID. Firstly, we propose a novel Color-Sensitive Re-ID model, which learns color information by learning from real and fake images, and distinguishing images of the same person with different clothing colors. Secondly, we design a novel Color Translation GAN (CTGAN) to learn mappings between different colors, where an identity consistency is proposed to improve the stability of generating fake images of a person. Finally, experimental results on two benchmark datasets show the effectiveness.

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grants 61720106012, 61533016 and 61806203, the Strategic Priority Research Program of Chinese Academy of Science under Grant XDBS01000000, the Beijing Natural Science Foundation under Grant L172050.