Weakening the Influence of Clothing: Universal Clothing Attribute Disentanglement for Person Re-Identification

1 Introduction

Person Re-identification (Person Re-ID) aims at matching a specific person across non-overlapping cameras, which has large promising applications such as video surveillance and smart city. Extensive methods assuming a short-term cloth-consistent setting made great progress recently. However, these methods heavily rely on clothing features (texture, color, accessory, etc.) which makes them ineffective in clothchanging scenarios. Fig.1 gives intuitive illustrations that the same person could look different on different clothes and different people could look similar on the same clothes, clothes become a disturbance to retrieval under cloth-changing setting. Therefore, we seek to alleviate the influence of clothing and learn identity-related features (face, shape, haircut, etc.) so that our method can apply to cloth-changing scenarios.

∗Contact Author

To tackle the challenge of clothes change, a few works have shifted their attention to the extraction of cloth-agnostic representations like shape [Qian et al., 2020b], contour sketch [Yang et al., 2021]. However, they have two main drawbacks: (1) Off-the-shelf networks from other tasks (e.g. the Human Keypoints Estimator) are employed both in the training and testing stage, which are of high time and resource cost thus limiting their scalability. (2) They paid too much attention to extracting certain kinds of features while failing to consider all identifiable identity-related features. Other works [Xu et al., 2021; Li et al., 2020b] proposed to use generated adversarial learning to disentangle the identity and clothing features. However, we notice that the same clothes would look different on different people as shown in Fig.1 which means the worn clothing may reveal some kind of information (e.g.

body contour) of an identity. Therefore, the clothing features they extract contain part of the identity feature so they can not fully disentangle clothing features and identity features.

Considering the above problems, we propose a Universal Clothing Attribute Disentanglement (UCAD) network which can effectively weaken the influence of clothing and force the model to learn identity-related features that are unrelated to the worn clothing. Our proposed model includes two branches: the Clothing Attribute Extraction (CAE) branch and the Clothing Attribute Filtering (CAF) branch. First, with the help of the Human Parsing Estimator (HPE), the CAE branch is designed to obtain the clothing feature called the clothing attribute feature which truly contains no identity information (e.g. body contour). Next, the CAF branch distills the knowledge from the CAE branch aiming to decompose and disentangle clothing feature and identity feature to alleviate the influence of the worn clothing. Specifically, a consistent loss is used to distill from the clothing attribute feature to explicitly ensure the clothing feature contains no identity information. Besides, an orthogonal loss is utilized to encourage the clothing and identity features to be decoupled so that the identity feature could contain all identifiable information (shape, face, haircut, body contour etc.) except clothing thus weakening the influence of the worn clothing. The CAE branch is discarded at test stage to improve efficiency, which means the HPE is only used in the training stage.

To further study Person Re-ID in cloth-changing scenarios, large-scale datasets are acquired. Although some clothchanging datasets have been collected, they are under moderate cloth-changing setting [Yang et al., 2021] which means they do not include clothes change over four seasons and the maximum resolution of cameras they use is 1920×1080. However, changing clothes over four seasons is a common reality for human beings. In addition, it has been proven that higher resolution is good for Re-ID [Wang et al., 2020]. Therefore, we contribute a large-scale cloth-changing dataset called CSCC with the following characteristics: (1) Severe: A large number of people have cloth-changing over all four seasons. (2) High definition: The resolution of the used 13 cameras ranges from 1920×1080 to 3840×2160, which ensures that the recorded people are clear. Examples are shown in Fig.2. Furthermore, we provide two variants of CSCC considering different degrees of cloth-changing, namely moderate and severe, for researchers to evaluate their models from various aspects.

The main contributions can be summarized as follows:

• A novel Universal Clothing Attribute Disentanglement network (UCAD) is proposed. It contains two branches. The Clothing Attribute Extraction branch aims to extract clothing feature which contains no identity information (e.g. body contour). In addition, the Clothing Attribute Filtering branch distills the knowledge from the CAE branch aiming to decompose and disentangle the clothing feature and the identity feature to alleviate the influence of the worn clothing.
• A severe cloth-changing person Re-ID dataset, called CSCC, is introduced to advance the study of more challenging yet realistic cloth-changing Person Re-ID, and a new evaluation criterion considering moderate and severe cloth-changing settings is proposed.
• Extensive experiments on several cloth-changing benchmarks including CSCC and short-term dataset Market1501 demonstrate the effectiveness of our approach.

2 RelatedWorks

2.1 Cloth-Changing Person Re-IDDatasets

A few datasets were collected specifically for cloth-changing setting. Celeb-reID [2019a] are acquired from the Internet which consists of street snapshots of celebrities. Yu et al. [2020] collected a cloth-changing Re-ID dataset called COCAS with cloth templetes. Qian et al. [2020b], Yang et al. [2021] and Wang et al. [2020] constructed three real-world cloth-changing Re-ID datasets LTCC, PRCC and NKUP respectively. Wan et al. [2020], Li et al. [2020b] and Xu et al. [2021] used generation or rendering methods to construct three virtual datasets VC-Clothes, Div-Market and MarketClothes. However, all the existing datasets do not include clothes change over all the four seasons. Therefore, we construct a large-scale real-world cloth-changing dataset CSCC with a lot of people have cloth-changing over four seasons.

2.2 Cloth-Changing Person Re-IDMethods

The dramatic change in appearances when one person wears different clothes and the confusability when different people wear similar/same clothes are the main causes of failing short-termmethods in cloth-changing scenarios. Some clothchanging person Re-ID methods aim at finding cloth-agnostic representations. For example, Fan et al.[2020] proposed RFReID to use reflected wireless signals to identify persons. Yang et al.[2021] proposed a network to use human contour sketch information to identify persons. Chen et al.[2021] proposed to use 3D reconstruction to obtain the 3D shape feature to identify persons. We think certain kinds of feature is not discriminative enough and all identifiable features should be considered. Other works aim at separating clothing and identity features. Huang et al.[2019b] proposed a network using vector neurons instead of scalar neurons. The direction of the vector represents cloth-changing and the length of the vector stands for the identity. Qian et al.[2020b] proposed to distill shape information and disentangle cloth information using key points information. Yu et al.[2020] introduced a method using template clothes to achieve the purpose of retrieving a specific pedestrian wearing template clothes. Xu et al.[2021] proposed to use generated adversarial learning to disentangle the identity and clothing features. However, they ignore that the same clothes would look different on different people thus the clothing feature still contains some identity information (e.g. body contour). Therefore, in this paper, we aim to extract clothing feature which contains no identity information and learn the identity-related features to retrieve persons.

3 Severe Cloth-Changing Dataset

3.1 Data Collection and Labeling

A total of 13 cameras are used to record the raw videos. Five of the cameras have a resolution of 3840×2160, and the rest of them have a resolution of 1920×1080. Mask-RCNN [He et al., 2017] detector is applied to obtain the person images. Each image is annotated with person ID, camera ID, cloth ID, and universal clothing attribute ID.

3.2 Statistics and Comparison

The proposed CSCC dataset has 36,700 person images of 267 identities. Each identity is captured by at least 2 cameras. The number of outfit changes ranges from 2 to 21 for each identity. In addition, a large number of identities (34 identities) have cloth-changing during all four seasons and almost all the identities have clothes variation in different seasons.

We randomly split the identities for training and testing. The training set contains 21,392 images of 134 identities and the rest 15,308 images of 133 identities are partitioned into the test set. For gallery and query splitting, we randomly picked one image per suit for each identity in each camera view to form the query set, and the remaining test set is left as the gallery set. The query set contains 1,614 images and the gallery set contains 13,694 images. Comparison with existing cloth-changing datasets is summarized in Tab.1.

4 Methodology

In this section, a novel Universal Clothing Attribute Disentanglement (UCAD) model is proposed to tackle the challenge of clothing changes. We aim to weaken the influence of the clothing and force the model to learn identity features that are unrelated to the worn clothing. The UCAD includes two branches: Clothing Attribute Extraction (CAE) Branch and Clothing Attribute Filtering (CAF) Branch. More details are listed below. The overall architecture is shown in Fig.3.

4.1 Clothing Attribute Extraction Branch

The goal of the CAE branch is to extract the clothing feature which contains no identity feature. Specifically, an off-theshelf Human Parsing Estimation model is used to parse the clothing part from a person image to form a clothing image so that the extracted features are not affected by the background and head (clothing images are provided in our dataset). Next, we send the clothing image to the CAE branch which consists of a backbone network and an attention module to extract the clothing feature called the clothing attribute feature f CA.

How to decouple with identity. We notice that the same clothes would look different on different people (as shown in Fig.1) which means the worn clothing may reveal some kind of information (e.g. body contour) of an identity. To extract clothing features that contain no identity information, we must find the generality of the same/similar clothing worn by different people. Therefore, we annotate each image with a universal clothing attribute label indicating the general attribute of clothing such as dark coat/dark trouser and lightcolored coat/light-colored trouser etc.. In the training stage, a fully-connected classifier is used to predict the universal clothing attribute labels. In addition, a triplet loss is added to pull in the features of the similar clothes worn by different people and push away the features of different clothes worn by the same person. The features of similar clothes worn by different people will be clustered together thus containing no identity information.

where N indicates the number of training images, yc/ ˆ yc represents the ground truth and predicted clothing attribute label respectively. f a c represents the clothing attribute feature of an anchor sample. f p c /f n c represents the clothing attribute feature of a positive/negative sample that belongs to the same/different class as the anchor sample. C, K stands for the number of clothing attributes and images of each clothing attribute in a triplet batch. [·]+ represents Hinge loss.

4.2 Clothing Attribute Filtering Branch

After getting the clothing attribute feature f CA, the CAF branch distills knowledge from it aiming to decompose and disentangle the clothing feature and the identity feature so that the model would learn identity-related features that are unrelated to the worn clothing. Specifically, the original person image is sent into the backbone network to extract feature which contains both clothing and identity information. Then, the attention module is applied to decompose these two kinds of information and a clothing feature f C and an identity feature f ID are obtained. Another attention module is applied to get the final identity feature f ID and then a fully-connected classifier is used to predict the identity. A triplet loss is added to improve the discrimination of the identity feature.

where yi and ˆ yi represent the ground truth and the predicted identity label respectively.

The clothing feature is passed through a fully-connected classifier to predict the universal clothing attribute labels. Besides, a consistent loss is added to distill the knowledge from the clothing attribute feature to explicitly ensure the clothing feature contains no identity information. In addition, an orthogonal loss is utilized to encourage the clothing and identity features to be decoupled so that the identity feature could include all identifiable information except clothing.

where CF(·) stands for consistency alignment function (Mean Square Error (MSE) function in this paper). f CA i , f C i , and f ID i represent the clothing attribute feature, the clothing feature, and the identity feature of the i'th image respectively. A·B represents dot product between vector A and vector B.

In the training stage, the overall loss function is,

where α, β, γ, δ and ζ are all weighting factors which control the contribution of each term.

4.3 Optimization

First of all, the clothing images are acquired in advance in an offline way and the universal clothing attribute labels are manually annotated. Then, we use Eq.3 to train the CAE branch for NA epochs individually. Then, the clothing attribute feature f CA is obtained for the CAF branch to distill knowledge. The Eq.9 is used to supervise the CAF branch training for NF epochs with the CAE branch fixed.

In the testing stage, only the CAF branch network is remained to extract the identity feature for retrieving.

5 Experiments

In this section, we perform experiments on existing clothchanging datasets including LTCC [2020b], PRCC [2021], VC-clothes [2020], NKUP [2020], our CSCC and short-term dataset Market-1501 [2015] to evaluate our approach.

5.1 Datasets

LTCC [2020b] contains 17,138 images of 152 identities captured by 12 cameras. Training set contains 9,576 images of 77 persons and the testing set contains 493 query images and 7,050 gallery images of 75 identities.

PRCC [2021] contains 33,698 images from 221 identities captured by 3 cameras. The whole dataset is divided into a training set containing 22,898 images of 150 identities and a test set containing 10,800 images of 71 identities.

VC-clothes [2020] contains 19,060 images of 512 identities with 4 cameras. 9,449 images of 256 identities are randomly selected as the training set, and the remaining 256 identities are divided into the testing set where contains 1,020 query images and 8,591 gallery images.

NKUP [2020] contains 9,738 images of 107 persons captured by 15 cameras. This dataset was split into 5,336 training images of 40 identities, 332 query images and 4,070 gallery images of 67 identities.

CSCC (Ours) contains 36,700 images of 267 identities captured by 13 cameras. This dataset was split into 21,392 training images of 134 identities, 1,614 query images, and 13,694 gallery images of 133 identities.

Market-1501 [2015] contains 32,668 images of 1,501 identities captured by 6 cameras. 12,936 images from 751 identities are selected for training and 19,732 images from 750 identities form the testing set.

5.2 Protocols andMetrics

Following [Qian et al., 2020b], we use two settings in evaluation on our dataset CSCC as follows: (1)Standard Setting: For each query, the images in the test set with the same identity and same camera are discarded when evaluation. The test set contains both cloth-consistent and cloth-changing examples in this setting. (2)Cloth-changing setting: For each query, the images in the test set with the same identity and same camera (clothing) are discarded when evaluation. The test set contains only cloth-changing examples in this setting. The Cumulated Matching Characteristics (CMC) and mean average precision (mAP) are adopted as evaluation metrics.

As for the existing datasets, we follow the protocol as the authors proposed.

5.3 Implementation Details

Our model is implemented based on the Pytorch framework. The backbone network in our model is ResNet-50 [2016] pretrained on ImageNet, other parameters are all initialized randomly. We employ JPPNet [2019] to get the clothing images. All images are resized to 384×192. The stochastic gradient descent (SGD) with a momentum of 0.9 is adopted. The hyper-parameters α, β, γ, δ, ζ in Eq.9 and m in Eq.5 are set to 1.0, 0.8, 0.3, 0.8, 0.3, 0.8 respectively. The learning rate of the CAE branch is set to 10-4 and trained for 200 epochs. The learning rate of the CAF branch is set to 10-3 at the beginning and decays to 10-4 and 10-5 after 70 and 170 epochs respectively. The whole training process lasts for 200 epochs.

5.4 Comparison with State-of-the-Arts

Some cloth-changing Re-ID models and short-term methods are our competitors. Comparative results on four clothchanging datasets, CSCC and short-term dataset Market-1501 are shown in Tab.2, Tab.3 and Tab.4 respectively.

Long-term datasets. As is shown in Tab.2 and Tab.3, we have the following key findings: (1) Our method outperforms the short-term methods by a large margin as our method is specially designed for cloth-changing person Re-ID. (2) Our method still achieves improvements on five cloth-changing datasets compared with cloth-changing methods but underperforms 3DSL[Chen et al., 2021] on dataset PRCC [Yang et al., 2021]. The reason is that PRCC is a dataset that all people share the same clothes in one camera view. The 3D shape is unrelated to clothes thus more effective in this situation.

Short-term datasets. As is shown in Tab.4. Our method achieves comparable results on Market-1501[Zheng et al., 2015] compared with short-termmethods. This indicates that our method is effective under both cloth-changing and clothconsistent setting.

5.5 Ablation Studies

We conduct comprehensive and systematic ablation studies on CSCC to evaluate our method.

Effectiveness of losses. As is shown in Tab.5. Firstly, compared with the baseline model, adding the orthogonal loss or the consistent loss can lead to performance improvements, indicating their effectiveness. When they are both added to train the network, the maximum performance is observed.

Influences of identity and clothing feature. We change the retrieval feature to evaluate our model as shown in Tab.6. First, the maximum Rank-1 under the standard setting is obtained when we concatenate clothing and identity feature to retrieve which indicates combining clothing feature is useful when the gallery has cloth-consistent images. Second, the clothing is no more useful under the cloth-changing setting.

Influences of hyper-parameters. There are six hyperparameters in our model, α, β, γ, δ, ζ in Eq.9 controlling contribution of each item and m in Eq.5 controlling the margin of the triplet loss. We set all parameters to 1 at the beginning, and change from 1 to 0 with intervals of 0.1 one by one with others fixed to find the optimal value. It's worth noting that the best results yield at the same value for five datasets. It suggests these hyper-parameters are robust to different datasets. Tab.8 shows example of the influences of δ.

Effectiveness of the universal clothing attribute label. As shown in Tab.9, w/o universal clothing attribute label (UCA Label) refers to treating one clothing worn by each pedestrian as a distinct category. LTCC(Universal) → CSCC refers to the CAE branch trained on the LTCC dataset with the universal clothing attribute labels is used when the whole network is trained on CSCC. Results indicate the effectiveness of the universal clothing attribute label and the CAE branch trained on one dataset with the universal clothing attribute labels can be transferred to another dataset.

Influences of face and resolution. w/o face in Tab.10 refers to using face detector MTCNN [Zhang et al., 2016] to detect the faces and mask the faces. Resize in Tab.10 refers to resizing the images to half their original size (both width and height). We carried out experiments with our method UCAD. It can be observed in Tab.10 that there is no obvious performance degradation when we mask the faces and our method achieve improvements without faces even compared with SE+CESD with faces (w/o face in Tab.10 compared with SE+CESD in Tab.3). This indicates our method focus on all identifiable features (shape, haircut, face, body contour etc.) not just some kind of information (shape or face).

5.6 Degree of Cloth-Changing: ANewCriterion

Some methods [Yang et al., 2021] are specifically designed for the moderate cloth-changing setting which also have promising applications. To promote the research of related works, we provide two variants of our dataset: moderate, and severe subset. The moderate subset contains identities that have cloth-changing both in summer and spring (autumn). Clothes in the moderate subset would not strongly affect a person's contour sketch. It contains 15,870 images of 233 identities. The severe set is complete CSCC with people changing clothes during four seasons. We conduct experiments on these two subsets. The results are listed in Tab.7.

We have the following key findings: (1) Higher performance under standard setting is observed on severe set compared with the moderate set. The reason is that the persons with cloth-changing over four seasons (34 persons in total) are not abundant. Winter clothes they wear are distinct so that models overfit on clothes. (2) Lower performance under cloth-changing setting is observed on severe set compared with the moderate set. The reason is that the images with the same clothes in the gallery are discarded, cloth-changing over four seasons especially heavy winter clothing severely obstructs discernible features. This suggests the severe set is more challenging under cloth-changing setting. Researchers can effectively evaluate their models from various aspects.

6 Conclusion

In this paper, we proposed a novel Universal Clothing Attribute Disentanglement (UCAD) method which can effectively weaken the influence of clothing and force the model to learn identity-related features which are unrelated to the worn clothing. To further study Re-ID, we construct a severe cloth-changing person Re-ID dataset called CSCC containing 36,700 images of 267 identities. In addition, two variants considering moderate and severe cloth-changing setting are provided for researchers to evaluate their models from various aspects. Moreover, experiments on existing cloth-changing datasets including our CSCC and short-term dataset Market1501 demonstrate the effectiveness of UCAD.

Acknowledgments

This work was supported in part by the Research Project of ZJU-League Research & Development Center, Zhejiang Lab under Grant 2019KD0AB01.