Graph Consistency Based Mean-Teaching for Unsupervised Domain Adaptive Person Re-Identification

1 Introduction

Person Re-Identification (ReID) aims to match a query person image in a gallery set [Ge et al., 2020a; Zhai et al., 2020]. Supervised person ReID has been widely studied from different aspects. However, supervised models suffer from the expensive data annotation and substantial performance drop when applied on different target domains. To address these issues, recent works focus on unsupervised domain adaptive person ReID and exhibit promising performance. More details of recent works are summarized in Sec. 2.

Despite the significant success, there still remain several open issues unexplored. Firstly, previous works commonly perform training on both labeled source data and unlabeled target data [Liu and Zhang, 2020]. In real-world applications, however, it is always impractical to access the labeled source data due to privacy issue. Moreover, labeled datasets are also more expensive to store and process than pre-trained models. This hinders the training of these methods. Secondly, person ReID model optimized by mean-teaching framework requires an effective student network training strategy that should be robust to noisy labels and also effective in feature learning. Existing methods use soft triplet loss on selected samples based on pseudo labels [Zhai et al., 2020]. However, it is sensitive to noisy pseudo labels and ineffective in feature learning with few selected sample relationships involved. Thirdly, models pre-trained on different labeled datasets are public available and using them in training has potential to boost the performance on target datasets. However, current methods with soft triplet loss are not effective in collaborative learning with multiple models.

This paper is motivated to study an effective training method for unsupervised domain adaptive person ReID. To release the dependence on labeled source data, only unlabeled target data is used in unsupervised training. To enhance the effectiveness in feature learning of student networks and cooperation of teacher networks, we propose a Graph Consistency based Mean-Teaching (GCMT) method with a Graph Consistency Constraint (GCC) between teacher and student networks as shown in Fig. 1. GCMT merges sample relationships predicted by different teacher networks as supervision and optimizes more relationships among different samples in student networks, thus provides effective supervision and optimization for feature learning on student networks.

As illustrated in Fig. 1, the proposed Graph Consistency Constraint (GCC) in GCMT method is performed between teacher graphs and student graphs. Specifically, given public pre-trained models, GCMT initializes a student network and its temporal mean teacher network from each model. Features extracted by a teacher network are formed into a teacher graph with using features as nodes. Connections between nodes in teacher graphs are determined by KNN strategy, i.e., a node is only connected with its KNN nodes. Weights of connections are computed as feature similarities. Features extracted by a student network are formed into a student graph with weights of connections also computed as feature similarities. Teacher graphs are fused to guide the connection weight leaning in student graphs by GCC via encouraging the connection weights in student networks to approach the weights in the fused graph. Thus, GCC guides student graphs to describe the same sample relationship with the fused teacher graph. Compared with previous methods, proposed GCC avoids the dependency on pseudo labels. It involves relationships among more samples in feature learning and also can effectively fuse sample relationships predicted by multiple teacher networks, hence is more effective in feature learning and also robust against noises in pseudo labels. To the best of our knowledge, it is the first work to use GCC in meanteaching framework for unsupervised training.

We test our method on three large-scale datasets, i.e., Market-1501, DukeMTMC-reID , and MSMT17. Comparison with recent works shows that our method outperforms stateof-the-art methods by a clear margin. For instance, training from model pre-trained on ImageNet, GCMT achieves mAP accuracy of 63.6% on DukeMTMC-reID, outperforming recent UTAL [Li et al., 2019a] by 19%. By using model pre-trained on DukeMTMC-reID, our GCMT method achieves mAP accuracy of 77.1% on Market-1501, significantly outperforming recent MEB-Net [Zhai et al., 2020] and MMT [Ge et al., 2020a] by 4.9% and 5.9%, respectively. Experiments also show that GCMT boosts the performance with multiple pre-trained models, e.g., the mAP accuracy is boosted to 79.7% on Market-1501 with models pre-trained on MSMT17, CUHK03, and DukeMTMC-reID.

2 RelatedWork

Domain-invariant feature learning. Some works design GAN based models to transfer labeled images to target domains [Liu et al., 2019]. Several works map labeled and unlabeled images to a shared feature space to bridge the domain gap [Liu and Zhang, 2020]. These methods require labeled source data for training, which is hardly available due to privacy issue in real-world applications. Compared with them, this work only uses pre-trained models instead of labeled data, thus is more suitable for real-world applications.

Self-supervised learning. Some works locally predict pseudo labels for unlabeled samples [Zhong et al., 2019; Yu et al., 2019]. To acquire reliable pseudo labels, some researchers try to refine unsupervised clustering [Ding et al., 2019; Lin et al., 2019]. For unsupervised optimization, current works use pseudo labels as supervision and adopt triplet loss [Fu et al., 2019; Zhang et al., 2019] or contrastive loss [Liu and Zhang, 2020; Zhong et al., 2019] for training. However, these methods fail to consider the noise in pseudo labels, which substantially hinders the model training. Compared with them, proposed model adopts fused graph of temporal mean teacher networks to supervise feature learning, alleviating the effect of noisy labels and improves the performance. Ge et al. [Ge et al., 2020b] propose a self-paced learning method to boost the ReID performance. However, they have different motivations and propose different algorithms compared with this paper. They aim to make full use of labeled data by caching features in a hybrid memory bank. While, this paper aims to study an effective mean-teaching method on unlabeled datasets without labeled data by proposing a graph consistency constraint. Experiments show our method achieves better performance under the same setting.

Teacher-student training. Teacher-student framework is widely studied in semi-supervised learning [Tarvainen and Valpola, 2017; Han et al., 2018].However, these methods hold the assumption that labeled and unlabeled images are of same categories, making them not suitable for person ReID task. Recently, some works adopt mean-teaching method in unsupervised domain adaptive person ReID. MMT [Ge et al., 2020a] uses two pairs of teacher and student networks for pseudo label refinery. MEB-Net [Zhai et al., 2020] further uses three pairs of teacher and student networks of three different structures for training. MMT and MEB-Net use triplet loss in model training and soft triplet loss between teacher and student networks, which are sensitive to noisy label and ineffective in feature learning. Moreover, training with multiple teacher networks is still ineffective and the improvements by using three teacher networks is limited in [Zhai et al., 2020]. Compared with them, proposed GCMT is effective in both student network training and teacher network cooperation. Experimental results show that GCMT achieves better performance compared with these methods.

3 ProblemFormulation

Given an unlabeled dataset D, unsupervised domain adaptive person ReID aims to learn the ReID model on D without annotation. D can be denoted as {xi|i = 1...N}. xi and N denote the i-th image and the number of images in D, respectively. In this paper, training starts from public models pre-trained on different datasets, while labeled source data is not used. This setting is more practical in real-world scenario compared with several previous works as discussed in Sec. 1. Public pre-trained models are denoted as {M 1 , M 2 , ..., M m} where m denotes the number of models.

We adopt mean-teaching framework for unsupervised training on D as shown in Fig. 2. m student networks are initialized by m public pre-trained model, respectively. The parameters of student networks are denoted as {θ1 , θ2 , ..., θm} with θj denoting the parameters of the j-th student network. We use temporal average network as the teacher network for each student network as in [Tarvainen and Valpola, 2017]. Parameters in teacher networks are denoted as {Θ1 ,Θ2 , ...,Θm}, respectively. In the beginning of training, θj and Θj are both initialized byM j . In each iteration of the training, θj is updated by objective functions and Θj is then updated by temporal average strategy with updated θj as:

Coefficients in Eqn. (1) are empirical values following [Ge et al., 2020a; Zhai et al., 2020].

For unlabeled images xi, the extracted feature after L2normalization and class prediction after softmax normalization by the j-th student network θj are denoted as f j i and pji , respectively. And the extracted feature and prediction by the j-th teacher network Θj for xi are denoted as F j i and P j i , respectively. Note that we use the superscript and subscript to denote the index of networks and images, respectively.

The target of training on D is to make extracted features discriminative for person ReID task. In this paper, we propose a Graph Consistency based Mean Teaching (GCMT) method to achieve this goal via constraints in two spaces: 1) class prediction space, and 2) feature space. The framework of proposed method is illustrated in Fig. 2.

In class prediction space, we first generate pseudo ID labels on D by unsupervised clustering on averaged features extracted by teacher networks before each epoch. Based on pseudo labels, cross entropy loss LCE is computed for pseudo ID classification on each student network. We use a new fully connected layer for label prediction after each clustering step following [Ge et al., 2020a; Zhai et al., 2020]. After clustering features into C clusters, a mean feature for each cluster is computed by averaging all features in this cluster. This results in C mean features corresponding to C clusters. Then, a new fully connected layer with C-way outputs is used for label prediction and parameters are initialized with corresponding mean features, i.e., the parameters for the c-th class is initialized by the mean feature of the c-th cluster.

To eliminate the negative effects of noises in pseudo label inside each training batch, mutual cross entropy loss LMCE is used to encourage student networks to predict same class probabilities with teacher networks for training samples. Specifically, the probabilities of xi by different teacher networks are averaged as the training target for each student network, which is denoted as ˆ Pi = ∑mj P j i . The objective function of LMCE with m teacher networks is formulated as:

where C denotes the number of identities generated by clustering. ˆ Pi(c) and pji (c) denote the probability of xi being classified to c-th class in ˆ Pi and pji , respectively. The objective function in label prediction space LLP is the combination of LCE and LMCE as:

Classification probabilities describe the relationship between samples and classes. Thus, LLP focuses on classifying training samples to corresponding pseudo labels. For ReID task, however, we also need to optimize the model in feature space, as identities in testing set are different with training set and ReID is performed as retrieval by extracted features. Thus, we propose Graph Consistency Constraint (GCC) between teacher and student networks in feature space.

As shown in Fig. 2, GCC is proposed to train student networks in feature space. GCC computes sample similarity relationship for each network with corresponding extracted features, respectively. Sample similarity relationships predicted by different teacher networks are fused as the target relationship. Student networks are encouraged by GCC to output features that have the same sample similarity relationship as the target relationship as shown in Fig. 1. The objective function of GCC is denoted as LGCC and details of the computation on LGCC will be given in Sec. 4.

LLP is performed in class prediction space and focuses on relationship between samples and pseudo IDs. While LGCC is performed in feature space and focuses on similarity relationship among samples. Therefore, LLP and LGCC are complementary to each other. We hence denote the final objective function as the combination of LLP and LGCC:

where λGCC is the loss weight for LGCC.

4 Graph Consistency Constraint

In feature learning, we aim to fuse sample similarity relationships computed by different teacher networks to provide training supervision for student networks. Features extracted by different teacher networks are of different distributions. Thus, directly averaging similarities computed by different teacher networks is not reasonable for relationship fusion. Moreover, when only relationship between several selected samples are involved, e.g., soft triplet loss with hard sample mining in [Zhai et al., 2020], noise in pseudo labels will mislead both the sample selection and optimization.

To chase a reasonable relationship fusion and an effective student network optimization, we propose the Graph Consistency Constraint (GCC) that describes sample similarity relationship as weights of edges in graphs. Then, GCC performs relationship fusion and network optimization based on graphs. By fusing relationship predicted by different teacher networks and involving more samples in training student networks, GCC eliminates the effect of noises in pseudo labels and provides effective update for student networks.

4.1 Teacher Graph Construction

Similarity relationship predicted by each teacher network is described as a graph. Specifically, a K-nearest neighbour graph G(F j ,W j ) is constructed for features extracted by the j-th teacher network Θj . F j = {F j i |i = 1...N} and W j = {W j i,k |i, k = 1...N} denote the set of features of nodes and the set of weights of edges between nodes, respectively. Weights of edges describe the distance relationships among different features in the graph, which are computed based on K-NN relationship. Specifically, F j k is connected to F j i if F j k is among the K-nearest neighbours of F j i . The weight of edge from F j i to F j k , i.e., W j i,k , is computed as W j i,k = (F j i )T F j k . For pairs of nodes that are not connected, corresponding weights of edges are assigned as 0.

Graphs constructed by different teacher networks are fused to provide training target for student networks training. Weights in each graph are first normalized by softmax and then averaged to obtain the fused weights as follows:

The set of fused weights ˆWi,k is denoted as ˆW.

4.2 Graph Consistency Constraint Computation

Student networks are supervised by GCC to extract features that have the same similarity relationships between different samples described by ˆW. The computation of GCC should satisfy three conditions. 1): GCC should simultaneously update every relationship between different training samples for student networks, making optimization effective and precise. 2): GCC should be aware of the density of neighbours. Some samples have dense neighbours whose K-nearest neighbours should reach rather high similarities with them. While some other samples have sparse neighbours, and restriction on their neighbours can be relaxed. Thus, GCC should optimize relative similarity between samples, instead of absolute similarity. 3): GCC should be aware of the hardness of K-nearest neighbours of each sample, i.e., gradients of far neighbours should be larger than gradients of near neighbours.

For computing GCC, student graph G(f j ,wj ) is constructed for features extracted by the j-th student network θj , where f j = {f j i |i = 1...N} and wj = {wj i,k |i, k = 1...N} denote the set of features of nodes and the set of weights of edges between nodes, respectively. Weights of edges from f j i to f j k in G(f j ,wj ) is computed as:

where β is a hyper-parameter to adjust the scale of distribution. GCC is computed to encourage {wj |j = 1...m} to approach ˆW via cross entropy which is formulated as:

Proposed objective function in Eqn. (7) satisfies aforementioned three conditions. 1): Every edge in student graphs is updated by LGCC based on ˆW. 2): Due to the softmax normalization in Eqn. (6), LGCC normalizes loss of samples with different densities of neighbours to the same scale. Thus, LGCC is aware of the density of neighbours. 3): The gradient of LGCC relative to wj i,k when ˆWi,k ̸= 0 is computed as: ∂LGCC ∂wj i,k = - ˆWi,k Nwi i,k . The scale of gradient increases with wj i,k decreasing. This indicates that LGCC pays more attention on K-nearest neighbours with small similarities for each sample. Thus, LGCC is also aware of hardness.

4.3 Discussion

GCMT fuses sample similarity relationship computed by different teacher networks as supervise signal instead of pseudo labels. As networks are being more discriminative with training, similarity relationship computed in each iteration has potential to be more precise than pseudo labels computed offline before each epoch. Moreover, LGCC avoids sample mining algorithm and automatically focuses on hard neighbours.

Compared with soft triplet loss used in [Zhai et al., 2020] and [Ge et al., 2020a] that trains student networks based on pseudo labels, LGCC optimizes distance relationship in student graphs. This considers more relationship among samples and avoids the negative effect of noises in pseudo labels. Specially, soft triplet loss can be regarded as a special case of GCC that uses sub-graphs with selected samples. Advantages of our method will be shown in Sec. 5.3.

Compared with MEB-Net [Zhai et al., 2020] that fuses relationship predicted by different teacher networks via soft triplet loss based on pseudo labels, GCC performs relationship fusion based on graphs. This avoids effect of noises in pseudo label and involves more relationship in training. Advantages of our method in multiple teacher networks cooperation will be shown in Sec. 5.4.

5 Experiment

5.1 Dataset

Experiments are performed on three datasets, i.e., DukeMTMC-reID [Zheng et al., 2017], Market-1501 [Zheng et al., 2015], and MSMT17 [Wei et al., 2018].

DukeMTMC-reID contains 36,411 images of 1,812 identities. 16,522 images of 702 identities are used for training. In the rest of images, 3,368 images are selected as query images and remaining 19,732 images are used as gallery images.

Market-1501 contains 32,668 images of 1,501 identities. 12,936 images of 751 identities are selected for training. In the rest of images, 3,368 images are selected as query images and remaining 19,732 images are used as gallery images.

MSMT17 contains 126,441 images of 4,101 identities. 32,621 images of 1,041 identities are selected for training. In the rest of images, 11,659 images are selected as query images and remaining 82,161 images are used as gallery images.

For short, we denote DukeMTMC-reID as Duke, Market1501 as Market, and MSMT17 as MSMT in the rest of this paper. Following previous works [Ge et al., 2020a], Cumulative Matching Characteristics (CMC) and mean Average Precision (mAP) are used to evaluate the performance. Rank1, Rank5 and Rank10 accuracies in CMC are reported.

5.2 Implementation Detail

Public models pre-trained on ImageNet [Deng et al., 2009], CUHK03 [Li et al., 2014] (14,096 images with 1,467 identities for training), Duke, Market, and MSMT17 are obtained following [Ge et al., 2020a] and denoted as M I , M C , M D , M M , and M MS , respectively. These pre-trained models adopt ResNet50 [He et al., 2016] as backbone. Input images are resized to 256×128. We use random flipping, random cropping, and random erasing [Zhong et al., 2020] for data augmentation. We feed the same image batch to different pairs of teacher and student networks but with separately random augmentation. K-Means method is used for unsupervised clustering. The number of clusters is set to 500 on Duke and Market and 1,500 on MSMT following [Ge et al., 2020a; Zhai et al., 2020]. In each training batch, 16 identities are randomly selected and 4 images for each identity are selected, resulting 64 images. K is set as 12 in teacher graph construction. Loss weight λGCC is set as 0.6. β is set to 0.05 following [Liu and Zhang, 2020]. Adam optimizer is used for training. Learning rate is initialized as 0.00035 and decayed by 0.1 after 20 epochs. Models are totally trained for 120 epochs with 400 iterations in each epoch. After training, teacher networks are used to extract features for ReID and the best performance among different teacher networks is reported. Models are trained on a server with three GeForce RTX 2080 Ti GPUs and one Tesla V100 GPU.

5.3 Compared with State-of-the-art Methods

In this section, proposed GCMT method is compared with many state-of-the-art methods. For fair comparison, GCMT uses only one pre-trained model in this section. The comparison results are summarized in Table 1 and Table 2. It can be observed that proposed GCMT method outperforms previous methods by a clear margin. It is worth noting that GCMT with M I , i.e., in the unsupervised setting, even outperforms most recent domain adaptive methods.

On Market and Duke, proposed GCMT is compared with recent state-of-the-art methods as shown in Table 1. It is clear that GCMT achieves the best perform compared with others. For example, GCMT achieves 77.1% and 67.8% in mAP accuracy on Market and Duke, respectively. This outperforms MMT [Ge et al., 2020a] by 5.9% and 4.7%, respectively. MEB-Net§ [Zhai et al., 2020] uses DenseNet121 [Huang et al., 2017] as backbone, while GCMT still outperforms it by 1.1% and 1.7% in mAP accuracy on Market and Duke, respectively. This indicates that proposed GCMT is powerful to learn discriminative person features without annotation.

MSMT is more challenging than Market and Duke due to more identities and diverse appearance. We compare GCMT with state-of-the-art methods as shown in Table 2. It is clear that GCMT outperforms previous methods by a clear margin. For example, GCMT achieves 54.8% in Rank1 accuracy when using M M , outperforming DIM+GLO and MMT by 5.1% and 5.6%, respectively. Comparison on MSMT further demonstrates the effectiveness of proposed GCMT method on large-scale person ReID task.

5.4 Model Analysis

In this section, we first analysis the effect of β, λGCC and K respectively by varying the value of one parameter and keeping others fixed to the optimal value. Experiments show that hyper-parameters selected on one dataset can be applied to others. We then evaluate proposed GCC to show its validity. Finally, we show that GCMT could effectively boost the performance with multiple pre-trained models.

Evaluation on β is performed on Market and Duke using M I as source model, as shown in Table 3. It is clear that setting β to 0.05 achieves the best performance.

Evaluation on λGCC is performed on Market and Duke with pre-trained models M D and M M , respectively. The experimental results are shown in Fig. 3. It can be observed that setting λGCC in the range from 0.1 to 0.7 can always improves the performance and the best performance is achieve when setting λGCC to 0.6 on both dataset.

Evaluation on K is also performed on Market and Duke with pre-trained models M D and M M , respectively. The evaluation results are shown in Fig. 3. Setting K = 0 denotes method without LGCC. It can be observed that setting K larger than 0 always improves the performance compared with K = 0 and the best performance is achieve when K = 12 on both datasets. This shows the validity of using GCC in GCMT for feature learning.

Evaluation on proposed GCC is performed on Market and Duke with pre-trained models M D and M M , respectively. The comparison results are shown in Table 4. It can be observed that GCC improves the performance on both datasets. For example, GCC improves mAP accuracy by 2.6% and 5.2% on Market and Duke, respectively. It is also clear that GCC outperforms soft triplet loss used in [Zhai et al., 2020], e.g., by 3.6% in mAP on Duke.

With multiple pre-trained models, multiple pairs of teacher and student networks are used in training. The performance comparison is shown in Table 5. It can be observed that, given multiple pre-trained models, GCMT is able to effectively boost the performance. For example, GCMT improves the mAP accuracy to 78.6% on Market by using two pretrained models M D and M MS . GCMT further boosts the mAP accuracy to 79.7% on Market by additionally using M C . It is worthy noting that the improvement of using multiple source models is based on high performance by single source model, which approaches the supervised baseline as shown in Table 1. Thus, the improvement is significant though it may be of small scale. We also implement MEBNet [Zhai et al., 2020] method with three pre-trained models by provided source code. It can be observed that GCMT outperforms MEB-Net by a clear margin, e.g., by 6.6% in mAP on Market. This further shows the advantage in teacher networks cooperation of our method.

6 Conclusion

This paper proposes a Graph Consistency based MeanTeaching (GCMT) method for unsupervised domain adaptive person ReID task. GCMT fuses sample similarity relationships predicted by different teacher networks as supervise signal based on graphs. With fused sample similarity relationships, GCMT uses the proposed Graph Consistency Constraint (GCC) to train student networks with more sample similarity relationships involved. GCMT provides an effective method for teacher networks cooperation and student networks optimization. Extensive experiments show that our GCMT outperforms state-of-the-art methods by a clear margin. Experimental results also show that GCMT effectively improves the performance given multiple pre-trained models.

Acknowledgments

This work is supported in part by Peng Cheng Laboratory, in part by The National Key Research and Development Program of China under Grant No. 2018YFE0118400, in part by Natural Science Foundation of China under Grant No. U20B2052, 61936011, 61620106009, in part by Beijing Natural Science Foundation under Grant No. JQ18012.