SimMC: Simple Masked Contrastive Learning of Skeleton Representations for Unsupervised Person Re-Identification

1 Introduction

Person re-identification (re-ID) targets at retrieving and matching the same pedestrian from different views or occasions, which assumes a pivotal role in various applications such as intelligent surveillance, robotics, and security authentication [Ye et al., 2021]. Recently, person re-ID via 3D skeletons has drawn growing interests from academia and industry [Pala et al., 2019; Rao et al., 2021b; Rao et al., 2021d]. Compared with conventional image-based methods that typically rely on visual features such as human silhouettes and appearances for recognition [Liu et al., 2015], skeleton-based methods leverage 3D positions of key body joints to characterize discriminative structural and motion features of human body, which could enjoy smaller data size and better robustness against scale and view variation [Han et al., 2017].

† Corresponding author

Despite the great progress in skeleton-based person re-ID, existing endeavors require either extracting hand-crafted features (e.g., anthropometric attributes) [Pala et al., 2019] or learning skeleton representations with the supervision of labels. For hand-crafted methods, they typically require extensive domain knowledge while lacking the flexibility to explore latent features beyond human cognition. To tackle this issue, numerous recent works resort to convolutional neural networks (CNN) [Liao et al., 2020] and long short-termmemory (LSTM) [Rao et al., 2021a] to perform supervised or selfsupervised skeleton representation learning. However, these methods usually require a specific pre-modeling of 3D skeletons (e.g., skeleton graphs [Rao et al., 2021d]), and rely on massive manually-annotated data to train or fine-tune models, which is labor-expensive and unable to learn general pedestrian representations under the unavailability of labels.

To address these challenges, this paper presents a generic Simple Masked Contrastive learning (SimMC) framework, as shown in Fig. 1, which contrasts the typical features and inherent relationships of masked skeleton sequences to learn effective skeleton representations without using any label for person re-ID. Specifically, to fully utilize unique features within skeleton sequences, we first devise a masked prototype contrastive learning (MPC) scheme to cluster subsequence representations (referred as skeleton instances) randomly masked from raw sequences, and contrast the inherent similarity between them and the most typical features (referred as skeleton prototypes) to learn discriminative skeleton representations. By pulling closer skeleton instances belonging to the same prototype and pushing apart instances of different prototypes with the instance-prototype contrastive learning, MPC enables the model to capture discriminative skeleton features and high-level semantics (e.g., intra-class skeleton similarity) from unlabeled skeleton sequences for the person re-ID task. Then, motivated by the nature of motion continuity that typically endows different subsequences with strong correlations (e.g., motion similarity), we propose the masked intra-sequence contrastive learning (MIC) to learn the intra-sequence similarity between subsequences of the same skeleton sequence, which encourages capturing the pattern consistency within sequences to learn more effective representations of skeletons for person re-ID.

The proposed SimMC framework enjoys merits in terms of architectures, performance, and scalability. Firstly, SimMC is primarily built by multi-layer perceptron (MLP) networks with small model complexity, which can directly learn effective representations from raw skeleton sequences without any prior modeling. Secondly, the proposed unsupervised framework outperforms most existing self-supervised and supervised skeleton-based methods that utilize extra label information, and can also be efficiently applied to 3D skeleton data estimated from RGB-based scenes. Lastly, our framework can serve as a generic contrastive learning paradigm to finetune skeleton features learned from existing models, which benefits learning better skeleton representations for the task of person re-ID. In summary, our main contributions include:

• We present a simple masked contrastive learning (SimMC) framework that exploits typical features and relationships of masked unlabeled skeleton sequences to learn discriminative representations for person re-ID.
• We devise a novel masked prototype contrastive learning (MPC) scheme to fully contrast most representative features and learn high-level semantics from subsequence representations masked from skeleton sequences.
• We propose the masked intra-sequence contrastive learning (MIC) to learn inherent similarity and pattern consistency between subsequences, so as to encourage learning more effective representations for person re-ID.
• Empirical evaluations show that SimMC significantly outperforms most state-of-the-art skeleton-based methods on four benchmark datasets, and can be exploited to fine-tune existing skeleton representations and boost their performance with up to 28.2% mAP gains.

2 RelatedWorks

Skeleton-based Person Re-identification. Most existing methods typically extract hand-crafted anthropometric, morphological, and gait descriptors from 3D skeletons to characterize human body and motion features. Seven Euclidean distances between certain joints are utilized by [Barbosa et al., 2012] to construct a distance matrix for person re-ID. Further enhancement with 13 (D13) and 16 skeleton descriptors (D16) are made in [Munaro et al., 2014a] and [Pala et al., 2019], respectively, which leverage k-nearest neighbor, support vector machine or Adaboost classifiers to perform person re-ID. Recently, deep neural networks are widely applied to supervised and self-supervised skeleton representation learning. A CNN-based paradigm, PoseGait [Liao et al., 2020], is devised to encode 81 hand-crafted skeleton/pose features for human recognition. An LSTM-based skeleton encoding model with locality-aware attention (AGE) [Rao et al., 2020] is proposed to learn discriminative gait features from skeleton sequences. SGELA [Rao et al., 2021b] further combines multiple self-supervised pretext tasks (e.g., reverse sequential reconstruction) and inter-sequence contrastive scheme to enhance skeleton pattern learning for person re-ID. The graphbased methods MG-SCR [Rao et al., 2021d] and SM-SGE [Rao et al., 2021a] devise multi-level skeleton graphs and auxiliary self-supervised tasks for person re-ID tasks.

Contrastive Learning. Contrastive learning is widely applied to various self-supervised and unsupervised paradigms [He et al., 2020; Rao et al., 2021b; Rao et al., 2021c; Chen and He, 2021] to learn effective data representations by pulling together positive representation pairs and pushing apart negative ones in a certain feature space. An instance discrimination paradigm based on exemplar tasks [Wu et al., 2018] is devised for image contrastive learning. The contrastive predictive coding (CPC) model with the probabilistic InfoNCE loss [Oord et al., 2018] is proposed to learn general representations from various domains. Recent contrastive paradigms explore mini-batch negative sampling [Chen et al., 2020] and momentum-based encoders [He et al., 2020], while [Chen and He, 2021] devises a Siamese architecture for contrastive learning without using negative pairs or momentum encoders. In [Li et al., 2021], contrastive learning and kmeans clustering are combined for unsupervised learning of visual representations.

3 The Proposed Framework

Suppose that a 3D skeleton sequence S1:f =(S1, · · · , Sf ) ∈ Rf×K , where St ∈ RK is the tth skeleton with 3D coordinates of J body joints and K = J × 3. Each skeleton sequence S1:f belongs to an identity y, where y ∈ {1, · · · , I} and I is the number of different identities. The training set ΦT = {ST ,i 1:f }N1 i=1 , probe set ΦP = {SP,i 1:f }N2 i=1 , and gallery set ΦG = {SG,i 1:f }N3 i=1 contain N1, N2, and N3 skeleton sequences of different persons in different views and scenes. Our framework aims at learning an encoder (denoted as ψ(·)) built with neural networks to encode ΦP and ΦG into effective

skeleton representations {vPi }N2 i=1 and {vGj }N3 j=1 without using any label, such that the representation vPi in probe set can match the representation vGj of the same identity in gallery set. The overview of our framework is presented in Fig. 2.

As shown in Fig. 2, we firstly randomly mask each input skeleton sequence to sample ith and j th subsequences, which are encoded into skeleton instances v(i) and v(j) (see Sec. 3.1). Secondly, we cluster corresponding instance sets V(i) and V(j) individually to generate skeleton prototypes, and then enhance the similarity between instances of same prototype while maximizing the dissimilarity between different ones by minimizing LMPC. Meanwhile, a Siamese architecture is exploited to learn inherent intra-sequence similarity between v(i) and v(j) by minimizing LMIC (see Sec. 3.2).

3.1 Masked Prototype Contrastive Learning

Each person's skeletons typically possess unique features (e.g., anthropometric attributes), while their corresponding sequences could carry recognizable and highly consistent walking patterns [Murray et al., 1964]. Naturally, we expect the model to exploit the most representative skeleton patterns and traits within each sequence for person re-ID. A na¨ıve solution is to cluster skeleton sequences to learn the representative features by direct inter-sequence contrastive learning, while it could overlook some valuable intra-sequence representations (e.g., subsequences) that might contain key patterns. To encourage the model to fully mine intra-sequence skeleton features and high-level semantics (e.g., identityrelated patterns) from skeleton sequences, we propose a masked prototype contrastive learning (MPC) scheme to jointly focus on the most typical features (skeleton prototypes) of different subsequence representations (skeleton instances) randomly masked from original sequences, and exploit the instance-prototype similarity and dissimilarity to learn discriminative skeleton representations.

Given an input skeleton sequence S1:f = (S1, · · · ,Sf), we exploit an MLP encoder with one hidden layer to encode each skeleton as:

where ψ(·) represents the encoder function, W1 ∈ RH×K and W2 ∈ RH×H denote the learnable weight matrices to encode the j th skeleton Sj ∈ RK into a latent feature representation hj ∈ RH , and σ(·) is a ReLU non-linear activation function. Then, to sample subsequence representations from the encoded sequence representation (h1, · · · , hf) of S1:f , we utilize a masking function M to randomly produce x masks, i.e., zero-masking positions, for each skeleton sequence of length f with:

where mj ∈ {0, 1} is the mask status for the j th position of a sequence and ∑fj=1 mj =f-x. We apply the generated random masks to S1:f and its corresponding skeleton representations (h1, · · · , hf) (see Eq. (1)), which are then integrated into a subsequence representation as (see Fig. 2):

where v(i) ∈ RH (i ∈ {1, · · · , q}) denotes the feature representation of ith subsequence sampled from S1:f using x randommasks, q is the number of subsequence sampling, m(i),j denotes the mask status of the j th position at the ith sampling, while wj represents the importance of j th skeleton representation hj . Here each skeleton is assumed to equally contribute to representing sequence features, i.e., wj = 1. For clarity, we use V(i) = {v(i),j}N1 j=1 to denote all subsequence representations in the ith subsequence sampling of the training set ΦT . Note that we sample one random subsequence for each training sequence at each sampling. V(i) = {v(i),j}N1 j=1 are exploited as skeleton instances for the MPC scheme.

To group feature-similar skeleton instances and discover semantic clusters with arbitrary shapes, we leverage the DBSCAN algorithm [Ester et al., 1996] to perform clustering individually on the ith instance set V(i) corresponding to ith subsequence sampling, as shown in Fig. 2, and generate clusters Vc(i) = {vc(i),j }Nc j=1 , c ∈ {1, · · · ,C}, where C is the number of clusters (i.e., pseudo classes), and each cluster Vc(i) contains Nc instances belonging to the cth pseudo class. We averagely aggregate instance features of the same cluster to generate the corresponding skeleton prototype as:

where pc(i) ∈ RH denotes the skeleton prototype of the cth cluster Vc(i) . To jointly focus on the representative skeleton features in all instance sets and encourage capturing highlevel skeleton semantics from different prototypes, we exploit a masked prototype contrastive (MPC) loss to enhance the similarity of each skeleton instance to the corresponding prototype and maximize its dissimilarity to other prototypes by:

where N represents the number of all skeleton instances, Ci denotes the number of skeleton prototypes generated from the ith instance set V(i), Nc is the number of instances belonging to the cth prototype pc(i) in V(i), and τ represents the temperature for contrastive learning. It is worth noting that the na¨ıve prototype contrastive learning (denoted as NPC) using original sequences is a special case of the proposed MPC scheme when q = 1 and x = 0 (see Eq. (2) and (3)). The MPC scheme can be viewed as to perform finer prototype learning with different subsequences, and allow the model to jointly attend to key skeleton patterns from different representation subspaces of the original sequences, which encourages capturing more discriminative skeleton features for person re-ID (see Sec. 5). The objective of MPC can be theoretically formulated in the form of Expectation-Maximization (EM) algorithms. We prove the effectiveness of MPC and show its relations to existing contrastive losses in Appendix A.

3.2 Masked Intra-Sequence Contrastive Learning

The continuity of human motion typically results in very little variation of poses/skeletons within a small temporal interval [Rao et al., 2021b]. Due to this nature, subsequences of the same skeleton sequence usually possess strong inherent correlations. For example, they could locally share similar skeletons and partial sequences with consistent walking patterns. To exploit such intra-sequence relationships and inherent consistency (e.g., pattern invariance) within sequences to learn better skeleton representations, we propose the masked intra-sequence contrastive learning (MIC) below.

Given two skeleton instances (i.e., subsequence representations), v(i) and v(j), of the same sequence, we first map them into a contrasting space RH with a fully-connected (FC) layer Fc(·) by: Fc (v(i) ) =z(i) and Fc (v(j) ) =z(j), where z(i), z(j) ∈ RH . Inspired by [Chen and He, 2021], we leverage a Siamese architecture to contrast one instance in the original feature space with the other one in the new contrasting space, so as to symmetrically learn their inherent similarity. To this end, we exploit a masked intra-sequence contrastive learning (MIC) loss to minimize the negative cosine similarity between two instances of the same sequence by:

where ∥ · ∥2 denotes ℓ2-norm, α and β are weights for contrastive learning of representation pairs (z(i), v(j)) and (z(j), v(i)), respectively. Here LMIC is defined for two subsequence representations of a skeleton sequence and the total loss is averaged over all sequences. To enable more stable and better contrastive learning, we employ a symmetrized loss with equal weights for two contrastive representation pairs, i.e., α = β = 0.5, and adopt an alternating stop-gradient operation following [Chen and He, 2021] when contrasting each pair, as shown in Fig. 2 (Note that we only visualize one contrastive pair for conciseness). We provide hypotheses and proof for the effectiveness of MIC in Appendix A.

3.3 The Entire Framework

The proposed SimMC framework combines both MPC loss (see Eq. (5)) and MIC loss (see Eq. (6)) to perform unsupervised contrastive learning of skeleton representations with:

where λ is the weight coefficient to trade off the importance of different contrastive learning. For convenience, here we use LMIC to denote the total MIC loss averaging over all training skeleton sequences. To facilitate training and generate more reliable clusters, we optimize our model by alternating clustering and contrastive representation learning. For the person re-ID task, we exploit the encoder ψ(·) learned by our framework to encode each skeleton sequence of the probe set ΦP into corresponding representations, {vPi }N2 i=1 , which are matched with the representations, {vGj }N3 j=1 , of the same identity in the gallery set ΦG based on the Euclidean distance.

4 Experiments

4.1 Experimental Settings

Datasets. We evaluate our framework on four person reID benchmark datasets with 3D skeleton data, namely IASLab [Munaro et al., 2014b], KS20 [Nambiar et al., 2017], BIWI [Munaro et al., 2014a], KGBD [Andersson and Araujo, 2015], and a large-scale multi-view gait dataset CASIA-B [Yu et al., 2006], which contain 11, 20, 50, 164, and 124 different individuals, respectively. For BIWI and IAS-Lab, we set each testing set as the gallery and the other one as the probe. For KS20, we randomly take one skeleton sequence from each view as the probe sequence and use one half of the remaining sequences for training and the other half as the gallery. For KGBD, we randomly choose one skeleton video of each individual as the probe set, and equally divide the remaining videos into the training set and gallery set. In CASIA-B, all testing sequences are grouped by three conditions (Normal, Bags, Clothes), and we evaluate our framework with singlecondition and cross-condition settings following [Liu et al., 2015]. We repeat experiments with each setup for multiple times and report the average performance.

Implementation Details. We set sequence length f to 6 on IAS-Lab, KS20, BIWI, and KGBD datasets for a fair comparison with existing methods, and empirically employ x = 2 random masks for subsequence sampling. For the largest dataset CASIA-B with roughly estimated skeleton data from RGB videos, we set f = 40 with x = 10 random masks. The number of random subsequence sampling is q = 2 and the embedding size for skeleton representations is H = 256 for all datasets. We empirically set the temperature τ = 0.06 (KGBD), τ = 0.07 (BIWI), τ = 0.075 (CASIA-B), τ = 0.08 (KS20, IAS-Lab) for MPC learning, and adopt the weight coefficient λ = 0.5 for KS20, KGBD, and IAS-B, λ = 0.75

for IAS-A, and λ = 0.25 for BIWI and CASIA-B. We employ Adam optimizer with learning rate 0.00035 and batch size 256 for all datasets. To perform unsupervised fine-tuning with SimMC, we train SimMC on the unlabeled skeleton representations pre-trained by original models, and exploit the skeleton representations learned by SimMC for person re-ID. More implementation details are provided in Appendix B.

Evaluation Metrics. We compute Cumulative Matching Characteristics (CMC) curve and adopt top-1/top-5/top-10 accuracy and Mean Average Precision (mAP) [Zheng et al., 2015] to quantitatively evaluate person re-ID performance.

4.2 Comparison with State-of-the-Arts

We compare our framework with existing state-of-the-art self-supervised and unsupervised skeleton-based methods on KS20, KGBD, IAS-Lab, and BIWI in Table 1 and 2. We also include the latest supervised skeleton-based methods and representative hand-crafted methods as a performance reference.

Comparison with Self-supervised and Unsupervised Methods. Our framework shows evident advantages in terms of performance and efficiency over existing state-ofthe-art self-supervised and unsupervised methods. As reported in Table 1 and 2, SimMC significantly outperforms AGE [Rao et al., 2020] and SM-SGE [Rao et al., 2021a] that manually design pretext tasks based on pre-defined skeleton modeling such as skeleton graphs by a large margin of 7.452.0% top-1 accuracy and 2.2-13.4% mAP on all datasets. Compared with the SGELA model [Rao et al., 2021b] using direct inter-sequence contrastive learning, our framework achieves remarkably better performance on five out of six testing sets (KS20, KGBD, IAS-A, IAS-B, BIWI-W) by up to 28.1% top-1 accuracy and 8.9% mAP, which demonstrates that the proposed SimMC combining both prototype (MPC) and intra-sequence contrastive learning (MIC) can capture more discriminative features within skeleton sequences for person re-ID on different datasets. Notably, SimMC also enjoys the smallest model size (only 0.15M) for skeleton representation learning among all approaches shown in Table 1, which suggests its higher model efficiency for person re-ID tasks.

By applying the proposed framework to fine-tuning SGELA and SM-SGE models, we can further improve their performance with an average gain of 16.9% and 8.1% top-1 accuracy respectively on all datasets. Such results demonstrate both effectiveness and scalability of proposed masked contrastive learning, which is compatible with existing models and can fully exploit their pre-trained features to achieve higher-quality skeleton representations for person re-ID.

Comparison with Hand-crafted and Supervised Methods.

In contrast to hand-crafted methods (D13 and D16) that rely on geometric joint distances and anthropometric descriptors, our approach obtains similar performance on IAS testing sets, while it achieves a distinct improvement of 7.5-37.9% top1 accuracy on BIWI, KS20, and KGBD datasets that contain more views and individuals. Despite utilizing unlabeled skeleton data as the sole input, the proposed SimMC still performs better than the latest supervised models PoseGait and MG-SCR in most cases. Interestingly, applying SimMC to SM-SGE achieves significantly higher performance gains than direct supervised fine-tuing (DF) in terms of top-1 accuracy (3.9-17.4%), top-5 accuracy (0.6-6.7%), top-10 accuracy (0.3-5.0%), and mAP (3.3-19.2%) on all datasets. With highly efficient performance and strong scalability, the proposed unsupervised SimMC can be a more general framework for skeleton-based person re-ID and related tasks.

5 Further Analysis

Application to Model-estimated Skeletons. To verify the effectiveness of SimMC when applied to RGB-based scenes with model-estimated 3D skeletons, we utilize pre-trained pose estimation models to extract skeleton data from RGB videos of CASIA-B, and compare the performance of SimMC with representative appearance-based and skeleton-based methods. As shown in Table 3, the proposed SimMC remarkably outperforms state-of-the-art skeleton-based models SMSGE and SGELA by a distinct margin of 5.6% to 42.5% top1 accuracy and 0.4% to 8.6% mAP in different conditions, which suggests the stronger ability of our framework on capturing discriminative features from estimated skeleton data for person re-ID. Compared with appearance-based ELF and MLRmodels that utilize visual features (e.g., colors, textures, and silhouettes) with extra label information, the skeletonbased SimMC also achieves superior performance in all conditions of CASIA-B, which demonstrates its great applicable value and potential for person re-ID under large-scale RGBbased scenarios and more general settings.

Ablation Study. We conduct ablation study to demonstrate the contribution of each component in our framework. We adopt 3D coordinates of raw skeleton sequences as the baseline representation for person re-ID. As reported in Table 4, the model exploiting NPC significantly outperforms the baseline by 9.8-47.8% top-1 accuracy and 2.0-11.0% mAP. Considering that NPC is a special case of the proposed MPC scheme (see Sec. 3.1), such results verify the effectiveness of the skeleton prototype contrastive learning in MPC, which can capture highly discriminative features within unlabeled skeleton sequences for the task of person re-ID. Employing the standard MPC scheme with randomly sampled subsequences consistently improves the model performance by up to 3.9% top-1 accuracy and 1.2% mAP on all datasets, which demonstrates that MPC is able to mine more representative key features from skeleton subsequences to perform person re-ID. Finally, incorporating MIC into MPC further improves model performance with 0.8-2.5% top-1 accuracy and 0.2%- 1.2%mAP gains on different datasets. This justifies our claim that capturing inherent intra-sequence similarity and pattern consistency within sequences could facilitate learning better representations of skeleton sequences for person re-ID.

Discussions. As shown in Fig. 3, we conduct a t-SNE visualization [Van der Maaten and Hinton, 2008] of representations. The skeleton representations learned by our framework are clustered with higher inter-class separation than AGE and SM-SGE, which suggests that SimMC may learn richer classrelated semantics and lower-entropy skeleton representations. We also show effects of different parameters on SimMC in Fig. 4, which indicates that the use of randommasks (x > 0) is the key to the proposed masked contrastive learning, regardless of adopting unified or non-unified mask numbers, while an appropriate fusion (λ > 0) of MIC and MPC facilitates better skeleton representation learning for person re-ID. Our framework with the optimal parameter setting is not sensitive to changes of some parameters such as temperatures τ . More results and proof are provided in the appendices.

6 Conclusion

In this paper, we propose a simple masked contrastive learning (SimMC) framework to efficiently learn representations of unlabeled skeleton sequences for unsupervised person reID. A novel masked prototype contrastive learning (MPC) scheme is devised to cluster the most typical skeleton features of subsequences randomly masked from original sequences, so as to contrast their inherent similarity to learn a discriminative skeleton representation from unlabeled skeletons. To fully exploit inherent relationships between subsequences, we propose a masked intra-sequence contrastive learning (MIC) to learn their similarity and pattern consistency within the sequence for more effective skeleton representations. Our framework outperforms existing state-of-theart skeleton-based methods and also enjoys high scalability and efficiency to be applied to different models and scenes.

Ethics Statement

Person re-ID as an important emerging research topic possesses great value for both academia and industry. However, illegal or improper use of person re-ID technologies could pose serious threat to the public privacy and society security. Therefore, it should be noted that all datasets used in our experiments are officially shared by reliable public (IAS-Lab, BIWI, KGBD) or private research agency (KS20, CASIAB), which have guaranteed that the collecting, processing, releasing, and using of all data are with the consent of participated subjects. For the protection of privacy, all individuals are anonymized with simple identity numbers. Our models and codes must only be used for the purpose of research.

Acknowledgements

This work was supported by Alibaba Group through Alibaba Innovative Research (AIR) Program and Alibaba-NTU Singapore Joint Research Institute (JRI), Nanyang Technological University, Singapore.