Semantics-Aware Deep Correspondence Structure Learning for Robust Person Re-Identification

1 Introduction

Person re-identification (Re-ID) is a task of matching persons captured from non-overlapping camera views, which has important applications in video surveillance systems including tracking in the camera network, human retrieval and threat detection.

In computer vision, the task of person Re-ID focuses on constructing a robust feature space for describing the person in the image, so as to minimize the intra-personal variance while maximizing the inter-personal margin. Many existing efforts employ low-level appearance feature representations, including color, texture, shape, to describe the appearance of each person. However, there exist a variety of appearance which lead to instability of these features. One of the major challenging problems is the misalignment caused by viewpoint changes. For example, in Figure 1, the yellow dashed bounding boxes of the same person captured by two camera views appear to be quite different since they respectively show the front and back views. Such variations can enlarge the intra-personal difference which the traditional feature representations fail to cope with.

⇤Corresponding author

Inspired by the perception of humans, the images of persons can be composed of a set of latent semantic units (e.g., head, front and back upper body, belongings), and the spatial configurations of each person are intrinsic even if the viewpoint changes. To identify the correct person given the probe image, we first want to recognize these semantic components according to the inherent structure representations and correspond them to those of the images from other camera views, respectively. Based on such intuition, we are able to analyse the person by a semantic-based perspective, which is robust to misalignment and variations.

With such structure representations, the remaining task of person re-identification is to match these components of the image pair across camera views. Many methods focus on building the correspondence structure in pursuit of establishing spatial correlations between images, which can be achieved by one-to-one matching strategy or a pre-defined matching function [Li et al., 2014; Ahmed et al., 2015; Shen et al., 2015]. However, such matching strategies may either be of low computing efficiency or have limitations by ignoring the spatial structures. From our point of view, matching is supposed to be a data-driven approach that proper matching function will automatically select the critical components and their feature representations of each image and compare them respectively while ignoring the misaligned or missing components (e.g., the front and back of body in Figure 1).

Thus the robust person Re-ID mainly focuses on the following two aspects: (1) Find the semantic components to better describe the appearance and the spatial structure of each person. (2) Match the representations of the image pair using a learned function to adaptively discover the correspondence between the images. Based on the aforementioned observations and motivations, we propose a novel and unified approach called Deep Correspondence Structure Learning (DCSL) as shown in Figure 2 to deal with the challenges in person Re-ID. In our approach, we try to capture the intrinsic structural information of persons by learning a semanticsaware image representation, and then we match each component of the structure using a learned hierarchical matching strategy. We unify our approach into an end-to-end deep learning framework to learn feature representation without any heuristic motivations. In the experiments, we employ mixture models for cross-domain person Re-ID learning to further enhance the robustness of the person Re-ID system for cross-dataset evaluation.

To summarize, we have several contributions as follows.

• ÷ We propose an end-to-end deep learning framework to deal with the challenges in person Re-ID task. The proposed approach enables the feature representation of an image to keep stable under lots of variations. Furthermore, it also seeks to adaptively learn a data-driven feature matching function with efficient hierarchical matching strategy. The learned image and correspondence representations are robust to be adapted to other camera settings or scenarios.
• ÷ The proposed DCSL approach employs a unified framework to learn the semantics-aware image representation and adaptive correspondence representation jointly by a data-driven approach. The framework enables both representations to be regularized from the person Re-ID task to find the most discriminative representations.

2 RelatedWork

In the proposed work, we mainly aim at establishing semantics-aware correspondence between images captured from non-overlapping cameras. In the literature, previous efforts of person Re-ID mainly focus on the following two points: feature extraction for image representation and image matching based on the extracted features.

Early papers pay attention to effective feature representations for better modelling the person in the image. People employ hand-crafted features for detecting determinative information including HSV color histogram [Farenzena et al., 2010], SIFT [Zhao et al., 2013], LBP features [Li and Wang,

2013] and the combination of them. Unlike hand-crafted features, the methods based on deep learning [Li et al., 2014; Zhao et al., 2014a] learn feature representation directly from tasks which have shown significant improvement compared with traditional features .

With the extracted features, many existing efforts are devoted to handling the overall appearance variations between the given images [Paisitkriangkrai et al., 2015; Hirzer et al., 2012]. In order to learn the spatial relationship between images from different views, people utilize locationrelated matching strategies to find higher-level correlation between the images. Among these efforts, patch-based matching methods [Li et al., 2014; Shen et al., 2015; Ahmed et al., 2015] decompose images into patches and perform patchwise matching strategy to find the spatial relationship. To reduce the computing cost due to the comparisons among the patch compositions, some saliency based methods are proposed to guide spatial matching methods [Zhao et al., 2014b]. Both patch-based and saliency-based methods capture the appearance correlations such as color or texture. Compositional approaches [Xu et al., 2013] first localize the body parts and search for part-to-part correspondence between reference samples and observations, which have promising results on the challenging datasets. These methods depend on the performance of body parser, which are limited in some specific scenes.

3 Our Approach

Before describing this work, we first introduce a number of notations used hereinafter. For notational convenience, we let an uppercase boldfaced letter denote a feature map blob, and a lowercase boldfaced letter denote a feature vector. Specifically, for a given feature map blob X, xij is associated with the feature vector at the (i↪ j)-th location of X, while xijk stands for the k-th element of xij. Namely, X is generated from a 2D spatial stack of xij along the two spatial dimensions of X. To distinguish feature maps or vectors from different views, we employ the superscript letters for notational representations (e.g., XA and XB respectively for camera views A and B). Moreover, a general function is written in the form of f(X1↪X2↪ ▷▷▷; ✓) with ¶Xi♦ and ✓ respectively being the input variables and the parameters to be learned.

3.1 ProblemFormulation

In the beginning, we collect the person Re-ID dataset ¶(Xn↪ ln)↪ n = 1↪ ▷▷▷↪N♦, where Xn = (IA↪ IB )n denotes the paired images of the n-th sample and ln 2 ¶0↪ 1♦ denotes its label, and here ln = 1 indicates that the pair belongs to the same person. In our approach, the task of person Re-ID is to learn a classifier s(Xn; ✓) parameterized by ✓ to distinguish whether the image pair Xn represents the same person or not by measuring the similarities between them.

In order to perform the robust similarity measurement, the proposed approach involves two factors: (1) image representation of each image by a semantics-aware feature extraction approach, (2) image matching based on the representations of the image pair to adaptively discover the correspondence of them and measure the similarity guided by the correspondence.

3.2 Semantics-aware Image Representation

In person Re-ID, image representation is confronted with a number of challenges, such as occlusion and viewpoint variations. As discussed previously, there exists inherent semantic structure in an image, and such structure is composed of intrinsic latent components such as upper body and head, which are robust to the variations of views and background. Based on the semantic representations of the image, we will be able to capture the correlations of the image pair.

To effectively encode these components in an adaptive manner, we adopt convolutional neural networks (CNNs) to model and abstract the multi-scale semantic information for a hierarchical representation parameterized by ✓1 to represent the image:

As observed in [Zeiler and Fergus, 2014], these representations contain both low-level visual information and high-level semantic components of the image. Without loss of generality, we decompose the representations to Y = ¶V↪C♦, where V denotes the low-level representations and C stands for the representation of semantic components. The representation of each component is associated with a subset of feature maps in C, and here we assign E for representing one specific component.

3.3 Image Matching via Correspondence Structure Learning

Given paired images from different camera views, we extract their corresponding feature representations by Eq. (1), written as YA = ¶VA↪CA♦ and YB = ¶VB ↪CB ♦. With YA and YB , the problem of image matching is converted to the task of how to correspond the components by their representation maps and to perform the similarity measurement based on the correspondence representation. First we will discuss the correspondence structure learning of the specific component representations EA and EB .

Correspondence structure learning based on EA and EB . We first define the correspondence structure based on the semantic components of the person to cope with the misalignment in person Re-ID.

In our approach the correspondence structure EA ! EB encodes the spatial correspondence distributions between the specific component of the image pair.

Many methods adopt a discrete distribution as the correspondence structure [Shen et al., 2015], which is a set of pair-wise matching probabilities: ¶P(eAij ↪EB )♦, where P(eAij ↪EB ) describes the correspondence distribution in EB for the feature vector eAij in EA. The distribution P(eAij ↪ EB ) can be obtained through different strategies. For example, (1) compute an optimal assignment matrix from eAij to every vector ¶eBst ♦ in EB by dense matching; (2) conduct local search in a small neighbourhood of the given location. For (1), one-to-one location-specific matching strategy searches in the whole space of possible solutions, resulting in low computational efficiency. Moreover, it may induce additional matching noise since it lacks the capability of modelling the spatial neighbourhood matching consistency of each image; for (2), local spatial neighbourhood matching is often trapped in local correspondence optimums as a result of heuristically predetermined searching ranges.

To deal with these difficulties, we construct the correspondence structure from a different perspective, which aims to adaptively capture the correspondence relationships between the multi-scale semantic component representations in a hierarchical matching fashion. As shown in Figure 3, the correspondence structure is discovered through a pyramid-like matching strategy. We generate the correspondence maps between the feature maps of each image. The value of each location (i↪ j) indicates the correspondence probability at that location. To deal with misalignment, we further down-sample the feature maps by max-pooling so as to preserve the most discriminative information of the component and align it in a larger region.

In this case, we denote the set of multi-level feature maps as ¶EA↪t ♦ and ¶EB↪t ♦ generated from EA and EB by maxpooling using the stride of 2:

where kp denotes the kernel size in the pooling stage, and t is the level of max-pooling.

The final correspondence representation can be defined as a set of hierarchical correspondence maps ¶Qt ♦, where Qt denotes the correspondence distribution at level t. Inspired by [Han et al., 2015], we calculate it by learning the interactions between all the feature vectors eA↪t ij and eB↪t ij using the learned weights ✓t2 :

where [EA↪t ↪EB↪t ] is the concatenation of the two feature maps, and the set ¶Q♦t will be augmented by introducing more semantic-level components from CA and CB .

To summarize, the correspondence structure learning by pooling strategy has two advantages in learning context similarities: 1) it effectively improves the cross-view matching robustness by using a hierarchical multi-stage matching strategy; 2) it makes full use of multi-scale spatial matching consistency by multi-level max pooling. The above two parts are modelled in a totally data-driven learning scheme, resulting in the flexibility in practice.

Similarity measurement based on VA and VB . Before matching two images by their correspondence representations, we have to first measure the similarity given the lowlevel feature vectors of the selected pairs. We first construct the hierarchical feature representations ¶VA↪t ♦ and ¶VB↪t ♦, which are built fromVA↪t and VB↪t in the same way as that of ¶EA↪t ♦ and ¶EB↪t ♦. Then the similarity representations ¶St ♦ upon the feature representations learn the interactions between the vectors by the weights ✓t3 :

Correlation of the images based on ¶Qt ♦ and ¶St ♦. Correlation of the paired images is characterized by evaluating the similarities of the mutually corresponding semantic components after correspondence structure learning. It is formulated as a multi-level matching function with ✓4:

3.4 Unified Correspondence Structure Learning

In principle, image representations and correspondence structure learning are two correlated and complementary problems. Namely, better semantic representations contribute to guiding correspondence network to find more meaningful structures. Meanwhile, better correspondence structures regularize feature representation network to learn more effective features for matching.

Thus we develop a unified end-to-end data-driven framework, where the feature, correspondence and metric representations are learned jointly and adaptively in a supervised setting using the structure illustrated in Figure 2. We train the network by minimizing the cross-entropy error:

over a training set of N pairs using stochastic gradient descent. ln is the 1/0 label for the input pair Xn while ˆln and (1-ˆln) is the Softmax activations computed on the two output nodes of the similarity function s(Xn; ✓), namely s0(Xn; ✓) and s1(Xn; ✓):

The proposed approach is finally simplified to the unified framework by synthesizing Eq. (1) for semantics-aware image representation and Eqs. (3)-(5) for correspondence-based image matching, which is defined as the deep correspondence network (DCN) with ✓ = ¶✓1↪ ¶✓t2 ♦↪ ¶✓t3 ♦↪ ✓4♦:

In summary, the deep correspondence structure learning is performed to capture the semantics-aware components automatically from the person Re-ID task. We employ convolutional neural networks for robust semantic representations. The concatenation of feature maps captures correspondence between images from different camera views and also enables part-wise matching by the correspondence structures, together with the efficient hierarchical alignment strategy by max-pooling operations. The models of the two tasks are learned jointly by combining the appearance and semantic representation with the correspondence structures to find the optimal representation of the input image pair to be matched.

4 Experimental Results

We have implemented our architecture using Caffe [Jia et al., 2014], which is a widely used deep learning framework in recent years. We use an NVIDIA TITAN X GPU for training the networks. It takes around 30 hours for training one DCSL model and 1.7ms for computing the similarity given a pair of images.

In this section, we compare our approach with the stateof-the-art approaches on several datasets. To avoid accidental results, experiments are conducted on the datasets with 10 random training and testing splits. We evaluate all the approaches with Cumulative Matching Characteristics (CMC) curves by single-shot results. The CMC curve characterize a ranking result for every image in the gallery given the probe image. We first report results of our proposed model on three challenging datasets with the same evaluation criteria. Finally, we report cross domain transfer learning on the trained networks with cross dataset evaluation.

4.1 Dataset

We evaluate our method on three public datasets, namely CUHK03(labelled) [Li et al., 2014], CUHK01 [Li et al., 2012], VIPeR [Gray et al., 2007]. Table 1 shows the description of each dataset and our experimental settings with the training and testing splits.

Among the three datasets, CUHK03 is a relatively large dataset, which can cover a number of variations to construct the spatial configurations and also contain several challenging situations including illumination changes, cross-view deformations, occlusions, etc.

In addition, we also conduct cross-dataset experiments to evaluate the performance on CUHK01 and VIPeR using the trained model on CUHK03 and Market-1501 [Zheng et al., 2015].

4.2 Training the Network

In this section, we use pre-trained AlexNet [Krizhevsky et al., 2012] and GoogLeNet [Szegedy et al., 2015] models as our deep architecture for extracting the feature maps as shown in Figure 2. The parameters of these parts are initialized with the model trained on ImageNet-1K [Russakovsky et al., 2014]. In our experiments, we use "norm2" maps of AlexNet and "inception 5b/output" maps of GoogLeNet to represent semantics-aware information of each image. To improve the ability of spatial representations, we remove "pool4/3x3 s2" layer of GoogLeNet. As a result, given the input shape of 160 ⇥ 80 ⇥ 3, the shapes of feature maps which we obtained are 19⇥9⇥256 in AlexNet and 10⇥5⇥1024 in GoogLeNet. These feature maps are directly fed to the concatenation layer to learn the correspondence of the image pair.

In our method, we use stochastic gradient descent for updating the weights of the network. Training pairs are divided into mini-batches. For training efficiency, we set the base learning rate as ⌘(0) = 0▷01 and use a polynomial decay for learning rate to train around 100,000 batches of size 100. We use a momentum of θ = 0▷9 and weight decay λ = 0▷0002.

Data Augmentation. To generate more training pairs, we also apply the data augmentation strategy for training the neural network. We randomly perform affine transformation over the shape of the original images and thus generate 5 augmented images per image for training.

Hard Negative Mining. There still exist some scenarios that are hard to distinguish, which require more negative samples for training. To increase the efficiency of model training, we use the trained model to classify all the negative pairs and choose the top ranked ones for retraining our network.

Mixture of the DCSL networks. To deal with the dataset biases when transferring the trained model to real-world applications, We employ a mixture of DCSL networks to represent the domains of different distributions inspired by [Ge et al., 2015]. First we denote βk(Xn; ✓e) as the probability of the image pair Xn belonging to the k-th component which can be calculated using parameters ✓e. The output score of the image pair is finally composed by all the K domain components using the weights ¶βk♦Kk=1 : s(Xn; ✓) = PKk=1 βks(k)(Xn; ✓(k))) where s(k)(Xn; ✓(k)) is the output score of the k-th DCSL model .

4.3 Experimental Results

We compare our deep correspondence structure learning (DCSL) method with several methods in recent years, including KISSME [Koestinger et al., 2012], FPNN [Li et al., 2014], ImprovedDL [Ahmed et al., 2015], LDM [Guillaumin et al., 2009], eSDC [Zhao et al., 2013], LMNN [Weinberger et al., 2005], mFilter [Zhao et al., 2014a], SalMatch and PatMatch [Zhao et al., 2013]. Figure 4 and Table 2 illustrates the recognition rate of these methods. Figure 1 shows a few qualitative results.

Comparisons on CUHK03 and CUHK01. From Figure 4 and Table 2, we see that DCSL outperforms the state-of-theart methods by more than 20% on the performance of rank-1 accuracy (80.20% vs. 54.74%, 89.60% vs. 65.00%, 76.54% vs. 47.53%). Deep learning based methods outperform the traditional methods by its outstanding ability of data-driven feature learning. Note that the rank-5 recognition rate of DCSL reaches around 95%, meaning that the trained model has high probability of finding the correct person from other cameras given more than 100 candidate images.

Comparisons on VIPeR. The last column of Table 2 illustrates that DCSL is able to compete with the state-ofthe-art methods even though the training samples are limited for deep neural networks. We have a relative better performance than deep learning based method ImprovedDL (44.62% vs. 34.81%), which employs the pre-trained model on the CUHK03 and CUHK01 dataset.

Effect of different feature extraction models. We both try AlexNet (AN) and GoogLeNet (GN) models for feature extraction with the same correspondence networks. Figure 4(a) illustrates that the performance of GoogLeNet exceeds that of AlexNet due to its outstanding ability to abstract the image features.

Effect of hard negative mining. Hard negative mining also has positive impact on the performance of our DCSL method by mining more training pairs for further tuning the model. We see the absolute gain over 1% compared with the same model without hard negative mining, as shown in Figure

4(a)(b).

Results of cross-dataset evaluations. We both use the single DCSL model (1-DCSL) and a mixture of DCSL models (m-DCSL) for the cross-dataset evaluations. Table 3 illustrates that our proposed DCSL model is robust to be generalized to other target domains. With the mixture models of DCSL, we are able to deploy our proposed method to the realworld applications. Note that the rank-1 recognition rate of CUHK01 using the model trained by CUHK03 and Market1501 can reach the score of 69.14%, which outperforms most of the state-of-the-art methods that conduct training on the CUHK01.

5 Conclusion

In this paper, we propose and evaluate a unified approach for person Re-ID that jointly learns a deep convolutional neural network for representing the semantics-aware information of the image as well as a network for robust image matching based on correspondence structure learning. Our model outperforms the state-of-the-art approaches by a large margin and has a considerable computing efficiency. This work demonstrates that unified deep neural networks can be effective for general person Re-ID and can be easily generalized through the mixture of the knowledge from different domains.

Acknowledgments

This work was supported in part by the National Basic Research Program of China under Grant 2012CB316400 and Grant 2015CB352300, National Natural Science Foundation of China under Grant U1509206 and Grant 61472353, Zhejiang Provincial Engineering Center on Media Data Cloud Processing and Analysis, Fundamental Research Funds for the Central Universities, and equipment donation by Nvidia.