Person Re-Identification by Deep Joint Learning of Multi-Loss Classification

1 Introduction

Person re-identification (re-id) is about matching identity classes in detected person bounding box images from nonoverlapping camera views over distributed open spaces. This is an inherently challenging task because person visual appearance may change dramatically in different camera views due to unknown changes in illumination, occlusion, and background clutter [Gong et al., 2014]. Existing re-id studies typically focus on either feature representation [Gray and Tao, 2008; Farenzena et al., 2010; Kviatkovsky et al., 2013; Zhao et al., 2013; Liao et al., 2015; Matsukawa et al., 2016a; Ma et al., 2017] or matching distance metrics [Koestinger et al., 2012; Xiong et al., 2014; Zheng et al., 2013; Paisitkriangkrai et al., 2015; Zhang et al., 2016; Wang et al., 2014b; 2016b; 2016c; Chen et al., 2017b] or their combination in deep learning framework [Li et al., 2014; Ahmed et al., 2015; Wang et al., 2016a; Xiao et al., 2016; Subramaniam et al., 2016; Chen et al., 2017a]. Regardless, the overall objective is to obtain a view- and location-invariant representation. We consider that learning any matching distance metric is intrinsically learning a global feature transformation across domains (two disjoint cameras) thus obtaining a "normalised" representation for matching.

Most re-id features are typically hand-crafted to encode local topological and/or spatial structural information, by different image decomposition schemes such as horizontal stripes [Gray and Tao, 2008; Kviatkovsky et al., 2013], body parts [Farenzena et al., 2010], and patches [Zhao et al., 2013; Matsukawa et al., 2016a; Liao et al., 2015]. These localised features are effective for mitigating the person pose and detection misalignment in re-id matching. More recent deep re-id models [Xiao et al., 2016; Wang et al., 2016a; Chen et al., 2017a; Ahmed et al., 2015] benefit from the availability of larger scale datasets such as CUHK03 [Li et al., 2014] and Market-1501 [Zheng et al., 2015] and from lessons learned on other vision tasks [Krizhevsky et al., 2012; Girshick et al., 2014]. In contrast to local hand-crafted features, deep models, in particular Convolutional Neural Networks (CNN) [LeCun et al., 1998], favour intrinsically in learning global feature representations with a few exceptions. They have been shown to be effective for re-id.

We consider that either local or global feature learning alone is suboptimal. This is motivated by the human visual system that leverages both global (contextual) and local (saliency) information concurrently [Navon, 1977; Torralba et al., 2006]. This intuition for joint learning aims to extract correlated complementary information in different context whilst satisfying the same learning constraint therefore achieving more reliable recognition. To that end, we need to address a number of non-trivial problems: (i) the model learning behaviour in satisfying the same label constraint may be different at the local and global levels; (ii) any complementary correlation between local and global features is unknown and may vary among individual instances, therefore must be learned and optimised consistently across data; (iii) people's appearance in public scenes is diverse in both pattens and configurations. This makes it challenging to learn correlations between local and global features for all appearances.

This work aims to formulate a deep learning model for jointly optimising local and global feature selections concurrently and to improve person re-id using only generic matching metrics such as the L2 distance. We explore a deep learning approach for its potential superiority in learning from large scale data [Xiao et al., 2016; Chen et al., 2017a]. For the bounding box image based person re-id, we consider the entire person in the image as a global scene context and body parts of the person as local information sources, both are subject to the surrounding background clutter within an image, and potentially also misalignment and partial occlusion from poor detection. In this setting, we wish to discover and optimise jointly correlated complementary feature selections in the local and global representations, both subject to the same label constraint concurrently. Whilst the former aims to address detection misalignment and occlusion by localised fine-grained saliency information, the latter exploits holistic coarse-grained context for more robust global matching.

Our contributions are: (I) We propose the idea of learning concurrently both local and global feature selections for optimising feature discriminative capabilities in different context whilst performing the same person re-id tasks. This is currently under-studied in the person re-id literature to our best knowledge. (II) We formulate a novel Joint Learning Multi-Loss (JLML) CNN model for not only learning both global and local discriminative features in different context by optimising multiple classification losses on the same person label information concurrently, but also utilising their complementary advantages jointly in coping with local misalignment and optimising holistic matching criteria for person re-id. This is achieved with a deep two-branch CNN architecture by imposing inter-branch interaction between the local and global branches, and enforcing a separate learning objective loss function to each branch for learning independent discriminative capabilities. (III) We introduce a structured sparsity based feature selection learning mechanism for improving multi-loss joint feature learning robustness w.r.t. noise and data covariance between local and global representations. Extensive evaluations demonstrate the superiority of the proposed JLML model over a wide range of existing stateof-the-art re-id models on four benchmark datasets.

Related Works. The JLML method is related to the saliency learning based models [Zhao et al., 2013; Wang et al., 2014a] in terms of modelling localised part importance. However, these existing methods consider only the patch appearance statistics within individual locations but no global feature representation learning, let alone the correlation and complementary information discovery between local and global features as modelled by the JLML. Whilst the more recent SCS [Chen et al., 2016] and MCP [Cheng et al., 2016] consider both levels of representation, the JLML model differs significantly from them: (i) The SCS method focuses on supervised metric learning, whilst the JLML aims at joint discriminative feature learning and needs only generic metrics for reid matching. (ii) The local and global branches of the MCP model are supervised and optimised by a triplet ranking loss, in contrast to the proposed multiple classification loss design. (iii) The JLML is uniquely capable of performing structured feature sparsity regularisation.

2 Model Design

2.1 ProblemDefinition

We assume a set of n training images I = {Ii}ni=1 with the corresponding identity labels as Y = {yi}ni=1 . These training images capture the visual appearance of nid (where yi ∈ [1, · · · , nid]) different people under non-overlapping camera views. We formulate a Joint Learning Multi-Loss

(JLML) CNN model that aims to discover and capture concurrently complementary discriminative information about a person image from both local and global visual features of the image in order to optimise person re-id under significant viewing condition changes across locations. This is in contrast to most existing re-id methods typically depending only on either local or global features alone.

2.2 Joint LearningMulti-Loss

The overall design of the proposed JLML model is depicted in Figure 1. This JLML model consists of a twobranches CNN network: (1) One local branch of m streams of an identical structure with each stream learning the most discriminative local visual features for one of m local image regions of a person bounding box image; (2) Another global branch responsible for learning the most discriminative global level features from the entire person image. For concurrently optimising per-branch discriminative feature representations and discovering correlated complementary information between local and global feature selections, a joint learning scheme that subjects both local and global branches to the same identity label supervision is formulated with two underlying principles:

(I) Shared low-level features. We construct the global and local branches on a shared lower conv layer, in particular the first conv layer, for facilitating inter-branch common learning. The intuition is that, the lower conv layers capture lowlevel features such as edges and corners which are common to all patterns in the same images. This shared learning is similar in spirit to multi-task learning [Argyriou et al., 2007], where the local and global feature learning branches are two related learning tasks. Sharing the low-level conv layer reduces the model parameter size therefore model overfitting risks. This is especially critical in learning person re-id models when labelled training data is limited.

(II) Multi-task independent learning. To maximise the learning of complementary discriminative features from local and global representations, the remaining layers of the two branches are learned independently subject to given identity labels. That is, the JLML model aims to learn concurrently multiple identity feature representations for different local image regions and the entire image, all of which aim to maximise the same identity matching both individually and collectively at the same time. Independent multi-task learning aims to preserve both local saliency in feature selection and global robustness in image representation. To that end, the JLML model is designed to perform multi-task independent learning subject to shared identity label constraints by allocating each branch with a separate objective loss function. By doing so, the per-branch learning behaviour is conditioned independently on the respective feature representation. We call this branch-wise loss formulation as the MultiLoss design.

Network Construction. We adopt the Residual CNN unit [He et al., 2016] as the JLML's building blocks due to its capacity for deeper model design whilst retaining a smaller model parameter size. Specifically, we customise the ResNet50 architecture in both layer and filter numbers and design the JLML model as a 39 layers 2-branches ResNet (JLML-ResNet39) tailored for re-id tasks. The configuration of JLML-ResNet39 is given in Table 1. Note that, the ReLU rectification non-linearity [Krizhevsky et al., 2012] after each conv layer is omitted for brevity.

Feature Selection. To optimise JLML model learning robustness against noise and diverse data source, we introduce a feature selection capability in JLML by a structure sparsity induced regularization [Kong et al., 2014; Wang et al., 2013]. Our idea is to have a competing-to-survive mechanism in feature learning that discourages irrelevant features whilst encourages discriminative features concurrently in different local and global context to maximise a shared identity matching objective. To that end, we sparsify the global feature representation with a group LASSO [Wang et al., 2013]:

where WG = [w1 g , · · · ,wdg g ] ∈ Rcg×dg is the parameter matrix of the global branch feature layer taking as input dg dimensional vectors from the previous layer and outputting cg dimensional (512-D) feature representation. Specifically, with the ℓ1 norm applied on the ℓ2 norm of wig , our aim is to learn selectively feature importance subject to both the sparsity principle and the identity label constraint simultaneously. Similarly, we also enforce a local feature sparsity constraint by an exclusive group LASSO [Kong et al., 2014]:

where WL is the parameter matrix of the local branch feature layer with m × dl and cl (512) as the input and output dimensions (m the image stripe number), and wi l,j ∈ Rdl×1 defines the parameter vector for contributing the i-th output feature dimension from the j-th local input feature vector, j ∈ [1, 2, · · · ,m]. This ℓ1,2 regularisor performs sparse feature selection for individual image regions in conjunction with the global feature selection learning.

Loss Function. For model training, we utilise the crossentropy classification loss function for both global and local branches so to optimise person identity classification given training labels of multiple person classes extracted from pairwise labelled re-id dataset. Formally, we predict the posterior probability ˜ yi of image Ii over the given identity label yi:

where xi refers to the feature vector of Ii from the corresponding branch, and wk the prediction function parameter of training identity class k. The training loss on a batch of nbs images is computed as:

Combined with the group sparsity based feature selection regularisations, we have the final loss function for the global and local branch sub-networks as:

where λglobal and λlocal control the balance between the identity label loss and the feature selection sparsity regularisation. We empirically set λlocal = λglobal = 5×10-4 .

Choice of Loss Function. Our JLML model learning deploys a classification loss function. This differs significantly from the contrastive loss functions used by most existing deep re-id methods designed to exploit pairwise re-id labels defined by both positive and negative pairs, such as the pairwise verification [Varior et al., 2016; Subramaniam et al., 2016; Ahmed et al., 2015; Li et al., 2014], triplet ranking [Cheng et al., 2016], or both [Wang et al., 2016a; Chen et al., 2017a]. Our JLML model training does not use any labelled negative pairs inherent to all person re-id training data, and we extract identity class labels from only positive pairs. The motivations for our JLML classification loss based learning are: (i) Significantly simplified training data batch construction, e.g. random sampling with no notorious tricks required, as shown by other deep classification methods [Krizhevsky et al., 2012]. This makes our JLML model more scalable in real-world applications with very large training population sizes when available and/or imbalanced training data sampling from different camera views. This also eliminates the undesirable need for carefully forming pairs and/or triplets in preparing re-id training splits, as in most existing methods, due to the inherent imbalanced negative and positive pair size distributions. (ii) Visual psychophysical findings suggest that representations optimised for classification tasks generalise well to novel categories [Edelman, 1998]. We consider that re-id tasks are about model generalisation to unseen test identity classes given training data on independent seen identity classes. Our JLML model learning exploits this general classification learning principle beyond the strict pairwise relative verification loss in existing re-id models.

Model Training. We adopt the Stochastic Gradient Descent (SGD) optimisation algorithm [Krizhevsky et al., 2012] to perform the batch-wise joint learning of local and global branches. Note that, with SGD we can naturally synchronise the optimisation processes of the two branches by constraining their learning behaviours subject to the same identity label information at each update. This is likely to avoid representation learning divergence between two branches and help enhance the correlated complementary learning capability.

2.3 Person Re-Id by Generic Distance Metrics

Once the JLML model is learned, we obtain a 1,024-D joint representation by concatenating the local (512-D) and global (512-D) feature vectors (the fc layers in Table 1). For person re-id, we deploy this 1,024-D deep feature representation using only a generic distance metric without camera-pair specific distance metric learning, e.g. the L2 distance.

3 Experiments

Datasets. For evaluation, we used four benchmarking re-id datasets, VIPeR [Gray and Tao, 2008], GRID [Loy et al., 2009], CUHK03 [Li et al., 2014], and Market-1501 [Zheng et al., 2015]. These datasets present a wide range of re-id evaluation scenarios with different population sizes under different challenging viewing conditions (Figure 2 and Table 2).

Evaluation Protocol. We adopted the standard supervised re-id setting to evaluate the proposed JLML model (Sec. 3.1). The training and test data splits and the test settings of each dataset is given in Table 2. Specifically, on VIPeR, we split randomly the whole population (632 people) into two halves: One for training (316) and another for testing (316). We repeated 10 trials of random people splits and used the averaged results. On GRID, the training/test split was 125/125 with

775 distractor people included in the test gallery. We used the benchmarking 10 people splits [Loy et al., 2009] and the averaged performance. On CUHK03, following [Li et al., 2014] we repeated 20 times of random 1260/100 training/test splits and reported the averaged accuracies under the single-shot evaluation setting. On Market-1501, we used the standard training/test split (750/751) [Zheng et al., 2015]. We used the cumulative matching characteristic (CMC) to measure re-id accuracy on all benchmarks, except on Market-1501 we also used the recall measure by mean Average Precision (mAP).

Competitors. We compared the JLML model against 10 existing state-of-the-art methods as listed in Table 3. They range from hand-crafted and deep learning features to domainspecific distance metric learning methods. We summarise them into three categories: (A) Hand-crafted (feature) with domain-specific distance learning (metric); (B) Deep learning (feature) with domain-specific deep verification (metric) learning; (C) Deep learning (feature) with generic nonlearning L2 distance (metric).

Implementation. We used the Caffe framework [Jia et al., 2014] for our JLML model implementation. We started by pre-training the JLML model on ImageNet (ILSVRC2012). Subsequently, for CUHK03 or Market, we used only their own training data for model fine-tuning, i.e. ImageNet → CUHK03/Market; For VIPeR or GRID, we pre-trained JLML on CUHK03+Market (whole datasets), and then finetuned on their respective training images, i.e. ImageNet → CUHK03+Market → VIPeR/GRID. All person images were resized to 224×224 in pixel. For local branch, according to a coarse body part layout we evenly decomposed the whole shared conv feature maps (i.e. the entire image) into four (m = 4) horizontal strip-regions. We used the same parameter settings (summarised in Table 4) for pre-training and training the JLML model on all datasets. We also adopted the stepped learning rate policy, e.g. dropping the learning rate by a factor of 10 every 100K iterations for JLML pre-training and every 20K iterations for JLML training. We utilised the L2 distance as the default matching metric.

3.1 Comparisons to State-Of-The-Arts

(I) Evaluation on CUHK03. Table 5 shows the comparisons of JLML against 8 existing methods on CUHK03. It is evident that JLML outperforms existing methods in all categories on both labelled and detected bounding boxes, surpassing the 2nd best performers DGD and X-Corr on corresponding labelled and detected images in Rank-1 by 7.9%(83.2-75.3) and 8.6%(80.6-72.0) respectively. XCorr/GOG/JLML also suffer the least from auto-detection misalignment, indicating the robustness of the joint learning approach to mining complementary local and global discriminative features.

(II) Evaluation on Market-1501. We evaluated the JLML against four existing models on Market-1501. Table 6 shows the clear performance superiority of JLML over all state-ofthe-arts with more significant Rank-1 advantages over other methods compared to CUHK03, giving 19.3%(85.1-65.8) (SQ) and 13.7%(89.7-76.0) (MQ) gains over the 2nd best SCNN. This further validates the advantages of our joint learning of multi-loss classification for optimising re-id especially when the re-id test population size increases (750 people on Market-1501 vs. 100 people on CUHK03).

(III) Evaluation on VIPeR. We evaluated the performance of JLML against 8 strong competitors on VIPeR, a more challenging test scenario with fewer training classes (316 people)

and lower image resolution. On this dataset, the best performers are hand-crafted feature methods (SCS and NFST) rather than deep models. This is in contrast to the tests on CUHK03 and Market-1501. Nevertheless, the JLML model remains the best among all deep methods with or without deep verification metric learning. This validates the superiority and robustness of our deep joint global and local representation learning of multi-loss classification given sparse training data. We attribute this property to the JLML's capability of mining complementary features in different context for both handling local misalignment and optimising global matching.

(IV) Evaluation on GRID. We compared JLML against 4 competing methods on GRID. In addition to poor image resolution, poor lighting and a small training size (125 people), GRID also has extra distractors in the testing population therefore presenting a very challenging but realistic re-id scenario. Table 8 shows a significant superiority of JLML over existing state-of-the-arts, with Rank-1 12.8%(37.5-24.7) better than the 2nd best method GOG, a 51.8% relative improvement. This demonstrates the unique and practically desirable advantage of JLML in handling more realistically challenging open-world re-id matching where large numbers of distractors are usually present.

3.2 Further Analysis and Discussions

We further examined the component effects of our JLML model on the Market-1501 dataset in the following aspects.

(I) Complementary of Global and Local Features. We evaluated the complementary effects of our jointly learned local and global features by comparing their individual reid performance against that of the joint features. Table 9 shows that: (i) Any of the two feature representations alone is competitive for re-id, e.g. the local JLML feature surpasses S-CNN (Table 6) by Rank-1 13.1%(78.9-65.8) (SQ) and 10.4%(86.4-76.0) (MQ); and by mAP 18.3%(57.8-39.5) (SQ) and 20.0%(68.4-48.4) (MQ). (ii) A further performance gain is obtained from the joint feature representation, yielding further 6.2%(85.1-78.9) (SQ) and 3.3%(89.7-86.4) (MQ) in Rank-1 increase, and 7.7%(65.5-57.8) (SQ) and 6.1%(74.5-68.4) (MQ) in mAP boost. These results show the complementary advantages of jointly learning the local and global features in different context using the JLML model. (II) Importance of Branch Independence. We evaluated the importance of branch independence by comparing our

MultiLoss design with a UniLoss design that merges two branches into a single loss [Cheng et al., 2016]. Table 10 shows that the proposed MultiLoss model significantly improves the discriminative power of global and local reid features, e.g. with Rank-1 increase of 9.0%(85.1-76.1) (SQ) and 6.0%(89.7-83.7) (MQ); and mAP improvement of 13.3%(65.5-52.2) (SQ) and 11.7%(74.5-62.8) (MQ). This shows that branch independence plays a critical role in joint learning of multi-loss classification for effective feature optimisation. One plausible reason is due to the negative effect of a single loss imposed on the learning behaviour of both branches, caused by the potential divergence in discriminative features in different context (local and global). This is shown by the significant performance degradation of both global and local features when the UniLoss model is imposed.

(III) Benefits from Shared Low-Level Features. We evaluated the effects of interaction between global and local branches introduced by the shared conv layer (common ground) by deliberately removing it and then comparing the re-id performance. Table 11 shows the benefits from jointly learning low-level features in the common conv layers, e.g. improving Rank-1 by 1.9%(85.1-83.2) / 1.4%(89.7-88.3) and mAP by 2.4%(65.5-63.1) / 2.4%(74.5-72.1) for single-/multiquery re-id. This confirms a similar finding as in multi-task learning study [Argyriou et al., 2007].

(IV) Effects of Selective Feature Learning. We evaluated the contribution of our structured sparsity based Selective Feature Learning (SFL) (Eqs. (1) and (2)). Table 12 shows that our SFL mechanism can bring additional re-id matching benefits, e.g. improving Rank-1 rate by 1.7%(85.1-83.4) (SQ) and 1.0%(89.7-88.7) (MQ); and mAP by 1.7%(65.5-63.8) (SQ) and 1.6%(74.5-72.9) (MQ).

(V) Comparisons ofModel Size andComplexity. We compared the proposed JLML-ResNet39 model with four seminal classification CNN architectures (Alexnet [Krizhevsky et al., 2012], VGG16 [Simonyan and Zisserman, 2015], GoogLeNet [Szegedy et al., 2015], and ResNet50 [He et al., 2016]) in model size and complexity. Table 13 shows that the JLML has both the 2nd smallest model size (7.2 million parameters) and the 2nd smallest FLOPs (1.54×109 ), although containing more streams (5 vs. 1 in all other CNNs) and more layers (39, more than all except ResNet50).

4 Conclusion

We presented a novel Joint Learning of Multi-Loss (JLML) CNN model (JLML-ResNet39) for person re-identification feature learning. In contrast to existing re-id approaches that employ either global or local appearance features alone, the proposed model is capable of extracting and exploiting both and maximising their correlated complementary effects by learning discriminative feature representations in different context subject to multi-loss classification objective functions in a unified framework. This is made possible by the proposed JLML-ResNet39 architecture design. Moreover, we introduce a structured sparsity based feature selective learning mechanism to further improve joint feature learning. Extensive comparative evaluations on four re-id benchmark datasets were conducted to validate the advantages of the proposed JLML model over a wide range of the stateof-the-art methods on both manually labelled and more challenging auto-detected person images. We also provided component evaluations and analysis of the model performance in order to give insights on the JLML model design.

Acknowledgements

This work was partially supported by the China Scholarship Council, Vision Semantics Ltd and Royal Society Newton Advanced Fellowship Programme (NA150459).