Attribute Aware Pooling for Pedestrian Attribute Recognition

1 Introduction

Pedestrian attribute recognition has appealed much research effort due to its continuing demands for intelligent video surveillance [Layne et al., 2012; Peng et al., 2016; Liu et al., 2017b]. It is a challenging task because of the large variance in pedestrian images, such as the viewpoint, lighting or pose changes. Thanks to the development of deep learning, pedestrian attribute recognition based on convolutional neural networks (CNNs) has made tremendous progress on the benchmark datasets, e.g. PETA [Deng et al., 2014], RAP [Li et al., 2016], and PA-100K [Liu et al., 2017b]. However, existing methods still have a long way to the practical application where the scenario is pretty complex.

∗We thank anonymous reviewers for their helpful comments. Chang Xu was supported by the Australian Research Council under Project DE180101438.

Extraordinary capability of CNN to accomplish the singlelabel image classification task has been comprehensively validated [He et al., 2016; Ren et al., 2015; Wang et al., 2016]. An example in the ImageNet Dataset [Russakovsky et al., 2015] has only one label, such as "dog", "cat" or "car". However, pedestrian often contains multiple attributes, e.g. "age", "gender" and "clothing". Moreover, the fact that these attributes usually do not correspond to some certain objects will bring in hidden semantic recognition problem. Multilabel pedestrian attribute classification is thus a more practical and challenging problem to be thoroughly investigated. The superiority of CNNs in the standard single-label natural image classification cannot be straightforwardly extended to the attribute classification scenario due to the difference in input image domains and problem settings. Specifically, in multi-attribute recognition, candidate attributes may locate in different regions of the given image and the label space of a k-category classification task becomes 2k , which requires more training data and parameters for establishing a deep neural network to achieve a comparable accuracy. Current CNN solutions for the pedestrian attribute classification problem mainly focus on extracting high-level features from the entire image [Sudowe et al., 2015; Li et al., 2015; Xiao et al., 2016], and multi-branch architecture for different attribute groups [Zhu et al., 2015; Lu et al., 2017; Zhao et al., 2018; Han et al., 2018a], to accomplish the subsequent classification.

The multi-branch manner for pedestrian attribute recognition achieves better performance via sharing the bottom l layers then breakuping into tree-like architecture, resulting in task-specific sub-networks for similar attribute groups. This is inspired by the common observation that bottom layers in CNNs mainly extract low level visual features which can be shared across different tasks, while the top layers capture high level semantic features that are more task specific. [Hand and Chellappa, 2017] proposed a multi-task deep convolutional neural network (MCNN) for facial attribute classification. [Lu et al., 2017] dynamically created a tree-like deep architecture where similar tasks reside in the same branch until the top layers. [Wang et al., 2017; Zhao et al., 2018] utilized a CNN-RNN model to take advantage of the intragroup mutual exclusion and inter-group correlation. These methods mine the correlations of attributes, but ignore the prior knowledge underlying the attribute data.

the j-th dimensionality of the label space Y, θj ∈ Rd is the parameter of linear classification layer.

7 In practice, for a given image x (image index i is omitted to simplify notation), it is difficult to accurately estimate its conditional probabilities w.r.t. all attributes labels simultaneously, due to the tangle between attributes and the deformation of objects, e.g. occlusions, illumination changes, rotations, and scale transforms. To address this problem, the multi-branch architecture is utilized in this work, i.e. we have m branches {b1, ..., bm}. The lower layers are shared across branches and the top layers are separated to focus on the attributes at different positions, as shown in Fig. 1. The detailed multi-branch architecture is given in the experiments section. Formally for each branch, we can construct the following m × k matrix,

HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential In this paper, we propose to develop a new CNN architecture for pedestrian multi-attribute recognition by exploring the correlation between different attributes, i.e. attribute co-occurrence priori [Mensink et al., 2014; Modiri Assari et al., 2014; Guo and Gu, 2011; Boguraev and Ando, 2005]. In particular, there is a high probability that some attributes that often co-occur in the training set will also co-occur in the testing set. For example, given a pedestrian annotated with "long hair", "skirt" and "high-heeled shoes", we can easily deduce its gender attribute as "female". Based on this insightful observation, we develop a novel attribute aware pooling method (AAP) for integrating the information from different predictions, namely CoCNN. More specifically, the base network follows the multi-branch architecture, and then the context information in these branch is collected to estimate the attribute probabilities, which is then combined with the individual estimations of each branch for improving the resulting decision. The diagram of the proposed scheme is shown in Fig. 1. Experimental results on benchmarks demonstrate the superiority of the proposed algorithm over the state-of-the-art methods for pedestrian attribute classification.

2 Preliminaries

We first briefly introduce some related works on CNN and then build the relationship between the output of a conventional CNN and the proposed attribute aware pooling method.

Let X and Y denote instance space and k-label space, respectively. Given a labeled training set with n instances, {(x1 , y1 ), (x2 , y2 ), ..., (xn , yn )}, where xi is the i-th instance and yi denotes its label vector which is a kdimensional binary vector, i.e. yi = [yi1 , ..., yik ] and yik ∈ {0, 1}. Denote the feature of xi calculated by a given CNN N as xi = N(xi ) ∈ Rd , the conventional probability hypothesis for the j-th attribute is

where aj is the j-th attribute in the dataset corresponding to where Pr(aj |bl) stands for the conditional probability of aj given the feature of l-th branch, and each row in P corresponds to the softmax output of the branch bl. Moreover, conditional probabilities of the whole image x w.r.t. different attributes can be estimated through a max-pooling operation:

It is instructive to note that the accuracy of Eq. 3 strongly depends on the prediction result of different branches. These branches are extracted from input images by exploiting empirical knowledge such as position information. However, attributes tangled with each other are difficult to be separated by branches accurately. Hence, the estimated conditional probability vector using Eq. 3 could be biased when the j-th attribute in x has not be well separated. Elements in the jth column of P will be extremely small, which leads to the Pr(aj |x) after the max-pooling being small as well.

A widely accepted fact in the filed of multi-label classification [Boguraev and Ando, 2005; Wang et al., 2013; Guo and Gu, 2011] is that correlated labels have a strong probability to simultaneously occur in real-world images. In practice, for an attribute aj , we can use the other attributes to estimate the conditional probability w.r.t. the j-th attribute. The auxiliary estimation is generated as follow:

where Si l ∈ {0, 1} is an indicator and Si l = 1 denotes ai appears in the image, and vice versa, and s = ∑ki=1 Si l is the number of the positive attributes. Pr(aj |ai ) is the conditional probability of two attributes or labels which can be precalculated by accounting the number of their co-occurrence in the dataset. P + is an m × k matrix, which can be applied for refining the prediction result of conventional CNNs for the multi-label pedestrian attribute classification problem. Since the co-occurance information Sj l of any branch bl is not given in the dataset, Eq. 4 is hard to calculate directly. We therefore propose a novel attribute aware pooling method which first integrates the co-occurance information of every branch {b1, ..., bm} and the priori knowledge, and then generates the estimation of the whole image by exploiting the co-occurrence priori over different attributes.

3 Attribute Aware Pooling

To further explore the correlation between attributes using the priori knowledge in the real data, in this section, we first build conditional probability matrices w.r.t. different attributes using the co-occurrence priori and then develop the new attribute aware pooling method.

3.1 Co-occurrence Priori Embedding

Given a training set with k labels and n instances, let Ni denote the number of the i-th label occur in the dataset, and pi = Pr(ai ) = Ni/n be the probability of the i-th label. By denoting the co-occurrence number of the i-th label and the j-th label in the dataset as Ni,j, we can construct a matrix J ∈ Rk×k , where Ji,j = Pr(ai , aj ) = Ni,j/n is the joint probability of ai and aj . Subsequently, we can obtain the conditional probability of an attribute given another attribute as

After constructing this matrix, we can further estimate the auxiliary probability P + l,j by exploiting the context information of the l-th branch xl. Fig. 2 shows the matrix C discovered on PA-100K dataset [Liu et al., 2017b], we can see that some attributes frequently co-occur in the dataset, e.g. "Female" and "Skirt&Dress", "LongSleeve" and "Trousers".

In the k-label classification task, given an instance with m branches {b1, ..., bm}, the softmax classifier outputs m kdimensional vectors, which are stacked into an m × k matrix P as shown in Eq. 2. For any branch bl, there are m - 1 branches in x that surround with bl and we denote these branches as ˜bl. The probability of the occurrence of attribute aj in ˜bl can be calculated by

However, this high-order posterior probability cannot be accurately calculated. Alternatively, we use the following locally max-pooling as an approximation:

Attributes identification in ˜bl can be easily accomplished through the following threshold function with the parameter τ :

and we can use this indicator matrix for calculating Eq. 4. Compared with the output conditional probability Pl,j of the l-th branch bl given the j-th attribute derived from CNNs, P + l,j is calculated by exploiting outputs of other branches, and can be taken as another important auxiliary predictor.

However, given the biased recognition of the branches, this hard threshold strategy may bring in inaccurate classification result and the introduction of the hyper-parameter τ will make s various for different branches. In addition, since attributes have some conflicts, absence of an attribute may lead to the appearance of another one. Therefore, we use the total probability theorem to modify P + l,j as follow:

where ˜ai indicates that ai does not occur, and the second term can be calculated as

and we can first calculate the matrix

based on the training set in advance for convenience. By exploiting the pre-defined two matrices C and ˜ C, the additional estimated probability matrix P + can be simply calculated as

where 1 is a k × k full one matrix.

Eq. 12 is estimated by exploiting context information and co-occurrence tables, which provides another approach to estimate the classification result of the input instance x. The attribute aware pooling method is thus developed to combine the two decision matrices of different properties. Formally, the output of the attribute aware pooling layer is

where λ is the parameter for balancing the prediction results of the branch itself and context information from the other branches. We will further test its impact on the classification accuracy in the following section experimentally.

Finally, we assemble predictions of every branch to form a k-dimensional vector. Since every element in ˆ P is greater than zero, for a given image xi , the output of the proposed method is

where M ∈ {0, 1}m×k is the max-pooling projection which has only one 1 per column. Let E i = diag( ˆ P T M), and we have

where || · ||1 is the conventional ℓ1 norm for vectors. The ground-truth conditional probabilities of xi is pi = yi /||yi ||1. Therefore, the loss function of the network using the proposed method is

By optimizing the above loss function using the mini-batch SGD approach, we can fine-tune filters in the neural network N and obtain a new CNN with improvement in attribute recognition accuracy. The diagram of the proposed attribute aware pooling architecture is shown in Fig. 1.

3.2 Back Propagation

We proceed to introduce the optimization method for CNNs with the proposed attribute aware pooling method. Stochastic gradient descent (SGD) needs to calculate the gradient of the loss function and then utilizes the back propagation strategy for updating parameters. We next detail the gradient and the back propagation of the proposed method.

In fact, the proposed attribute aware pooling method is attached after the softmax layer of a general CNN on m branches and followed by some pooling and probability calculations as shown in Fig. 1. Therefore, we do not have to modify any additional convolution layer and filters, and we only need to calculate gradient of the loss function in the last layer w.r.t. the individual branch estimation matrix P in Eq. 2. Commonly, this gradient can be computed using the standard chair rule, i.e.

We first calculate the gradient of the first two terms for a given instance x as follow:

where ||E||1 = ∑kj=1 |Ej|, and E ∈ Rk×1 . Subsequently, the gradient of ˆ P is calculated as follow:

where the variables in diag(·) make up a k-dimensional vector and M is the max-pooling projection which maps the gradient into a m × k sparse matrix.

As for the gradient of P , consider Ql = diag(P T Sl) where Sl ∈ {0, 1}m×k is the l-th mask w.r.t. the locally maxpooling (Eq. 7) for generating the l-th row of Q. Elements in the l-th row of Sl are zeros. The l-th row of ˆ P can be reformulated as

Removing terms irrelevant to P, the j-th element in P + l can be simplified as

where Cj and ˜ Cj are the j-th columns in C and ˜ C, respectively. The gradient of P + l,j to P is

Therefore, the gradient of the classification loss w.r.t. P is

Algorithm 1 Attribute aware pooling method for pedestrain attribute classification with CNNs.

Input: The multi-branch CNN N, a given image x and its label vector y. Two co-occurrence matrices C and ˜ C, weight parameter λ, and learning rate η.

• 1: Feed Forward
• 2: Calculate conditional probabilities from the m branches and constitute P (Eq. 2);
• 3: for l = 1 to mdo
• 4: Locally max-pooling: Ql ←maxi̸=l Pi (Eq. 7);
• 5: Estimate P + l using the co-occurrence priori: P + l ← (QlC + (1 -Ql) ˜ C)/k (Eq. 12);
• 6: Form the estimation: ˆ Pl ← Pl + λP + l (Eq. 13);
• 7: end for
• 8: Calculate the overall estimation ˆp of x (Eq. 14);
• 9: Back Propogation
• 10: ∇(ˆp) ← ˆp-p;
• 11: ∇(E) ←∇(ˆp)/(E T 1) - (∇(ˆp)T E)/(E T 1)2 ;
• 12: ∇( ˆ P ) ← M diag(∇(E ));
• 13: ∇(P) ←∇( ˆ P );
• 14: for l = 1 to mdo
• 15: for j = 1 to k do
• 16: ∇(P) ←∇(P)+ λk ∇( ˆ Pl,j)Sl diag(Cj - ˜ Cj) (Eq. 23);
• 17: end for
• 18: end for
• 19: P ←P - η∇(P);

Output: The classification result ˆp and ∇(P).

and P can be updated as

where η is the learning rate. Alg. 1 summarizes the feed forward and the back propagation of the proposed method.

4 Experiments

Here we will present the setting of experiments for evaluating the proposed method and compare it with the state-ofthe-art methods on benchmark pedestrian attribute recognition datasets.

4.1 Datasets andMetrics

The evaluation is conducted on three largest publicly available pedestrian attribute datasets:

• PETA [Deng et al., 2014]: The PEdesTrian Attribute dataset consists of 19,000 pedestrian images with 65 attributes (61 binary and 4 multi-class). Following [Li et al., 2015; Zhao et al., 2018], we randomly divide the whole dataset into three partitions: 9,500 images for training, 1,900 for validation, and 7,600 for testing, and focus on the 35 attributes whose positive proportions are bigger than 1/20.
• RAP [Li et al., 2016]: The Richly Annotated Pedestrian attribute dataset contains 41,585 pedestrian images with
• 72 attributes (69 binary and 3 multi-class). It is split into 33,268 images for training and the remaining 8,317 for testing. The same 51 binary attributes are evaluated for fair comparison following [Li et al., 2016].
• PA-100K [Liu et al., 2017b]: The PA-100K dataset has 100,000 images with 26 commonly used binary attributes. The whole dataset is randomly split into training, validation and test sets with a ratio of 8 : 1 : 1.

We use five evaluation metrics including a class-centric metric mean accuracy (mA), and four instance-centric metrics, i.e. accuracy, precision, recall and F1-score, following [Li et al., 2016; Liu et al., 2017b; Zhao et al., 2018]

4.2 Implementation Details

In the proposed method, there are m branches for attribute prediction. We use ResNet-50 [He et al., 2016] as the backbone network for building our multi-branch architecture. The main body of ResNet-50 except the last residual block is shared and the the last block is adopted as the architecutre of each branch. For a 448 × 224 input image, the feature maps F from the shared CNN is of size 2048 × 14 × 7. Specially, we design 4 branches in our model and the whole feature F are fed into the first branch. Inspired by [Yi et al., 2014; Sun et al., 2018], we spatially partition the feature maps F into 3 parts with sizes of 4 × 7, 6 × 7 and 6 × 7, serving for the other three branches, as shown in Fig 3. Each part focuses on different human position, i.e. head, upper body and lower body, for position-aware attribute recognition.

In our method, we first counted occurrence and cooccurrence numbers of different categories among train images and built two matrices C and ˜ C for each dataset. The backbone model parameters are initialized by directly using the pre-trained models on ILSVRC 2012 [Russakovsky et al., 2015]. Adam optimizer [Kingma and Ba, 2014] with a batch size of 32 is used to fine-tune the entire model with an initial learning rate of 0.01. The images are resized to 512 × 256 and during training randomly cropped to the size of 448 × 224 with random horizontal flippling. All methods were implemented using PyTorch [Paszke et al., 2017] and run on NVIDIA V100 graphics cards. We denote the multibranch architecture with the proposed attribute aware pooling method as CoCNN.

The proposed CNN architecture with the attribute aware pooling method has only one hyper-parameter λ whose functionality is to combine the original CNN output and the estimation from the context information. We tested the impact of λ by tuning it from 0 to 0.5 with the step of 0.05 on the mA metric using the PETA dataset. In fact, a larger λ makes the result ˆp more inclined to the auxiliary estimation P + and vice versa. As a result, we obtained a 86.97% mA value at λ = 0.2 which is the best trade-off between estimations in Eq. 13. Moreover, for RAP and PA-100K datasets, we keep λ = 0.2 in the following experiments.

4.3 Results and Analysis

Effect of Multi-branch Architecture and AAP

The improvement of CoCNN mainly comes from two aspects, i.e. multi-branch architecture and attribute aware pooling. In this section, we will measure how much improvement these two aspects bring. The vanilla ResNet-50 with binary cross entropy loss is adopted as baseline model, and the multibranch ResNet-50 without attribute aware pooling is denoted as multi-branch model. From the results in Tab. 1, we can see that multi-branch architecture improve the metrics generally in the three datasets by focusing on different parts of the human body. With attribute aware pooling, CoCNN further improves the accuracy to higher level.

HUAWEI TECHNOLOGIES CO., LTD. Huawei Confidential 10 79.95, 87.65). The recall of CoCNN is the second best one. This improvement mainly comes from that the proposed method takes multiple attributes into account when generating the classification annotation.

Two examples from the PA-100K dataset are given in Fig. 4 for qualitative analysis. From the results, we found that multibranch model made wrong predictions "Skirt&Dress" and "AgeLess18" for the two pedestrians, respectively. By exploiting their correlation with other attributes, especially the "Female" in the left person, and the "UpperPlaid" and "Backpack" in the right example, CoCNN well corrected the wrong predictions in multi-branch model.

Comparison with State-of-the-art Methods

After verifing the effect of the proposed method, we compared the proposed approach with several state-of-the-art approaches, e.g. DeepMAR [Li et al., 2015], HP-net [Liu et al., 2017b], SR C-RNN [Liu et al., 2017a], VAA [Sarafianos et al., 2018], and GRL [Zhao et al., 2018]. Results of comparison models are mostly reported from their papers directly. The results are listed in Tab. 1.

From results on PETA dataset, we found that, CoCNN clearly outperformed other state-of-the-art methods and achieved the highest mA, acc, prec and F1 values (86.97, In addition, similar observation also can be seen on RAP and PA-100K datasets. This phenomenon further illustrates that the proposed attribute aware pooling method is a general auxiliary regularization for enhancing pedestrian attribute recognition and can be embedded into any similar task.

5 Conclusions

Existing vanilla CNNs cannot be effectively applied to handle the pedestrian attributes recognition task. Therefore, we propose using the co-occurrence priori for improving performance of CNNs, namely, attribute aware pooling. With a multi-branch CNN as base architecture, we first construct two co-occurrence tables through training sets and use context information of every branch to complete the decision derived from each branch itself, which not only excavates properties of different human parts but also investigates the relationship between different branch. Experiments conducted on several benchmark datasets show that the performance of the proposed method is better than those of state-of-the-art methods in terms of the commonly used metrics.