Variation Generalized Feature Learning via Intra-view Variation Adaptation

1 Introduction

Person re-identification (re-ID), matching persons across camera views, is playing an important role in large-scale surveillance camera networks. When the person is represented by a video sequence, it is termed as video-based re-ID. Within this field, supervised learning methods [Deng et al., 2018] [Chen et al., 2018] [Ye et al., 2019a] have achieved superior performance with large amount of annotated crossview video sequences on different benchmarks. With the growth of robust visual tracking and detection algorithms, unlabelled video sequences can be easily obtained. The video sequence provides abundant intra-view positives by assuming that the image frames within the tracking sequence belong to the same person. These characteristics motivate us to investigate an unsupervised solution for video re-ID task, which does not require any cross-view(camera) annotated labels.

Sensitive to intra-view variations

Discriminative for cross-view matching

Intra-view invariant Less discriminative

Figure 1: Intra-view invariant features (highlighted in red) are less discriminative than the variation parts (highlighted in green) for cross-view re-identification.

A popular approach in unsupervised video re-ID is pseudolabel estimation [Ye et al., 2017; Liu et al., 2017], i.e., iteratively estimate pseudo-labels and refine the model learning process. However, the performance relies heavily on a reliable model for initialization. To achieve a reliable initialized video representation, existing unsupervised feature learning methods usually extract the invariant components to intra-view variations [Khan and Bremond, 2016; Wu et al., 2018]. However, it may lose discriminative information for cross-view re-identification as demonstrated in Fig. 1. Some other methods [Khan and Bremond, 2016] also directly use the intra-view variant representation for crossview re-identification with the assumption that the intra-view and the cross-view variations are close. However, such an assumption is invalid in practical uncontrolled environments. It raises a issue about how to utilize the intra-view positive samples to simultaneously address various cross-view variations and handle the challenging unseen testing variations.

Inspired by pose-aware person re-identification, a possible solution is to design a variation-aware framework, in which multiple variation-specific models (model pool) are learned to extract the frame-level features. For precise variation-specific models, a huge number of intra-view variations should be considered and modeled, which is a very time-consuming. In this paper, we propose to learn the adaptable feature representation as shown in Fig. 2, which avoids learning a large model pool. And it can be generalized from the intra-view variation to cross-view and unseen testing variation. Specifically, we propose a novel video variation dictionary learning algorithm for intra-view variation modeling without using any manual labels. The popular variation dictionary learning methods usually assume that the variation-free images are available for all the persons in training stage [Yang et al., 2013]. However, this assumption is invalid in re-ID task due to unseen testing person identities with various unseen variations. To address this issue, a video variation dictionary is proposed to model the intra-view variation without any variation-free images for training. Secondly, a generalized variation modeling algorithm is proposed to learn frame level adaptable features, which guarantees that the learned model can be generalized to unseen variations. And the video level adaptable features are obtained by an Adaptability-Discriminability (AD) fusion of frame level adaptable features.

In summary, the contributions of this work are as follows.

• A variation generalized feature learning method is proposed by modeling intra-view variations. The learned representation could simultaneously handle the cross-view variations and generalize well on unseen testing variations.
• An Adaptability-Discriminability (AD) fusion method is proposed to generate video level adaptable feature representations using the learned frame-level adaptable features.

2 RelatedWorks

2.1 Unsupervised Person Re-identification

Early unsupervised image based re-ID methods mainly focus on designing hand-crafted descriptors. In these works, the reliable features across frames are extracted to represent a video, i.e. the stable color region [Farenzena et al., 2010], the recurrent structured region [Farenzena et al., 2010], the motion-invariant local body-action features [Liu et al., 2015], and the invariant subspace [Zhang et al., 2016]. Above methods extracting the components invariant to intra-view variations will lose discriminative information as demonstrated in Fig 1.

To extract discriminative information in an unsupervised manner, a dictionary learning algorithm with implicit label mining is proposed [Kodirov et al., 2016]. Along this direction, a label estimation algorithm using reciprocal nearest neighbor search and negative mining is presented to update the model iteratively [Liu et al., 2017]. Incorporate with global matching, Ye et al.proposed a dynamic graph matching method to estimate the labels and learn the classification model iteratively [Ye et al., 2017]. Since the label-estimationbased learning methods can extract the discriminative information only when such information is preserved in video level features, the adaptable feature can further improve the performance of label-estimation-based methods.

Some other methods adopt auxiliary labeled data for unsupervised domain adaptation and transfer learning [Lv et al., 2018; Wang et al., 2018b]. Besides, some methods adopt Generative Adversarial Network (GAN) to bridge the gap between source and target domains [Wei et al., 2018] [Deng et al., 2018]. In comparison, we do not need additional labeled data for unsupervised re-ID model learning.

2.2 Variation Dictionary Learning

Variation dictionary learning is a popular intra-class variation modeling approach for face recognition [Deng et al., 2012] [Yang et al., 2013] [Ding et al., 2015]. By introducing an auxiliary intra-class variation dictionary, the samples are represented by the sum of a target-appearance component and an intra-class variation component [Deng et al., 2012]. Since the intra-class variations of training data may not be the same to those of gallery data, the sparse variation dictionary is adaptive to the gallery set by learning a variation-model projection [Yang et al., 2013]. To deal with the pose variation problem, a patch-based transformation dictionary is learned to connect corresponding patches across poses under the multitask learning scheme [Ding et al., 2015]. The existing variation dictionary based methods are mainly designed for image based recognition problem and require a generic set for training, which make they suboptimal or even inapplicable in our video-based re-ID problem.

3 Variation Generalized Feature Learning

This section gives a detailed description of the proposed method. Suppose we have a collection of person image sequences {I a i,j } and {I b i,j }, where I a i,j (I b i,j ) refers to jth frame of person i captured by camera a (b). xcID i,j is a Nddimension feature vector extracted from image I cID i,j , i = 1, 2, ...,NcID , cID = {a, b}. For simplification, camera ID a and b are skipped when all data are captured by the same camera. Then the feature of image {Ii,j} is denoted as {xi,j}, where i = 1, 2, ..., N, j = 1, 2, ..., Ni.

The proposed framework is shown in Fig. 3. At the training stage, three main blocks, i.e. variation model, structural bottleneck and AD estimator, are learned by minimizing the corresponding three losses. The adaptable component flow (red part) extracts a frame level adaptable feature, while the confidence of adaptable flow mines reliability of the frame level adaptable feature. The outputs of the two flows are then combined to generate video-level adaptable feature. Note that the frame level feature extractor is fixed for simplicity. At the testing stage, the three main blocks are fixed to extract the feature representations of the input image sequence.

The organization of this section is as follows. Section 3.1 introduces the variation modeling method via video variation dictionary learning. Section 3.2 introduces the generalized variation modeling. Section 3.3 introduces the frame level adaptable feature learning method. And the AD fusion for video level feature representation is shown in Section 3.4.

3.1 Video Variation Dictionary Learning

For the variation dictionary learning, we first introduce the variation component decomposition of frame level feature vectors. Following [Deng et al., 2012], the feature vector of a frame xi,j can be represented by the sum of the personspecific component xpi , representation of Variation Dictionary Dv and small dense noise z, i.e.

where ˆDv = [ ˆDv 1 , ˆDv 2 , ..., ˆDvND ], each column of Dv models one particular variation and ˆωv i,j denotes the responses of xi,j to the variations w.r.t the dictionary.

To learn a sparse variation dictionary ˆDv∗ , the noise term z is minimized over all frames, i.e.

where || · ||1 denotes ℓ1 norm and λ is the weight of sparsity regularization term.

Since the person-specific component is usually unknown, we also estimate it in (2), i.e.

The inconsistent components in neighboring frames are considered as unadaptable features, which should be removed to handle the random noise. Following [Wang et al., 2014], we introduce the temporal consistency of variation modelings across frames to remove the inconsistent components, i.e.

where Si,m denotes the set of frames in the mth viewconsistent subsequence. To obtain such subsequences Si,m, we divide each sequence by employing an off-line change point detection algorithm [Basseville and Nikiforov, 1993], which can be solved via expectation-maximization (EM).

Combining constraint (4) with (3), we can estimate the person-specific component simultaneously, i.e.

Optimization method to solve (5) can be solved as searching for minimal reconstruction error with fixed sparsity following [Aharon et al., 2006].

3.2 Generalized VariationModeling

Given the variation modeling in (5), a straightforward method is to represent the video by an "invariant" component ˆxpi , which is also widely used technique in existing methods [Wang et al., 2014] [McLaughlin et al., 2016]. However, the person-specific component ˆxpi is not "variation-free" feature for the cross-view variations and hence may not be discriminative for recognition across different camera views.

Alternatively, we use a set of aligned features {xli,j |∀j} to represent a video, which is obtained by sum of the personspecific component xpi and the variation component represented by the dictionary Dv∗ learned in (5), i.e.

In (6), the components w.r.t. unseen variations are removed from the aligned feature as the reconstruction error.

To generalize the feature representation to unseen variations, we formulate it as a variation-specific domain generalization problem, where a variation-specific domain means a particular variation. In this manner, the learned model in training domains can be generalized to testing domains with unseen variations. In the following part, we will propose an adaptable matching method, in which the variation modeling is employed to construct the domains.

To deal with the large intra-view variations, we define variation feature sets using aligned feature vectors under the same variation condition, i.e. feature vectors xli1,j1 and xli2,j2 belong to the same variation feature set when ωv∗ i1,j1 = ωv∗ i2,j2 . Let Ω = (ωa,v∗ i,j , ωb,v∗ i′ ,j ′ ) denote the variation condition of a image pair (xa,l i,j , xb,l i′ ,j ′ ), and

∆Ω = {(xa,l i,j , xb,l i′ ,j ′ ) | (ωa,v∗ i,j , ωb,v∗ i′ ,j ′ ) = Ω} denotes a domain w.r.t. Ω. When the domain adaptable model fΩ is learned, matching score of the pair (xa,l i,j , xb,l i′ ,j ′ ) is given by fΩ (xa,l i,j , xb,l i′ ,j ′ ). Our goal is to learn domain adaptable models {fΩ} for different variation domains.

Due to the huge number of potential adaptable matching models, it is impossible to learn all the matching models one by one. To estimate such a large number of matching models efficiently, we assume that similar domains share similar matching models i.e.,

where fΩi , i = 1, 2 denotes the matching models on variation domains ∆Ω1 and ∆Ω2 . λΩ1,Ω2 denotes the similarity between domains ∆Ω1 and ∆Ω2 , and Cf is a positive number measuring the cross-domain recognition model dependency. For simplicity, the average correlation between two pairs of variation factors is employed to measure the domain similarity, i.e.

Let {ek} denotes the ND-dimensional standard basis vector set. According to (7), fen1 and fen2 are uncorrelated while all the matching models are correlated to the basis models {fen }. So we use the basis domains {∆ek } as source domains and represent matching models for all the other domains using the basis models. A matching model f ∗Ω for domain ∆Ω can be estimated by minimizing the average matching model differences in (7) to the source domains, i.e.

According to (9), the adaptable model set {fΩ} can be estimated when the basis models {fen } are obtained. Note that the number of basis models depend on the type of variations. When multiple types of variations occur simultaneously, the variation dictionary can contain over 100 atoms [Yang et al., 2013]. So it is still difficult to learn all the basis models especially when multiple types of variations occur. Therefore, optimizing (9) is impractical for complicated environment with various variations.

3.3 Frame Level Adaptable Feature Learning

To address above issue, a frame level adaptable feature learning method will be proposed in the section as an efficient solution for generalized variation modeling rather than multiple matching models under a mild assumption. We transfer the complicated adaptable matching model learning problem to be an adaptable feature learning problem. Let ˆ sli,j,i′ ,j ′ denote the matching score of a feature vector pair (xa,l i,j , xb,l i′ ,j ′ ) given by domain adaptable model fΩ. When we estimate the adaptable feature representation of xa,l i,j , xb,l i′ ,j ′ is skipped for simple denotation of the matching score, i.e. the matching score ˆ sli,j,i′ ,j ′ = fΩ (xa,l i,j , xb,l i′ ,j ′ ) is simplified as ˆ sli,j = fΩ (xa,l i,j ).

When {fΩ} are linear model and ||fΩ - fen || = ||fΩ - fen ||22 , we obtain fΩ = ∑k λΩ,en fen /||Ω||1 by solving (9). The domain specific matching score ˆ sli,j,i′ ,j ′ for an aligned feature pair (xli,j , xli′ ,j ′ ) is given by the weighted linear combination of the scores from basis models, i.e.

where F = [fe1 , fe2 , ..., feDn ] is the summarized recognition model matrix and Tr (·) denotes the trace of input matrix.

The multiple adaptable classification problem is transferred to be a single domain classification problem on high dimensional feature space xli,j (ωv i,j )T in (10). However, such a high dimensional classifier F is usually noisy and unreliable when the labeled training data are limited. Therefore, we reduce the dimension of feature ωv i,j (xli,j )T to reduce computational cost of recognition model F .

It is reasonable to assume that there exists invariant structure in person re-ID. Therefore, we propose to utilize the similarity between models {f T en } to improve model learning. So feature ωv i,j (xli,j )T cannot be directly vectorized for dimension reduction until common part between {f T en } is removed. To maintain the common part after dimension reduction, we decompose the basis model fen into a universal component f0 and a domain specific component f δ en , i.e. fn = f0 + f δn . Dimension reduction will be conducted only on the domain specific component. The recognition score ˆ sli,j in (10) can be represented by the sum of score from the universal component and the domain specific components, i.e.

where F δ = [f δ e1 , f δ e2 , ..., f δ eDn ] is the summarized domain specific model matrix.

According to (11), F is decomposed into a lowdimensional common part f0 and a domain specific part F δ . So the dimension of feature ωv i,j (xli,j )T can be reduced after vectorization. Here we employ 1D Principal Component Analysis (PCA) on the vectorization of xli,j (ωv i,j )T for its simplicity.

Let xdr i,j denote the reduced feature of ωv i,j (xli,j )T and Fdr denote the counterpart of Fδ, the summarized score sli,j can be estimated as follows.

where δdr is a small number, denoting the error derived from dimension reduction. From (12), the adaptable feature xai,j is given by the concatenation of the weighted reduced feature xdr i,j /||ωv i,j ||1 and the aligned feature xli,j , i.e.

where ⊗ denotes the Kronecker product and PPCA denotes the projection matrix of PCA.

Note that above adaptable feature estimation method is given under the linear classifier assumption. It can be easily further extended to the non-linear cases by utilizing Taylor series of classification model and introducing higher order statistics in (10).

3.4 Adapatability-Discriminative Fusion

Following weighted sum-rule fusion scheme, we estimate the Adapatability-Discriminative (AD) index and propose a feature level fusion method to combine adaptable frame-level features for video representation. The video based adaptable feature vector sav i is given by the weighted sum of the frame level adaptable feature vectors {xai,j }, i.e.

where ρi,j is the AD index to be optimized in the following.

Since the reconstruction error of the dictionary measures the magnitude of unadaptable features and the variation component measures the magnitude of adaptable features, they both indicate the reliability of the adaptable feature. Thus, we introduce the adaptability feature vector ϵi,j as the concatenation of the reconstruction error and the variation component, i.e.

It is reasonable to assume that the elements of the adaptability component may not be equally discriminative. Therefore, we learn discriminative weights to approximate a discriminability measure for the AD index. Inspired by Fisher's linear discriminant, we define the discriminability measure by the ratio ρd,cID i,j of standard deviation between false matching images (inter-class) and correct matching images (intra-class) to represent the discriminability, i.e.

where c′ID ̸= cID and d (·, ·) denotes a distance function between the two input vectors. The false matching and the correct matching image pairs are estimated by a pre-learned matching model [Ye et al., 2017]. To approximate the discriminability measure ρdi,j , we formulate the learning problem by linear regression on the adaptability component, i.e.

where λα is the weight of regularization term and it is set to be 1 in our experiments.

With the learned weight vector α∗ , the AD index is determined as the linear combination of elements in xpi,j and ϵi,j using weight α∗ learned in (17), i.e.

4 Experiment

4.1 Datasets and Settings

Datasets. We evaluate our method on three datasets, i.e., the iLIDS-VID dataset [Wang et al., 2014], the PRID 2011 dataset [Hirzer et al., 2011], the MARS dataset [Zheng et al., 2016]. iLIDS-VID was captured by a multi-camera surveillance camera network at an airport arrival hall. It contains 300 person image sequences under each of the two camera views. The PRID-2011 dataset consists of person image sequences recorded from two static surveillance cameras outdoor. Following [Wang et al., 2014], person video pairs with more than 27 frames are employed in the experiment. MARS dataset is a large-scale dataset captured by six cameras. It contains 20,715 different image sequences of 1261 persons.

Feature extraction. We employ a hand-crafted Local Maximal Occurrence (LOMO) feature [Liao et al., 2015] fo two small datasets (PRID-2011 and iLIDS-VID) and a deep Multiple Granularities Network (MGN) feature [Wang et al., 2018a] for the large-scale MARS dataset for evaluation. The LOMO feature analyzes the horizontal occurrence of local feature, while the MGN feature is extracted from a multibranch deep network architecture with both global and local feature branches. In our proposed method, pseudo-labels are estimated following [Ye et al., 2017] for training MGN.

Classifier. For the hand-crafted feature, a state-of-the-art unsupervised re-ID learning method, i.e. DGM [Ye et al., 2017], is employed to learn the classification model in the scheme of unsupervised learning. For deep feature, we employ Euclidean distance following [Wang et al., 2018a].

Evaluation protocol. For iLIDS-VID and PRID-2011 datasets, the sequence pairs are randomly separated into half for training and the other half for testing. For MARS dataset,

83

50.3

82.5

52.1

2016], UnKISS [Khan and Bremond, 2016], SMP [Liu et al., 2017], DGM [Ye et al., 2017], DGM+ [Ye et al., 2019b] and TAUDL [Li et al., 2018], are used for comparison. LOMO feature and MGN feature are employed on small-scale and large-scale datasets, respectively. The top-r matching accuracies and mAP on three datasets are shown in Table 3.

For the results on iLIDS-VID dataset, we can see that the proposed method achieves the best rank-1 accuracy and 5.5% improvement is achieved compare to the state-of-theart methods. Similar trends can be observed in experimental results in PRID-2011 and MARS datasets. Note that the adaptable feature learning and AD fusion can be conducted on more advanced features to achieve better performance.

5 Conclusion

In this paper, we propose a variation generalized feature learning method for unsupervised video based person reidentification. Rather than minimizing the intra-view variations in existing methods, we propose to adapt the intra-view variations to unseen testing variations. Specifically, frame level adaptable feature is learned via domain adaptation and video variation dictionary learning. The video level adaptable feature is then estimated using AD fusion, which achieves much better performance than the widely used mean/min pooling strategy. Experimental results on three public person re-identification datasets show that the proposed method achieve better performance than existing methods.

Acknowledgments

This work is supported by Hong Kong RGC General Research Fund HKBU (12200518).

Table 3: Top r rank matching rate (%) and mAP (%)comparing with state-of-the-art unsupervised video-based person re-ID methods on three datasets.

the training and the testing separation follows the protocol suggested in [Zheng et al., 2016]. The results are shown in Cumulated Matching Characteristics (CMC) curves. For evaluation in MARS dataset, mAP (mean average precision) value are also reported following [Zheng et al., 2016]. For stable statistical results, the experiments was repeated 10 times and the mean accuracy is reported.

Implementation. 10 iterations are conducted with λ = 0.5 in DGM for all three datasets following [Ye et al., 2017]. We set tmax,1 = 1 for efficiency. λ = 1, λα = 1. The dimension of xdr i,j is set to be the same as xi,j . For iLIDS-VID and PRID2011 datasets, the subsequences {Si,m} are estimated follow [Basseville and Nikiforov, 1993]. For MARS, each tracking sequence is considered as a subsequence.

4.2 Self Evaluation

Effectiveness of each component. This part evaluates the effectiveness of frame level adaptable feature representation and the AD fusion, as shown Table 1. To evaluate the frame level adaptable feature (FAF, xdr i,j ), we compare it with the original feature vectors (Baseline, xi,j ) with min/mean pooling. We observe that it achieves much better and stable performance than baseline feature representation. In addition, we also compare with the set-based distance by calculating the minimum (Min) and mean (Avg) of the frame-to-frame distances. The consistent improvements on two datasets demonstrate the effectiveness of the proposed AD fusion.

Iterative updating. We also report the performance at different iterations on PRID-2011 and iLIDS-VID datasets in Table 2. We observe that the iterative updating process improves the performance significantly on both datasets. And the performance is converged with less than 10 iterations.

Dictionary size. We also plot the rank-1 accuracy with different dictionary sizes on the PRID-2011 and iLIDS-VID datasets as shown in Fig. 4. According to the experimental results on two datasets, we find that the larger dictionary usually produces higher performance, especially on challenging iLIDS-VID dataset. However, larger dictionary size means higher computational cost in both training and test stage. For the efficiency considerations and avoid over-fitting, the dictionary size is set to 100 in all the experiments.

4.3 Comparison with the State-of-the-arts

Six state-of-the-art unsupervised multi-shot/video based person re-identification methods namely GRDL [Kodirov et al.,