Discriminative Log-Euclidean Feature Learning for Sparse Representation-Based Recognition of Faces from Videos

1 Introduction

In many practical applications such as surveillance-based face recognition and smartphone video-based face authentication, the test example contains a set of face images that share the same, yet to be determined label. As video-capable consumer devices and surveillance cameras are becoming more abundant, the interest in using image sets for face recognition has grown. While the multiplicity could mean improved recognition, low-resolution and variations in pose, illumination and occlusion limit significant improvements in performance.

Over the years, many methods have been proposed for using image sets for object classification in general and face recognition in particular. In order to capture the variations within an image set and/or model the properties inherent in face images, many methods employ descriptors that live on some non-Euclidean spaces such as the Symmetric PositiveDefinite (SPD) manifold or the Grassmann manifold. In such cases, the machine learning algorithms originally designed to work on Euclidean spaces have to be carefully modified to properly and meaningfully work on non-Euclidean ones.

Among the many machine learning algorithms that have been successfully used for image set classification, Sparse Representation-based Classification (SRC) over dictionaries has been shown to be very effective [Chen et al., 2012; Ortiz et al., 2013]. The standard SRC algorithm has become popular in visual identification tasks since the seminal work of Wright et al. [2009]. The success of this method is justified by the discriminative low-dimensional subspace structure that is naturally exposed in the space of the visual images of an object. More specifically, it has been mathematically proved that the images of a fixed object taken under varying Lambertian illumination from a fixed viewpoint lie on a low-dimensional subspace [Basri and Jacobs, 2003]. As illustrated in Fig. 1, this suggests that the instances from a particular class lie on (or close to) a low-dimensional linear subspace (assuming same-pose viewing of a static object) or a small number of such subspaces (to account for variations in pose and deformations).

Although originally designed for vector spaces, algorithms for sparse coding have also been extended to work on nonEuclidean spaces such as the SPD manifold [Yuan et al., 2010; Guo et al., 2010]. However, the subspace property was mathematically proved when images are represented using intensities [Basri and Jacobs, 2003]. In other words, the property (mathematically) applies only when pixel raw intensity features or other features derived linearly from the intensities are used. If intensities are nonlinearly transformed (e.g. to extract nonlinear features prior to sparse coding), algorithms based on sparse coding may lose the discriminative advantage offered by the subspace property. Since the SPD descriptors used for image sets are obtained nonlinearly from the input image features, one way to enhance the performance of SRC under that setting is to further embed the nonlinear features in a manner that improves the discriminative subspace structure of the data.

In this paper, we propose an approach for "face identification from image sets" using sparse coding on the LogEuclidean (LE) tangent space TISQ+ of the SPD manifold SQ+ . In this approach, we first describe each image by generating a number of local covariance descriptors; then we use a dictionary of atoms from the LE tangent space TISQ+ to represent each gallery image set. While previous LE approaches for image set classification extract from each image set, a single or very few LE samples that have very high dimensionality, our approach extracts from each set many LE samples of a much lower dimensionality, reducing the possibility of over-fitting. Given the LE features, we then formulate an optimization problem for learning an embedding into a lower-dimensional LE tangent space TISq+ in which the data has a more discriminative subspace structure. To classify a probe image set, we use the LE feature transform computed during training to embed the dictionary of LE atoms extracted from the probe image set. Next, we apply the LE sparse coding approach [Yuan et al., 2010; Guo et al., 2010] to classify the embedded probe atoms with respect to the augmented gallery dictionary. Extensive experiments on three public datasets show that our method outperforms many state-of-the-art methods. In addition, we run an empirical ablation analysis to understand how the different components of our approach contribute to its final performance. In order of importance, the contributions of the paper can be summarized as follows:

• ÷ An LE dimensionality reduction algorithm that leads to a more discriminative subspace structure, subsequently enhancing the performance of SRC with nonlinear features. Since it reduces the learning problem into that of solving a single generalized eigenvalue in a non-iterative fashion, the algorithm is also efficient.
• ÷ An image set feature extractor which models each image set as a dictionary of LE atoms that is more robust to local deformations and has significantly fewer dimensions than other LE image set descriptors [Wang et al., 2012; 2015; Huang et al., 2015b], making our LE features more robust to over-fitting. In our experiments, we show that the proposed features also improve the performance of another recent LE image set method proposed

in [Huang et al., 2015b].

• ÷ To the best of our knowledge, this paper is the first to apply SRC for image set classification on non-Euclidean spaces. Note that SRC has been extended to image set classification on Euclidean space by Ortiz et al. [2013] and to other classification problems on LE tangent spaces [Guo et al., 2010; Yuan et al., 2010]. Our experiments show that the proposed approach outperforms existing methods on three public video face datasets.

The rest of this paper is organized as follows. Section 2 reviews the relevant literature. The proposed approach is presented in Section 3 and the results of the extensive empirical evaluation are presented in Section 4. The paper is concluded in Section 5.

2 RelatedWork

The image set classification problem has been formulated in various ways. One popular formulation is to compute the distance, either over a vector space or a manifold, between the probe set and each gallery set and then associate the probe with the class of its nearest gallery set. These include discriminative [Hamm and Lee, 2008; Wang and Chen, 2009; Harandi et al., 2011; Wang et al., 2012; Huang et al., 2015b; 2015a; Wang et al., 2015] and non-discriminative methods [Wang et al., 2008; Cevikalp and Triggs, 2010; Hu et al., 2011; Chen et al., 2012; 2013]. Other formulations that do not rely on nearest neighbor-based classification include the binary SVM reverse-training approach [Hayat et al., 2014b], neural network-based methods [Hayat et al., 2014a; Lu et al., 2015], linear representation/coding methods [Ortiz et al., 2013; Zhu et al., 2013] and clustering methods [Mahmood et al., 2014].

Linear Representation (Coding) Methods: The SRC algorithm, originally designed for classification of a single image [Wright et al., 2009], was extended to image sets by Ortiz et al. [2013] who proposed the Mean-Sequence SRC (MS-SRC) algorithm. While MS-SRC assumes Euclidean space, Harandi et al. extended the sparse coding approach to Grassmann manifold [2013] where it has been applied to face recognition from image sets. Sparse coding over SPD manifolds was also considered but for non-image set classification tasks as in [Yuan et al., 2010; Guo et al., 2010; Harandi et al., 2012]. Since video-based face recognition typically involves large number of gallery samples, we intentionally avoid the scalability issues associated with kernel methods by developing the proposed embedding algorithm and performing sparse coding on the LE tangent space.

Log-Euclidean Feature Learning: Various approaches for learning features, metrics, and/or dimensionality reduction embeddings have been proposed within the LE framework [Wang et al., 2012; Li et al., 2013; Vemulapalli and Jacobs, 2015; Yger and Sugiyama, 2015; Wang et al., 2015; Xu et al., 2015; Huang et al., 2015b]. The goal of these approaches is to boost the performance of nearest neighbor classification whereas the goal of our work is to boost the performance of subspace-based classification. Qiu and Sapiro [2015] proposed an approach for learning linear transformations that improve the performance of subspace-based classification in Euclidean space. The approach uses subgradient descent to minimize a non-convex cost function and so it requires too many iterations to converge and is subject to local minima. In addition, the approach requires performing Singular Value Decomposition (SVD) in each iteration, which makes it even more expensive. The proposed LE feature learning approach is significantly more robust as it is not subject to local minima. In addition, our approach is faster and more scalable as the optimal solution is obtained by solving a single generalized eigenvalue problem.

3 Our Approach

We describe the three components of our approach (description, embedding, and coding) in the following subsections.

3.1 Image Set Descriptor: Dictionary of LE Atoms

Existing SPD image set descriptors, like those used in [Wang et al., 2012; Huang et al., 2015b; Wang et al., 2015] compute a single or a small number of SPD matrices per image set. These descriptors suffer from some drawbacks. One such drawback is the curse of dimensionality as each SPD matrix descriptor has the dimensions WH ⇥WH (or more) assuming the images have size W ⇥H (the descriptor is 160,000D for images as small as 20 ⇥ 20). The image set may also contain too few images to reliably compute such highdimensional descriptors, leading to undersampling at the level of each descriptor. Undersampling at the gallery level (and subsequent overfitting) may also be a problem as the typical gallery contains few image sets per class. This results in a correspondingly few number of high-dimensional descriptors that may not be enough to reliably train a machine learning model.

To avoid these problems, we extract a symmetric matrix feature L 2 TISQ+ from each image I, and we use a compressed version of its vectorization a = comp (vec (L)) 2

RD as an atom in a dictionary corresponding to the image set. The steps for computing the image-level features are summarized in Fig. 2 and described below.

At the heart of our image-level descriptor is the use of Region Covariance Matrices (RCMs) [Pang et al., 2008]. This is justified by the ability of covariances to fuse various types of features and keep track of their statistics. In addition, the covariance of a set of samples is invariant to rearrangement of these samples, giving the covariance more robustness to misalignment, a problem that face identification systems have to deal with even when automatic face alignment is applied, as the detection and alignment algorithms are still not perfect.

To compute the covariance matrices, we first compute a feature image φ(I) similar to [Pang et al., 2008; Harandi et al., 2012] which produces at each pixel the following M = 43 responses:

where Gu↪v(x↪ y) = gu↪v(x↪ y) ⇤ I(x↪ y) is the response of the image to the 2D Gabor wavelet gu↪v(x↪ y) [Harandi et al., 2012]:k2

where u is the orientation index, v is the scale index, kv = 1◁p2v-1 , and ✓u = ⇡u◁8. To balance the trade-off between robustness to misalignment and spatial encoding, we follow the tradition of breaking the image into nr ⇥ nc cells and compute a covariance matrix for each cell based on the pixel responses in that cell.

To avoid the curse of dimensionality in the extracted descriptor, we compress the M responses at each pixel in cell (r↪ c), prior to computing the cell-specific covariance matrix, by projecting the M-D response vector into a subspace of a lower dimensionality m using a cell-specific, M⇥m columnorthogonal projection matrix Ur↪c. Each matrix Ur↪c is computed by performing PCA on the M-D response vectors at all pixels within the cell (r↪ c) from all the images in all gallery image sets. In our experiments, we set m = 10.

After compressing the responses, we calculate the m⇥m covariance matrix ⌃r↪c from the m-D responses in cell (r↪ c). Next, we arrange the nr ⇥ nc covariances into the diagonal blocks of a Q ⇥ Q matrix ⌃, where Q = nrncm. The matrix ⌃ can be easily shown to be SPD and so it lives in the non-Euclidean SPD manifold SQ+ . To measure the similarity in this non-Euclidean space, we endow SQ+ with the LE Metric [Arsigny et al., 2007] which measures the distance between any pair of SPD matrices X1 and X2 by first using the Log map: Log : SQ+ ! TISQ+ to map them to the LE tangent space TISQ+ and then computing the Frobenius distance kLogX2 -LogX1kF . If the Singular Value Decomposition (SVD) of an SPD matrix of dimensions m ⇥ m is X = Udiag(s1↪ ▷▷▷↪ sm)VT , the Log map is defined as:

The LE tangent space TISQ+ is equivalent to the space of symmetric matrices SQ, which is a vector space. This allows us to apply the familiar Euclidean machine learning algorithms to SPD matrices once they are mapped to the LE tangent space. Accordingly, the final steps are (a) mapping the SPD matrix ⌃ to the LE tangent space by computing L = Log⌃, (b) computing the uncompressed atom ˜a = vec (L) 2 RQ2 , and then obtaining the compressed atom a = comp (˜a) 2 RD where the operator comp() retains only the D = nrncm2 entries of ˜a corresponding to the nrnc diagonal blocks of L while discarding the rest (see the structure of a in Fig. 2).

Arranging the cell covariances into the diagonal blocks of ⌃ and mapping ⌃ to the LE tangent space unnecessarily requires more memory and processing time. Instead, we apply the equivalent but more efficient process of separately mapping each cell covariance matrix ⌃r↪c to the LE tangent space, which gives Lr↪c = Log⌃r↪c. In addition, we store only the D = nrncm2 nonzero entries of L, which correspond to its diagonal blocks Lr↪c, into the compressed atom a 2 RD . All the remaining steps use the compressed atom a instead of the uncompressed, higher dimensional atom ˜a 2 RQ2 .

The feature extraction step can be very efficiently implemented by making use of GPUs for performing convolutions and matrix multiplication. For each image, nr ⇥ nc small eigenvalue problems need to be computed for SPD matrices of size m⇥m in order to compute their matrix logarithms. Additional nr ⇥ nc eigenvalue problems of M ⇥ M matrices need to be solved for performing PCA during training but these are done only once for the complete gallery set rather than for each image.

3.2 Log-Euclidean Feature Learning

The goal of this step is to map the image descriptors from the LE tangent space TISQ+ into a lower-dimensional LE tangent space TISq+ in which they have a more discriminative subspace structure. In other words, we want the samples from one class to stay, in the new space, as far as possible from other-class subspaces while staying close to the same-class subspaces. In this new space, the sparse coding of a query sample y from class c over the dictionary A will more likely find that the subdictionary Ac provides better reconstruction of y compared to other subdictionaries. Consequently, the sparse coding will more likely associate y with its true class c. Tangent Map Formulation: There are different ways to formulate the tangent map W : TISQ+ ! TISq+. One way is by the linear formulation given by:

where W 2 RQ2⇥q2 . To guarantee that L0 2 Sq for any L 2 SQ, the matrix W has to be constrained, such that it has q(q + 1)◁2 unique columns while the other q(q - 1)◁2 columns are permutations of other columns 1 .

The second formulation W2 is a variation of W1 that avoids placing constraints on W by keeping only the unique q(q + 1)◁2 columns inW so that we just compute the (vectorized) lower triangular submatrix tril(L0 ) 2 Rq(q+1)◁2 instead of the complete matrix L0 2 Rq⇥q :

where W 2 RQ2⇥q(q+1)◁2 . Since ˜a = vec (L) has only D nonzero entries at known locations, the projection matrixW in both W1 and W2 needs only to contain the D rows corresponding to these nonzero entries. In this case, the dimensions of W in W1 can be reduced to D ⇥ q2 while in W2 it will be D⇥q(q+1)◁2. For simplicity, we use the second formulation, in which W is unconstrained and has dimensions D ⇥ q(q + 1)◁2.

It is worth noting that a third formulation was used in [Xu et al., 2015; Huang et al., 2015b] which has the advantage of using much fewer parameters in the projectionW. However, the formulation is quadratic in the projection parameters compared to linear formulations W1 and W2. The quadratic formulation is useful for applications in which the SPD descriptors are very high-dimensional such as the 400 ⇥ 400 image set covariance used by Wang et al. [2012]. The SPD descriptors in this paper have considerably fewer dimensions, and so we opt to use the simpler linear form W2. As we see later, our choice of a formulation that is linear in the parameters leads to an easier-to-solve optimization problem in which finding the globally optimal solution is straightforward and efficient.

Optimization Problem: Let Ac 2 RD⇥Nc be the dictionary containing all the Nc compressed atoms from all image sets associated with class c (after removing all identical atoms due to identical images):

Furthermore, let N be the total number of atoms in the gallery, C be the number of classes, c(i) be the class associated with atom ai, and let zi↪c be the dense representation

1 There are other algebraically equivalent ways to express the constraint on the columns of W, all of them leading to the same measure of distance between symmetric matrices. Since we do not use the formulation W1 in this paper, further elaboration on these ways is beyond the scope of this paper.

B

B

C

of an atom ai with respect to the dictionary Ac of a different class c:

where we use λ1 = 0▷001. If we let Jc = ATc Ac + λ1I, we obtain zi↪c = J-1 c ATc ai. The first goal we need the tangent map W2 to achieve is to maximize the distance between every atom ai, from a certain class c(i), and its dense projection Aczi↪c on the dictionary of each other class c 6= c(i) (see

The reason we use dense, l2-regularized representations is that it has a closed form solution that is more efficient to evaluate. Moreover (and more importantly), we want to maximize the distance between each atom ai and the span of as many as possible of those different-class atoms that may contribute to reconstructing ai. This makes the dense representation a more appropriate choice.

Before describing the other goal, we redefine the dense representation zi↪c for the case in which we project ai on the dictionary of its own class (i.e. c = c(i)):

The only difference between (4) and (7) is the constraint z(i) = 0 which excludes any solution in which ai contributes to its own representation. If we let ui↪c = J-1 c ATc ai, and wi = u(i) i↪c ◁J-1(i↪i) c , we obtain zi↪c = ui↪c - wicoli(J-1 c ).

The other goal the tangent map has to meet is minimizing the distance between every atom ai from a certain class c to its dense projection Aczi↪c on the dictionary of its own class:

where S2 = PCc=1 1 Nc Pi↪c(i)=c (ai - Aczi↪c)(ai - Aczi↪c)T . In addition, we add a regularization term kWk2F = trWT W to the quantity to be minimized. We then combine all goals in one criterion by maximizing the following ratio of quadratic forms:

The optimal solution to this problem is obtained by finding the q(q+1)◁2 generalized eigenvectors with the largest eigenvalues of the following generalized eigenvalue problem:

After findingW, we use it to embed the dictionaries of all classes. If we assume all the C classes have the same number of images Nc = N◁C, the computational complexity of feature learning is O (D3 +C ⇥ (CN 3 c +DN 2 c +CD2Nc )), where it takes O(D3 ) for the solution of the D ⇥ D generalized eigenvalue problem in (10), O(DN 2 c +N 3 c ) for computing Jc and inverting it for one class, O(CN 3 c ) for computing the representations of same-class samples and other-class samples with respect to the dictionary Ac of one class, and O(CD2Nc) for computing the contribution of one class to the two scatter matrices S1 and S2.

3.3 Coding and Classification

Given a probe image set Y = [y1↪ ▷ ▷ ▷ ↪ yNy ], the method extracts the LE dictionary from the set Y as described in Section 3.1; then uses the tangent map W to project each atom in Y's dictionary to the LE feature space. Subsequently, we apply SRC to compute the label for Y. More specifically, we solve for the sparse representation vector x 2 RN corresponding to the mean ¯ y of the embedded feature vectors (i.e. LE Frechet mean [Arsigny et al., 2007]):

where A is the dictionary containing all the embedded LE atoms from all classes. Given the sparse representation ¯x, we can find the class contributing the most to the representation, and with which ¯ y should be associated, using the minimum residual rule of [Wright et al., 2009]. If we let δc be the N ⇥N diagonal matrix with all zeros except at the Nc diagonal entries corresponding to the atoms of class c, the residual rc(y; ¯x) corresponding to class c is given by:

The class for which rc is minimum is chosen as the label for the probe set.

4 Experimental Evaluation

We have conducted extensive experiments to compare the performance of the proposed algorithm, i.e. Log-Euclidean Feature Learning with SRC (LEFL-SRC), against several existing algorithms for image-set classification. The compared methods include Affine Hull-based Image Set Distance

(AHISD) [Cevikalp and Triggs, 2010], its convex variant (CHISD) [Cevikalp and Triggs, 2010], Sparse-Approximated Nearest Points (SANP) [Hu et al., 2011], Dictionary-based Face Recognition from Videos (DFRV) [Chen et al., 2012], Mean Sequence Sparse Representation-based Classification (MS-SRC) [Ortiz et al., 2013], a variation of MS-SRC that uses Collaborative Representation-based Classification (CRC) [Zhang et al., 2011] for classifying the mean of the sequence (MS-CRC), Set to Set Distance Metric Learning (SSDML) [Zhu et al., 2013], Deep Reconstruction Models (DRM) [Hayat et al., 2014a], Projection Metric Learning (PML) [Huang et al., 2015a], and Log-Euclidean Metric Learning (LEML) [Huang et al., 2015b]. To understand the contribution to performance made by the different components of our classifier, we are also comparing with two variants of our classifier: one without LE features but with feature learning applied to intensity features (FL-SRC), and another with the LE features but without the feature learning (LE-SRC).

For existing methods, we have used the source code provided by the original authors and set the parameters according to the recommendations made in their respective papers. The only exception to this are MS-CRC and MS-SRC which we have implemented ourselves. To guarantee a fair comparison, the same features and dataset splits were used to compare all the methods. We made an exception for the DRM approach where we report the performance using the 1475-D LBP features extracted from the intensity features used with the rest of the methods. The reason for this exception is that the original paper of DRM [Hayat et al., 2014a] and its publicly available source-code included the extraction of LBP features as one of the preprocessing steps of DRM. For PML, we have modified the method to deal with the situation in which the number of images ns in a given image set s is lower than the dimensionality d of the subspace PML computes from each image set. In that case, we synthesize additional images by small random translations and rotations of the original ns images so that s has 2d images. Since PML requires the gallery to have at least two image sets for each class whereas MobFaces-I provides a single gallery image set per class, we randomly split each gallery set in MobFaces-I into two subsets of nearly equal sizes (the difference in size is at most one).

Parameter Settings: We use the following parameters in our proposed method. We resize each input image to w = 120 and h = 144. We then divide each image into a nr = 6 ⇥ nc = 6 grid of non-overlapping cells for calculating the RCMs. As stated earlier, we use m = 10. Finally, we set the dimension of the lower-dimensional LE tangent space to q = 28 which corresponds to an LE tangent map W with q(q + 1)◁2 = 406 columns. It is worth noting that smaller grids (i.e. smaller nr and nc) lead to inferior recognition performance. Grids larger than nr = 6 ⇥ nc = 6 could possibly lead to better performance at the expense of increasing the memory footprint of the algorithm. However, we have not tried such larger grids.

4.1 YouTube Celebrities (YTC)

The YTC dataset contains 1,910 YouTube-downloaded videos of 47 subjects [Kim et al., 2008]. For a given subject, the videos are short segments clipped from three longer, parent videos downloaded from YouTube. YTC has been built to be very challenging for face tracking and recognition by choosing low resolution videos with wild variations in pose, scale, hair style, make-up, illumination, motion and number of people per frame.

Experimental Protocol: We run ten-fold cross-validation experiment. The 9 ⇥ 47 = 423 videos in each fold are randomly selected from the complete dataset while minimizing the overlap between different folds as much as possible.

Feature Extraction: We use the Viola-Jones (VJ) detector [Viola and Jones, 2004] to locate the faces in each video. Then we use the eye locations detected using the method of Asthana et al. [2013] to align the subject's face to a standard, 30⇥36 pixel frame. The intensities are histogram equalized and arranged in a 1080-D feature vector. We use the feature vectors from a given video define the corresponding image set. 2

4.2 YouTube Faces (YTF)

The YTF dataset contains 3,425 videos of 1,595 subjects with diverse ethnicities [Wolf et al., 2011]. Similar to YTC, YTF videos are downloaded from YouTube and are very challenging for face recognition. We conduct our experiments on those subjects with four or more videos available. This results in 226 subjects. After randomly dropping one subject, we randomly split the remaining 225 subjects into five mutually exclusive groups, with 45 subjects each. We repeat the experiment for each group where we use the first three videos of each subject as gallery sets and the remaining videos for testing. We stress that we run face identification (multiclassification) experiments rather than binary verification as is usually done on YTF in the literature. Since the dataset provides aligned face images, we extract intensity features from each image by cropping the central 100⇥100 box from each image, resizing it to 30 ⇥ 36, and histogram-equalizing it.

2 We have not cleaned any of the bad detections or misaligned faces in an effort to test the robustness of the compared methods to such outliers.

4.3 Mobile Faces (MobFaces)

The MobFaces dataset contains 750 videos of 50 subjects taken by a smartphone's front camera during various user interactions with the phone [Fathy et al., 2015]. There are three sessions of five videos each (one enrollment + four tasks) per subject where each session is taken under a different illumination and/or in a different place. The dataset includes some of the unique challenges of mobile-based continuous facial authentication such as wild variations in illumination and context due to the mobility of the device. We compute the features using the same pipeline we developed for the YTC dataset. Although the features used in this paper are different from [Fathy et al., 2015], we adopt the two evaluation protocols suggested in [Fathy et al., 2015] by dividing the task videos into ten-second long clips and treating each clip as a separate query. In the first protocol (MobFaces-I), training is done using only the 50 enrollment videos of one session and testing is performed on the ten-second long task video clips from the two other sessions. In the second protocol (MobFaces-II), training is done on the 100 enrollment videos of two sessions and testing is done on the task video clips of the remaining session. Results are reported for each of the six scenarios possible with these protocols. The clipping of the 600 task videos results in 1065 ten-second clips for the first session, 587 the second, and 666 for the third.

4.4 Results

Table 1 shows the mean and standard-deviation of the recognition rates of the compared methods for the YTC and YTF datasets while Table 2 shows the recognition rates for the six different evaluation scenarios for the MobFaces dataset. Both tables clearly show the superiority of the proposed method (LEFL-SRC) in comparison with other methods.

The improvement in performance on MS-SRC by FL-SRC is not significant except on YTC. This is because feature learning does not help much with intensity features, which inherently have a good discriminative subspace structure. Accordingly, the only significant advantage FL-SRC provides over MS-SRC is the reduction of dimensionality without loss in identification accuracy. On the other hand, the results show that the improvement due to feature learning is significant when we compare LE-SRC with LEFL-SRC. Since LE features are nonlinear in intensities, they do not preserve the sparse linear dependencies between samples though these nonlinear features are more robust. In addition to reducing the dimensionality of the LE features, the feature learning algorithm improves their discriminative subspace structure which in turn boosts the performance of SRC. It is worth noting that although LEFL-SRC outperforms MS-SRC and LE-SRC, LEFL-SRC uses only 406 features per atom which is fewer than the 30 ⇥ 36 = 1080 features used by MS-SRC and the D = 3600 features used by LE-SRC.

We emphasize that we are using a slightly different experimental pipeline. In particular, we detect face landmarks and use them to align the detected faces. While this should generally improve the performance of all methods, this is not the case for LEML [Huang et al., 2015b] as alignment makes many of the face images in each set near identical, resulting in a near-zero image set covariance, which is the major component of LEML's Gaussian descriptor. Such degeneracy explains the relatively low accuracy obtained for LEML. We have also derived a modified version of LEML that we refer to as LE-LEML. It uses the image-level SPD descriptors proposed in this paper instead of the set-level Gaussian descriptors. Tables 1 and 2 clearly show the use of the image-level descriptors significantly boosts the performance of LEML.

5 Conclusion

We proposed LEFL-SRC, an approach for face identification from image sets using sparse coding on the Log-Euclidean (LE) tangent space of the SPDmanifold SQ+ . In this approach, we first describe each image by generating a number of local covariance descriptors arranged in a grid; then we use a dictionary of atoms from the LE tangent space TISQ+ to represent each gallery image set. While previous LE-based approaches for image set classification extract a single or very few LE samples with very high dimensionality, our approach extracts from each set many LE samples of much lower dimensionality. We formulated an optimization problem for learning an embedding into a lower-dimensional LE tangent space TISq+

in which the data has a more discriminative subspace structure allowing accurate subsequent application of SRC. Extensive experiments on three public datasets (YTC, YTF, and MobFaces) show that our method outperforms many state-ofthe-art methods. In addition, we made an empirical ablation analysis where we have shown how the different components of our approach contribute to the final perofrmance.

Acknowledgments

This work was supported by a cooperative agreement FA8750-13-2-0279 from DARPA. We thank Pouya Samangouei for helpful discussions and the anonymous reviewers for their valuable feedback.