Nonlinear Hierarchical Part-Based Regression for Unconstrained Face Alignment

1 Introduction

Face fiducial feature detection is a fundamental step in many Human-Computer Interaction (HCI) applications, such as face recognition, human emotion analysis, autonomous driving, etc. Many different algorithms have been proposed and have shown promising results both in accuracy and speed [Xiong and la Torre, 2013; Belhumeur et al., 2011; Cootes et al., 2012; Sagonas et al., 2013]. They aim towards not only the near-frontal faces but also faces in the wild. However, due to large head pose variation, various types of occlusions, unpredictable illumination and some other factors, the fiducial point (landmark) localization task still remains challenging.

Early representative work, such as the Active Shape Model (ASM) [Cootes et al., 1995; 1998; Matthews and Baker, 2004], uses a parametric model to represent a set of the face fiducial points and proposed an iterative framework for optimizing the fiducial positions. Following these efforts, researchers have attempted to build more robust and sophisticated models which can be robust to different types of interfering conditions, such as pose and expression variations exemplified in Fig. 1. The multi-view deformable part model [Zhu and Ramanan, 2012; Ghiasi and Fowlkes, 2014] alleviates the pose problem. However, discrete pose intervals and rigid shape modeling make it difficult to capture all possible facial variations. Recently, regression based methods [Cao et al., 2012; Xiong and la Torre, 2013; Ren et al., 2014] report accuracy approaching the human labeling level and their typical runtime can be within several milliseconds. However, the regression based methods may significantly suffer from uncommon part appearance due to extreme poses, lighting or expression. Noisy appearance results in bad features and the mapped landmark displacement is disturbed. On the other hand, faces in unconstrained environments are often affected by occlusion as shown in Fig. 1. Hence, occlusion handling becomes crucial for improving these methods. An ensemble of a set of occlusion-resistant regressors relying on the probabilistic inferring is proposed to implicitly overcome occlusion [Yu et al., 2014]. Explicitly detecting the occlusion and incorporating the occlusion information into landmark detection also show some satisfactory [Ghiasi and Fowlkes, 2014]. Yet the interaction between the occlusion detection and landmark localization is under investigation.

Another important aspect for the regression-based methods is the number of iterations. Usually there is no sophisticated rule to set the "ideal" number of iterations. An empirical choice for the number of iterations may cause the regressor to under-fit the data or drift away from the correct solution. Establishing an online evaluation strategy would help to indicate how well the landmarks are localized by the current stage. Furthermore, if the occlusion information can be simultaneously retrieved with the localization process, the framework would be more unified and efficient.

In this paper, we propose a two-stage framework consisting of a pose-dependent holistic regression model and a hierarchical part-based regression model to robustly localize the face fiducial points. Faces with different poses are fed to different sets of pose-dependent regressors. Consequently, the shape variation inside each set is largely reduced. From the holistic regressors, the hierarchical part regressors are automatically learned by our proposed projection optimization algorithm. Based upon the hierarchical part-based structure, the alignment likelihood is firstly evaluated to determine whether further local regressions are needed and also to estimate the occlusion information simultaneously; then the hierarchical part-based regression models are applied to corresponding parts to further refine the landmarks. Occlusion status are propagated to all the landmarks from the previous occlusion detection during the part-based regression.

2 RelatedWork

Numerous methods have been proposed in the facial feature localization literature, e.g. deformable part model [Cristinacce and Cootes, 2007a; Huang et al., 2007b; Saragih and Goecke, 2007; Saragih et al., 2011; Tzimiropoulos and Pantic, 2013; Cheng et al., 2013; Yu et al., 2013; Medina and Zafeiriou, 2014; Xing et al., 2014], regression [Liang et al., 2008; Cao et al., 2012; Xiong and la Torre, 2013; Ren et al., 2014; Zhu et al., 2015], convolutional neural network [Zhang et al., 2014b; 2014a], etc.

To overcome pose variation, multi-view shape models [Cootes and Taylor, 1997; Zhu and Ramanan, 2012] were proposed either by local search to estimate the head pose or by combining models from different view-points. The regression-based methods can also handle certain pose variations, which are incorporated into the training data. However, too much pose variation increases the training complexity. Cascaded pose regression [Dollar et al., 2010; Burgos-Artizzu et al., 2013] and conditional regression forests [Dantone et al., 2012] are the most similar works to ours. The former ones take pose as an explicit factor to regress, while ours treats the pose as a conditional hidden state. The latter one partitions the poses into subspaces before the regression fitting. Within each subspace, they aggregate many regression trees to predict landmarks, while in our method, we allow the pose state to change during each step of regressions.

A number of regression constructions have since been proposed, such as boosted regressions [Cristinacce and Cootes, 2007b; Valstar et al., 2010; Martinez et al., 2012], regression forests [Dantone et al., 2012; Yang and Patras, 2013; Kazemi and Sullivan, 2014], linear regressions [Xiong and la Torre, 2013; Dollar et al., 2010; Asthana et al., 2014], regression ferns [Cao et al., 2012; Burgos-Artizzu et al., 2013], etc. Regression-based methods are fast but are sensitive to occlusion. There have been several works introduced for handling occlusion, for example, Artizzu et al. [Burgos-Artizzu et al., 2013] proposed a block-wise statistical model to approximate the occlusion. Yu et al. [Yu et al., 2014] introduced multiple regressors which are specially designed to infer the particular occlusions. A similar work [Ghiasi and Fowlkes, 2014] also used a hierarchical deformable part model to localize landmarks. This method is similar to [Zhu and Ramanan, 2012] in which it sets up multi-view shape models and adopts detection based strategies to vote for the positions. To infer the occlusion status, both [Ghiasi and Fowlkes, 2014] and ours use part-based models. But for alignment, instead of detection of facial features in a pictorial structure in [Ghiasi and Fowlkes, 2014], we model both holistic and local landmark update as a regression based strategy and jointly learn the part regressors from holistic regressors in a projection optimization framework.

3 Our Approach

we propose a hierarchical regression method with pose-dependent and part based modeling. As shown in Fig. 2, we first propose a conditional cascaded regression model to separate the regression manifold into several subspaces. Then a hierarchical part-based model is proposed to decompose the holistic structure into a more flexible part-based hierarchical structure. Each part represents a facial component, e.g. mouth. By inferring the occlusion conditions for the parts, the landmarks' update model is regularized.

3.1 Preliminary

Given the definition of N facial feature points, denoted as s = [x1↪ x2↪ ▷▷▷↪ xN], and their starting position s0, the goal is to minimize the squared error in coordinates k(s0 +∆s)-s⇤ k2, where s⇤ is the ground truth. The evidence we could observe is only the appearance feature. Thus, Eq. 1 minimizes the error in feature space instead of the coordinate space.

h is the feature descriptor, i.e. SIFT feature [Lowe, 2004]. I(s) are the facial image patches surrounding each fiducial point of s. h(I(s)) is the concatenated feature descriptor applied on each of the image patches I(s). The cascaded regression based framework [Xiong and la Torre, 2013] has the shape update form as follows:

(a) Face Input

(b) Conditional Cascaded Regression

(c)Hierarchical Part-based Regression

(d)Parts Connections

Figure 2: Graphical structure illustration of the proposed framework. (a) The input face image. (b) Conditioned by head poses, the face key points are initialized with different priors and a cascaded regression is applied as global shape fitting. (c) The holistic shape is split into parts hierarchically to effectively overcome the local shape variance, e.g. the shape is firstly divided into left part (left profile, left eyebrow and left eye), middle part (nose and mouth) and right part (right profile, right eyebrow and right eye). The second layer is derived from the first layer by further dividing the components. (d) The geometric connections of the two layer parts defined in (c).

where Rt is the regression matrix and bt is the intercept. φt = h(I(st)) denotes a local feature descriptor all through the work. Typically the number of iterations is fixed to 4 or 5. The cascaded regression attempts to apply a set of linear regressions sequentially to predict landmark positions. Given ground truth s⇤ , the training process is to minimize the prediction error over all training samples T as follows:

3.2 Conditional Cascaded Regression

To reduce the complexity of the face shape manifold, we divide the manifold into several subspaces, as shown in Fig. 2 (b). Shapes are mainly clustered into three groups, the frontal view, the left view and the right view. We set the threshold angles to be -22▷5◦ and 22▷5◦ . Then given image I, by introducing head pose parameter ✓, the regression problem becomes equivalent to solving Eq. 4.

where ⇥ is the set of discrete head pose intervals, ⌅ is the distribution normalizer and the pose likelihood term is learned based on the logistic regression framework:

where is a holistic appearance feature, i.e. HoG and Φ is a normalization factor to make p(✓♣I) a distribution.

The conditional alignment likelihood p(∆s♣✓↪ I) is modeled by the coordinate displacement in Eq. 6,

where Γ is again a normalization factor. Notice that ∆s = Rt(✓)φ + bt(✓). We assume the alignment likelihood p(∆s♣✓↪ I) follows the exponential distribution. At each regression iteration, we maximize the alignment likelihood in Eq. 4 by conditioning on different head poses. The corresponding holistic regressors are applied to update the landmark positions. Such procedure largely reduces the shape complexity caused by head poses and are more likely to converge.

3.3 Hierarchical Part-based Regression

Holistic regression is effective in aligning the face as a whole, but it may not produce perfect fitting results at local parts due to appearance and shape deformation. For instance, assuming the same face with eyes fully open and half open, holistic regressions localize landmarks as a whole but may fail in the eye region due to the lack of local constraint from the holistic regression. A part-based regression step could alleviate this problem more by deformable local fitting.

Part-based Local Regression

From the holistic regression, each landmark's update utilizes exactly one row of R. By dividing the facial area into parts, we partition the regression matrix R into row-wise blocks. Recall from Fig. 2 (c), for the first layer, we divide the shape into left, middle and right parts. As shown in Eq. 7, the partition of R is denoted as R = [Rl↪Rm↪Rr]T , where Rl↪Rm and Rr correspond to left, middle and right parts, respectively. Such division is recursively applied by further partitioning the previous layer's blocks into smaller units. Fig. 2 (c) shows the second layer of facial components, i.e. left profile, mouth, left eye, etc.

Notice that the partition still uses the holistic feature φ for update ∆s = [Rl↪Rm↪Rr]T φ. In other words, inside the regressors Rl↪Rm↪Rr themselves, the correlation in between should be diminished. We aim to obtain local regressors from the holistic regressor R as shown in Eq. 7.

where each part regressor ˆRi↪ i = l↪m↪ r is a block-wise sub-matrix. The transformation optimization from Ri to ˆRi is introduced in Sec. 3.4. After generating the local part regressors directly from the holistic ones, the part-based regression confirms to the same update rule, ∆si = ˆRiφ + ˆbi↪ i = l↪ r↪ m.

Localization Evaluation

To determine when to halt the holistic and local regressions, we set up an evaluation function to validate the alignment. The function propagates each part's (a.k.a component's) alignment score to the upper layer and finally generate the overall alignment score. Given the kth component Gk and its landmarks si 2 Gk, the part score function can be defined as:

where U(I↪ si) is the unary term defined in Eq. 9, i.e. the inner product of the feature and its corresponding weights. φ(I↪ si) is the descriptor extracted at landmark si. The relationship between φ appeared in the previous sections as descriptor and φ(I↪ si) is φ = [φ(I↪ s1)↪ ▷▷▷↪ φ(I↪ sN)].

The second pair-wise term Q(si ↪ sj) is defined as the geometric smoothness term of two landmarks in one component, i.e. q(si ↪ sj) = [♣si - sj♣↪∆♣si - sj♣]↪ si↪ sj 2 Gk, which is independent from the image I, the landmark's alignment likelihood and occlusion condition.

Occlusion Regularization

We then independently train a classifier Oi = wo↪iφ(I↪ si) + co↪i to provide the occlusion likelihood of each landmark at each regression step. This additional classification step produces little overhead due to the sharing of features for both regression and occlusion detection.

Misalignment is not necessarily caused by occlusion, while occlusion can adversely affect the alignment. Suppose landmark si is occluded. The alignment score U(I↪ si) is close to 0, which does not contribute to the overall score. By detecting occlusion of si, we can equivalently set the feature at si as φ(I↪ si) = 0. During the process, the landmarks' occlusion condition is confidently predicted by both the occlusion detector and the alignment score.

All the landmarks' occlusion states are then modulated by the neighboring landmarks with a Markov Random Field. Landmarks with sufficiently small and large scores of Oi and alignment are selected as negative and positive boundary conditions respectively. By setting up the connection weights among the landmarks, a label propagation algorithm [Zhu et al., 2003] is applied to assign the unlabeled landmarks. Our system framework is illustrated in Algorithm 1.

3.4 Holistic and Part Regression Training

In this section we describe the training of the holistic regressors and how to derive the hierarchical part regressors directly from holistic regressors. We also introduce the training of the graphical model for evaluating the alignment likelihood.

Holistic Regressor Training: We firstly introduce the training details for holistic regressors. In experiments, we Algorithm 1 The two-stage regression algorithm.

• 1: Input: I, s0, threshold d
• 2: Output: s, O
• 3: repeat
• 4: run Eq. 2, Eq. 6 and Eq. 5, optimize Eq. 4.
• 5: evaluate Oi = wo↪iφ(I↪ si) + co↪i↪ i = 1↪ ▷▷▷↪N, set φ(I↪ sO0) = 0
• 6: until T1 times
• 7: fix ✓, evaluate Eq. 8, if E(G) > d, halt.
• 8: repeat
• 9: for layers in hierarchical structure
• 10: run part-based Eq. 2
• 11: evaluate Oi = wo↪iφ(I↪ si) + co↪i↪ i = 1↪ ▷▷▷↪ N, set φ(I↪ sO0) = 0
• 13: end
• 12: evaluate Eq. 8, if E(G) > d, halt.
• 14: until T2 times

tried the gaussian random perturbation of each landmark, even if the perturbation step is small, the regression result returns jittering shapes. When the training samples are not sufficient, we augment the initialization by rotation and random perturbation of global translation. Meanwhile, to prevent overfitting, denoting ∆s = s⇤ - s0, we modify Eq. 3 by adding the regularization terms, which is Eq. 10. The problem can be solved by splitting R into row pieces and each piece-wise sub-problem is convex.

Part-based Regressor Derivation: As we have introduced in Eq. 7, to convert holistic regressors Ri to part regressors ˆRi, we propose a projection matrix W to accomplish the transformation. The original partitioned regressor is projected onto a new subspace in which the correlation between parts is diminished, as shown in Eq. 11. Projection from holistic regressors is expected to preserve the global information for the local regression. Directly training part regressors only contains local information, which is sensitive to noise. Moreover, bounding the update error provides criterion for automatic update halting.

We expect that after transforming the original R into block-wise ˆR, by bounding the part regression error, the holistic regression error becomes a supreme of the part regression error in Eq. 12.

Thus, it leads to an optimization over Wi↪ i = l↪ r↪m such that the local part regression further reduces the update error based on the holistic result. The above optimization problem can be formulated as:

We simplify the notation of Ri↪ bi as ˜Ri = [Ri↪ bi] and the raw feature is rephrased as ˜ φ = ⇥φT ↪ 1⇤T . ˜Ri 2 R(m↪n) ↪Wi 2 R(n↪m) , m is the number of landmarks in the corresponding part and n is the original feature dimension plus one dimension of bi. WT i Wi = I constraints that the projection of each part should be orthogonal.

By solving independently each part's transformation matrix Wi, we obtain each local part's regressor ˆRi which is guaranteed to further shrink the localization error because the optimization of Eq. 13 is to find the optimal Wi such that the displacement from the ground truth is minimized from the holistic step. For each part's regressor, a second layer regressor can be achieved under the same construction.

Localization Evaluation Model Training: The weights for score calculation of landmark localization evaluation in Eq. 9 are learned in the following. We first concatenate bottom layer unaryweightswui aswu , bottom layer pair-wise weights wbi↪j as wb and upper layer pair-wise weight wgi↪j as wg . We denote q(s) = [q(si ↪ sj)] for all pairs (i↪ j), in which the pair-wise smoothness features for all landmarks are concatenated. Similarly, q(G) = [q(Gi↪ Gj)] denotes the upper level pair-wise feature for all the parts. Re-arranging all the weights as w = [wu ↪wb ↪wg ] and all the features as f = [φ↪ q(s)↪ q(G)], the evaluation score is wT f . We set the loss function as hinge loss, which is the first term in Eq. 14. By regularizing w with l2 norm, minimizing the loss function leads to solution of w for all the parts.

The training set C includes both positive and negative samples. The positive samples are the facial images with ground truth landmark positions while the negative samples are non-facial images with initialized landmarks or facial images with unaligned landmarks. ↵ is the ground truth label taking 1 if it is positive sample and -1 if the sample is negative. The above problem can be efficiently solved by gradient descent approach.

4 Experiments

We evaluate our method on six challenging benchmarks, i.e., Labeled Faces in the Wild (LFW) [Huang et al., 2007a], Labeled Facial Parts in the Wild (LFPW) [Belhumeur et al., 2011], Helen [Le et al., 2012], Annotated Faces-in-the-Wild (AFW) [Zhu and Ramanan, 2012], iBug [Sagonas et al., 2013] and Caltech Occluded Faces in the Wild (COFW) [Burgos-Artizzu et al., 2013]. To evaluate the localization performance under occlusion, subsets of LFPW and Helen are selected, which are denoted as LFPW-O and Helen-O. During all the experiments, LFPW and Helen refer to the whole datasets. Our occlusion detection method is evaluated on LFPW-O, Helen-O and COFW, with comparison to the state-of-the-art.

4.1 Experimental Setting

Evaluation: We compare our method, "hierarchical part-based regression" (HPR), with five state-of-the-art methods, Supervised Descent Method (SDM) [Xiong and la Torre, 2013], Robust Cascaded Pose Regression (RCPR) [Burgos-Artizzu et al., 2013], Consensus of Regression (CoR) [Yu et al., 2014], Dlib [Kazemi and Sullivan, 2014] and Chehra [Asthana et al., 2014]. These methods report the top performance among the most recent regression-based methods. Methods with different experimental settings, e.g. Neural Network structures are currently not compared. The codes are provided by the authors from internet. RCPR provides its training code in which the annotation can be defined by input data. To make the comparison consistent, the training databases for HPR, RCPR and CoR are the same, which are LFPW and Helen. SDM is reported to be trained with MultiPIE and LFW. Since SDM and Chehra uses 49 points annotation for training and testing, we also select the overlapped 49 points from our 66 points training setup for testing on LFPW, Helen, AFW and iBug, by neglecting 17 points of face profile. For fair comparison, we use the images in which faces are successfully detected by a third-party face detector, i.e., Viola-Jones detector [Viola and Jones, 2004].

4.2 Localization onWild Databases

As shown in Fig. 3, our method HPR performs consistently better, especially the performance on Helen database, which is 10% higher at relative error 0.02. The error is calculated by dividing root mean squal pixel error over the face size. Face size is calculated as the tight bounding box around the ground truth. The proposed method explicitly separates the landmarks into components which is more suitable for deforming the local shape variance. The average runtime on a 640 by 480 image is around 0.3s in Matlab with a dual core i7 3.4GHz CPU.

Furthermore, the AFW and iBug, two more challenging databases, are evaluated in Fig. 4. We provide each face in AFW and iBug with a bounding box according to the ground truth for fair comparison.The proposed method is still consistently on top of the other state-of-the-art methods.

4.3 Localization on Occluded Datasets

For validating accuracy, an evaluation is conducted on the selected occlusion datasets, LFPW-O and Helen-O and the specific occlusion database COFW. Quantitative results from Table. 1 show that HPR achieves consistently better results on the occlusion datasets especially comparing the two occlusion-robust methods, CoR and RCPR. Note that RCPR is trained based on the COFW itself while other methods including ours are not trained on this database.

Further, we synthesize occlusion data from Helen and AFW. A black bounding box is centered at the randomly selected points. The occlusion ratio is controlled by the black bounding box area over the facial area, i.e. 5%, 10%, 15% and 20%, respectively.The CDFs comparing on the synthesized occlusion data is shown in Fig. 5. Regarding the increasing levels of occlusion, all the methods show accuracy decrease. RCPR is relatively more sensitive to the occlusion conditions since the accuracy of different occlusion levels varies largely. Our method performs better on Helen and on AFW at lower occlusion level, while it shows sensitive trend on AFW when the occlusion portion increases.

5 Conclusion

We introduced a hierarchical non-linear regression model and facilitated it to the face landmark localization HCI. The non-linear regression model predicts quite well the highly non-linear manifold of the human face shape. With the proposed conditioned regressions, head pose variation is controlled within each separated subspace and the global shape is fast localized. With hierarchical part-based regression, the alignment is evaluated and occlusion information is fed back to the local regressions. Meanwhile, local shape variance is compensated by the part-based regression and the occlusion information is propagated to other landmarks at the last step. Demonstrated by the extensive experiments, the top localization accuracy and fast performance provide high potential for more applications such as human tracking.