RePFormer: Refinement Pyramid Transformer for Robust Facial Landmark Detection

1 Introduction

Facial landmark detection aims to localize a set of predefined key points on 2D facial images. This task has attracted significant attention due to its wide range of applications [Ko, 2018]. With the remarkable success of convolutional neural networks (CNNs), modern facial landmark detectors [Wang et al., 2019a; Qian et al., 2019; Jin et al., 2021] have achieved encouraging performance in constrained environments. However, as fundamental components in real-world facial applications, facial landmark detectors need to be able to consistently generate robust results for complex situations, which remains a significant challenge for existing methods. Figure 1 illustrates some examples of challenging face images and the detection results of existing algorithms and our method.

∗ Shengcai Liao is the corresponding author.

According to the ways of generating landmark coordinates from feature maps, previous works can be generally classified into three categories: heatmap-based methods, regression-based methods, and hybrid methods. Heatmap-based methods [Wang et al., 2019a; Dong and Yang, 2019] treat facial landmark detection as a segmentation task, where pixels belong to the classes of landmarks and background. These methods mainly face two difficulties: 1) To reduce the quantization error in CNN backbones, they usually leverage single-scale high-resolution heatmaps [Ronneberger et al., 2015] to represent the surrogate results of landmarks, which introduces high computational costs and prevents them from fully exploring the pyramid features. 2) They lack global constraints on facial landmarks and ignore the relationships between image contexts and landmarks, which decreases their robustness to complex scenes. Regression-based methods [Lv et al., 2017] directly transform image features into landmark coordinates with high efficiency. However, a single regression step is difficult to achieve satisfactory performance. Cascaded regression steps [Lv et al., 2017; Feng et al., 2018] are leveraged to refine results, but they are not fully end-to-end trainable and their performance is prone to saturation. Recently, Sun et al. [Sun et al., 2018] combined heatmap and regression together to inherit the advantages of both. However, their network designs are imbalanced, with most of the computational costs concentrated on the backbone. Tan et al. [Tan et al., 2020] demonstrated that imbalanced network architectures can limit the detection performance. Moreover, existing detectors often focus on directly learning the landmark representation from pure image features, and ignore the dynamics and interactivity of landmarks and context information.

To address above limitations, we propose a novel facial landmark detector named the Refinement Pyramid Transformer (RePFormer) which focuses on robust detection for complex scenarios. We treat facial landmarks as attention quires, and formulate the facial landmark detection as a task of refining landmark queries through pyramid memories in an end-to-end trainable procedure. Specifically, we propose a pyramid transformer head (PTH) to mimic the zooming-in mechanism. In PTH, cross-scale attention constructs pyramid memories to combine long-range context and propagate high-level semantic information into lower levels. Then, landmark-to-landmark and landmark-to-memory attentions are employed in each PTH stage to enable dynamic interactions of landmarks and pyramid memories, which helps to regress the landmarks in various scenarios. Besides, we design a dynamic landmark refinement (DLR) method to refine the landmark queries by predicting residual coordinates and dynamically aggregating queries in an end-to-end trainable way. The residual coordinates prediction decomposes the regression into multiple steps, and each step refines the landmark queries based on a specific level in pyramid memories, which fully leverages the multi-level information in pyramid memories and decreases the difficulty of prediction. We conduct extensive experiments on several facial landmark detection benchmarks. Our models achieve state-of-the-art performance and show strong robustness on complex scenarios.

2 RelatedWork

Facial Landmark Detection. Early facial landmark detection methods mainly focus on deforming a statistical facial landmark model into a 2D image by an optimization procedure. These methods typically apply different constraints on the statistical models, such as object shapes and texture features [Wu and Ji, 2019]. However, they are not robust under various scenarios in the wild. Recently, deep neural network (DNN) based methods [Wang et al., 2019a; Jin et al., 2021] show promising performance on this task. Their architectures are usually composed of a CNN backbone and a variant of a heatmap-based or/and regression-based detection head. The pixel values of heatmaps represent the likelihood of landmarks exiting in the corresponding positions [Wang et al., 2019a; Dong and Yang, 2019]. A drawback of them is that the down-sampling ratio of heatmaps introduces quantization error. Thus, high-resolution networks, such as U-Net [Ronneberger et al., 2015] and HRNet [Wang et al., 2019a], are commonly applied to address this difficulty. Instead of utilizing heatmaps as surrogate results, regression-based methods [Lv et al., 2017] directly predict facial landmarks by transforming image features into 2D coordinates. Cascaded regression steps [Lv et al., 2017; Feng et al., 2018] are widely applied to further refine the predicted coordinates. They usually follow an iterative croppingand-regression pipeline which first crops patches from CNN feature maps and then sends them to carefully designed regressors, such as Recurrent Neural Network (RNN) [Trigeorgis et al., 2016] or Graph Convolutional Network (GCN) [Li et al., 2020]. However, extracting patches limits the longrange information exchange and may hinder the end-to-end training due to the non-differentiable cropping operators.

Vision Transformer. Transformer [Vaswani et al., 2017] is first proposed for sequence-to-sequence tasks in natural language processing. Its network architecture simply consists of self-attention based encoders and decoders to model dependencies among any positions in a sequence. Recently, computer vision, from high-level tasks to low-level tasks, has also benefited from its strong representation capability and long-range interactions [Carion et al., 2020]. The Vision Transformer (ViT) [Dosovitskiy et al., 2021] directly flattens image patches and processes their feature and position embeddings using a pure transformer encoder for the image classification task. DETR [Carion et al., 2020] introduces a transformer-based detector to solve the object detection problem as unordered set prediction. Its fully end-to-end design removes the hand-crafted components, such as anchor assignment and duplication reduction, in modern object detectors. Transformers have also been explored for landmark detection tasks. Yang et al. [Yang et al., 2020] leverage a transformer as a non-local module to build long-range spatial dependencies in image features for more explainable human pose estimation. In contrast, the transformer in our model explicitly establishes the relations between landmarks to landmarks, and landmarks to image contexts, and gradually refines the landmark queries along the pyramid transformer memories.

3 Method

3.1 Overview

Problem Formulation. Given an input 2D image I, our RePFormer aims at learning a facial landmark detector fθ(I) to predict an ordered set L ∈ RN×D = {l1, ..., lN} which represents N ordered facial landmarks, where each landmark li is in D-dimensional coordinates. Note that the transformerbased object detector, DETR [Carion et al., 2020], also formulates general object detection as a set prediction task. However, the predicted set of DETR is unordered and with variable length, since the sequence and number of objects are not fixed in general object detection datasets. In contrast, the facial landmark detection task assumes that each image only contains one main face (if there are several faces in an image, only the largest central one is considered), which is annotated with a fixed number of landmarks. Besides, instead of solving a bipartite matching as in DETR, the label assignment in our framework is simply defined as a fixed one-to-one way.

Overall Architecture. The architecture of our RePFormer is illustrated in Fig. 2. It is built upon a CNN-based feature extractor and a transformer-based detection head. First, the input image is fed into a common CNN backbone which contains several network stages. The feature maps generated by these stages are of various resolutions and semantics, forming hierarchical image features. Next, our PTH module applies cross-scale attentions on these hierarchical feature maps to produce pyramid memories with multi-level semantic information. Then, the PTH stages use landmark-to-landmark and landmark-to-memory attentions to fuse the context information of memories into landmark queries and model the relations among landmarks. After each PTH stage, our DLR module performs mutual updates by predicting the residual coordinates of landmarks based on the status of current queries and evolving the landmark queries based on the coordinates of current landmarks. Thus, the whole facial landmark detection task is formulated as a step-by-step landmark refinement process.

3.2 Pyramid Transformer Head

Cross-Scale Attention. Feature pyramids have been explored for solving various challenges in computer vision, such as detecting objects at various scales [Li et al., 2019] and combining different levels of semantic features for segmentation [Ronneberger et al., 2015]. In this work, our transformerbased detection head, PTH, explicitly employs pyramid features to obtain cross-scale semantic information, mimicking the zooming-in mechanism of human annotators. Our crossscale attention takes the hierarchical feature set Z as input, and generates the pyramid memory set V following a topdown pathway. Z is obtained by a CNN backbone fB which is defined as follows:

where M is the number of network stages, and f iB is the ith network stage to take the feature maps zi-1 as input and generate feature maps zi as follows:

These hierarchical features form the CNN feature set Z = {zM, ..., z1}, which contains multi-level feature maps with various resolutions and semantics. Top-level features contain strong semantics but less spatial information, making them only suitable for robustly locating the rough positions of landmarks. To better propagate high-level semantics into all levels of features and combine long-range information, our PTH computes the ith memory vi by applying a cross-scale attention on every two adjacent levels of feature maps zi and memories vi+1, which is inspired by the top-down Grounding Transformer in [Zhang et al., 2020]. The architecture of cross-scale attention is shown in Fig. 3(a), which takes three inputs including query qi, key ki and value ˆ vi. Since the attention operation is permutation-invariant to input elements, qi is composed of zi and its fixed positional encodings pi by qi = zi + pi to maintain relative positions of pixels. ki and ˆ vi are both obtained by fusing the information of zi and vi+1. A bilinear interpolation upsamples vi+1 into ¯ vi+1 with the same resolution as zi. Then ˆ vi is generated by a Hadamard product as ˆ vi = zi ◦ ¯ vi+1, where ¯ vi+1 aims to mask output the noisy low-level details outside semantic areas and maintains the precise spatial information within semantic areas. To make the similarity calculation between qi and ki semantic and position sensitive, ki is also supplemented with fixed positional encodings as ki = ˆ vi + pi. In summary, the memory vi is computed as

where MHA is a multi-head attention [Vaswani et al., 2017], FFN is a feed forward network, and LN is a layer normalization. Thus, the pyramid memories form the set V = {vM, ..., v1}, where each vi has the same resolution as zi but with cross-level and long-range information.

PTH Forwarding. Our PTH detection head fH is composed of cascaded transformer stages, and defined as follows:

where K is the number of transformer stages and f iH is the ith stage. PTH introduces the landmark query set E ∈ RN×D = {e1, ..., eN} to store the states of landmarks, the size of which is the same as the number of landmarks. E can either be initialized from trainable embeddings as in DETR [Carion et al., 2020], or dynamiclly aggregated from memories as in our method, which will be introduced in next section. PTH takes the pyramid and initial landmark query set E1 as inputs and repeatedly updates the states of E based on the information from pyramid memories V , in a top-down pathway. This enables it to first robustly locate the rough positions of landmarks in the top-level memory, and then gradually integrates more and more precise spatial information from lower levels of memories. Specifically, f iH is the ith PTH stage to take the memory vM-i+1 and the landmark query set Ei in the ith state as inputs and generate the new query state Ei+1 for the next PTH stage. This is computed by

where each PTH stage is composed of two successive attentions, i.e. a landmark-to-landmark attention gLL and a landmark-to-memory attention gLM as shown in Fig. 3(b), which is inspired by the transformer decoders in [Vaswani et al., 2017; Carion et al., 2020]. They are both implemented with multi-head attentions. The query, key and value of gLL all come from Ei , and the query, key and value of gLM are the output of gLL, vM-i+1 and vM-i+1, respectively. gLL models the dynamic relations among landmark queries using their similarities, and updates their status. gLM computes the relations among landmarks and pyramid memories to endow cross-scale information and long-range image context into landmark queries. These dynamic relations are built on the fly and specific to each image, thus they are more robust in detecting facial landmarks without salient visual features.

3.3 Dynamic Landmark Refinement

Our model directly predicts the coordinates of landmarks, so it can be regarded as a regression-based method. Compared to heatmap-based detectors, a single regression step is hard to generate competitive results. Thus, multiple cropping-thendetection steps [Trigeorgis et al., 2016] are used to improve the performance of regression-based methods in a coarse-tofine way. However, existing multi-step methods usually have two drawbacks: 1) The hard cropping operator is not differentiable with respect to the input coordinates. 2) The detection is performed only based on the cropped features, without access to the image context information. To address these challenges, our RePformer introduces the DLR module to modulate facial landmark detection into a fully end-to-end refinement process using residual coordinates predictions and dynamically aggregated queries.

Residual Coordinates Prediction. The initial facial landmark set L0 is predicted based on the highest-level memory vM. Then, our DLR appends additional predictors to higher PTH stages, and the ith predictor f iP only needs to predict the residual coordinates of landmarks U i ∈ RN×D = {ui1 , ..., uiN } with respect to Li-1 , as follows:

where uij are the residual coordinates of the jth landmark in the ith stage, and eij is the jth landmark query in the ith stage. The predictor f iP is a two-layer FFN which is agnostic to different landmarks, because it only needs to predict the residual values instead of the absolute coordinates. And the ith landmark set Li is computed by:

L1 loss is used as the loss function between the ground-truths and Li generated by the ith PTH stage. Thus, with this stepby-step refinement, our DLR gradually pushes the regressed landmarks closer to the ground-truth coordinates.

Dynamically Aggregated Queries. To adapt the transformer architecture to the residual prediction task, the landmark query set Ei , as the inputs of the ith PTH stage, need to represent the status of the current results Li-1 using their semantic and spatial information. To solve this challenge in an end-to-end way, a dynamic aggregation method is proposed to extract landmark queries by aggregating pyramid memories weighted by their relative positional information. This can be seen as a soft version of the "cropping" operator, and the jth query for stage i is computed by as follows

where k is a pixel's memory index, Ω is the domain of all indexes, and vkM-i+1 is the kth memory pixel in the stage M - i + 1. The normalized similarity sijk between query ei-1 j and memory pixel vkM-i+1 is represented by

where ∥.∥ is the squared L2 norm, and ck represents the coordinates of memory pixel vkM-i+1 . τ is a temperature parameter which is bigger than 1 to amplify the differences in coordinates. Compared with hard cropping, dynamically aggregated queries are differentiable to coordinates of landmarks, making the entire multi-step refinement procedure fully endto-end trainable. Besides, these queries are composed of the whole memory to explore both the position sensitive semantic features and global image context information.

4 Experiments

4.1 Implementation Details and Datasets

We use ResNet [He et al., 2016] pretrained on ImageNet [Deng et al., 2009] as the backbone of RePFormer, and the default depth of backbone is 18 unless otherwise specified. The Adam optimizer without weight decay is used to train our models, β1 and β2 are set to 0.9 and 0.999, respectively. All models are trained for 360 epochs with a batch size of 16. The initial learning rate is 0.0001, which is decayed by a factor of 10 after 200 epochs. The temperature τ is set to 1000. The L1 loss is used as the loss function for all outputs, and the loss weights are simply set to 1. We conduct experiments on four popular facial landmark detection datasets including WFLW [Wu et al., 2018], 300W [Sagonas et al., 2013], AFLW-Full [Koestinger et al., 2011], and COFW [Burgos-Artizzu et al., 2013]. Most of our settings follow PIPNet [Jin et al., 2021]. All input images are resized to 256x256.

4.2 Comparison with State-of-the-art Approaches

We compare RePFormer with the state-of-the-art facial landmark detection methods using the evaluation metric of normalized mean error (NME).

WFLW. Table 1 shows the performance of state-of-the-art methods and our RePFormer models with three backbones including ResNet-18, ResNet-50 and ResNet-101. Thanks to our effective detection head, on the full set of WFLW, our model with the lightweight ResNet-18 backbone already achieves better performance (4.20% NME) than all existing methods including those with much heavier backbones, such as Cascaded U-net and Hourglass. With ResNet50 and ResNet-101, RePFormer further improves results to 4.14% and 4.11% NME, outperforming the most competitive method, DAG [Li et al., 2020], by 1.7% and 2.4%, respectively. We also conduct experiments on six subsets of WFLW. Our models outperform all previous methods on five subsets, and achieve all the best and second best results, demonstrating the strong performance and robustness of our framework in various evaluation scenarios.

300W. The third-to-last column of Table 1 compares the performance of our models and state-of-the-art methods on the full set of 300W. All the reported results are normalized by the inter-ocular distances. Our RePFormers with ResNet101 and ResNet-50 backbones achieve the best and secondbest results, respectively. Only with the lightweight backbone of ResNet-18, our method already significantly outperforms most of the state-of-the-art methods, such as LAB [Wu et al., 2018], STYLE [Qian et al., 2019], HRNet [Wang et al., 2019a], and PIPNet [Jin et al., 2021]. Note that although RePFormer has a deeper detection head than detectors with specially tailored head for time efficiency, we argue that their architectures of large backbones with small detection heads are not optimal, and our RePFormer head can achieve better performance gain than using heavier backbone. Tan et al. [Tan et al., 2020] also showed that balanced architectures can yield higher accuracy-speed ratio. For example, compared to the competitive method, PIPNet [Jin et al., 2021], our RePFormer with ResNet-18 outperforms PIPNet with ResNet-101 by 3.8% with similar inference speed (56 FPS vs. 59 FPS).

COFW. We compare our methods with the state-of-the-art works on the COFW dataset under the intra-data setting which is shown in the second-to-last column of Table 1. The inter-ocular distances are used to normalize the results. Our models with three different backbones achieve the top-3 performances among all methods.

AFLW. Comparative results of our models and state-of-theart results are shown in the last column of Table 1. Following previous works, all landmark coordinates are normalized by the image size. As can be seen that our RePFormers with different backbones achieve the second-best and third-best results, while only PIPNet [Jin et al., 2021] with ResNet-101 slightly outperforms us (1.42% NME vs. 1.43% NME).

4.3 Ablation Studies

RePFormer Components. Table 2 shows the peformance of RePFormer with different componets. First, we construct a strong baseline model by appending a three-stage transformer detection head on a CNN backbone. Note that only the feature maps from the last stage of the CNN are used as the memory for the transformer, and landmark embeddings are fixed after training. Thanks to the long-range information modeling of transformer, our baseline model already achieves competitive performance compared to the state-of-the-art methods. Secondly, we apply PTH module on the baseline model. With the help of PTH, the performance of our model is improved on all sets, which demonstrates the effectiveness of pyramid memories with cross-scale information. Thirdly, we integrate DLR into our baseline model. From Table 2, we observe that DLR significantly improves the performance on the challenge subset. Finally, as shown in the last row of Table 2, our RePFormer with PTH and DLR modules achieves large performance improvements over the baseline on all subsets. Note that, compared to the baseline, the performance gain of RePFormer is even larger than the sum of improvements of all individual components, which demonstrates that the two proposed components are complementary to each other and can work together to promote more accurate regression.

Temperature Values. We evaluate the performance of RePFormer with different temperature values on the full set of 300W. τ controls the weight map for dynamically aggregated queries. As shown in Table 3, the performance is continuously improved as the τ value increases from 10 to 1000, but an excessively large τ value decreases the performance. The results indicate that dynamic landmark queries mainly aggregate information from nearby pixels, while long-range information is also valuable for generating accurate results.

5 Conclusion

In this paper, we present a refinement pyramid transformer, RePFormer, for facial landmark detection. The proposed PTH utilizes a cross-scale attention to generate pyramid memories containing multi-level semantic and spatial information. Landmark-to-landmark and landmark-to-memory attentions are employed in each PTH stage to fuse memory information into landmark queries and model long-range relationships between landmarks. A DLR module is introduced to solve the task of landmark regression by a fully end-to-end multi-step refinement procedure where the landmark queries are gradually refined by a dynamic aggregation method through top-down pyramid memories.

Acknowledgments

This work is supported by Hong Kong Research Grants Council under General Research Fund Project No. 14201620.