HifiHead: One-Shot High Fidelity Neural Head Synthesis with 3D Control

1 Introduction

Neural talking head synthesis generates images with the appearance including identity, texture and lighting from a source face and the motion (e.g., pose, expression, gaze) from a driving image, which has attracted considerable interest due to great potential usage in computer games and film industry.

Existing methods mainly focus on neural head synthesis for 2562 resolution inputs and leverage generative adversarial networks (GAN) with conditions such as facial boundary [Wu et al., 2018], keypoints [Siarohin et al., 2019b] or 3D morphable face models (3DMM) parameters [Ren et al., 2021].

* Ying Tai and Chengjie Wang are corresponding authors. https: //github.com/TencentYoutuResearch/HeadSynthesis-HifiHead

Recent approaches typically follow a paradigm of flow prediction, warping and refinement. For example, [Siarohin et al., 2019b; Siarohin et al., 2019a] first perform keypoint detection on the driving image and then generate dense flow fields from sparse keypoints. After that, the source image is warped and further fed into a generator network for refinement. However, such approaches are not applicable for high resolution scenarios since it is very challenging to directly predict accurate 5122 flow fields or synthesize high-quality images with common generator architecture.

Recently, StyleRig [Tewari et al., 2020b] and PIE [Tewari et al., 2020a] have been investigated to manipulate the latent style codes of StyleGAN [Karras et al., 2019] to achieve high fidelity image editing. Unfortunately, the results are not faithful to the desired expression or posture, and could not preserve high fidelity source appearance due to the limited capacity of latent codes.

To overcome the above defects, we propose an elegant onestage framework, termed HifiHead, to generate high fidelity talking heads via 3D face representations and generative prior embedded in StyleGAN. First, we combine the appearancerelated 3DMM coefficients from the source image with the motion-related coefficients from the driving image to construct 3D face descriptors, which play the role of latent codes in StyleGAN. To better preserve texture details and rich spatial information, we further extract hierarchical representations from the source and rendered 3D images respectively to provide accurate appearance and shape guidance.

Next, we introduce a coarse-to-fine style-based generator to fuse the appearance and shape representations along with the latent codes to render a photo-realistic output. The latent codes are broadcasted to all generator blocks to modulate the convolutional weights. The hierarchical representations are injected into the generator through spatial concatenation progressively. Considering the appearance representations need high-resolution flow fields for spatial warping, we introduce feature alignment and refinement (FAR) blocks to better exploit the facial texture and geometry details. Different from existing works splitting the flow estimation and feature refinement into two stages, the FAR block utilizes the appearance and shape features as input, generating dense flow fields and refined features simultaneously for efficiency. Note the dense flow fields are not only leveraged to warp the appearance features in the next module, but also serve as the base estimates to alleviate higher resolution flow field learning in a residual manner. Extensive experiments demonstrate our results surpass other state-of-the-art neural head synthesis methods on wild face images with extreme pose, expression, or illumination. Our contributions are summarized as follows:

HifiHead (Ours)

Figure 1: Comparison with state-of-the-art methods. Our HiFiHead is able to generate realistic face details and accurate motion.

1. High-precise: We incorporate 3DMM parameters and 3D rendered images to provide strong motion constraint for head driven. The FAR block estimates accurate optical flow for better modeling the motion, preserving better facial textures and identity in the final results.
2. High-resolution: Benefiting from the StyleGAN-based generative architecture, HifiHead easily deals with faces with 512×512 resolution, higher than FOMM or PIRender along with better source texture preservation.
3. Practical: Our HifiHead drives real-world talking heads using only one source image. StyleRig cannot be applied for real-world image editing directly, while DynamicNeRF requires video sequence as training data for each source person.

2 RelatedWork

2D-based Head Synthesis. A large number of 2D-based approaches [Zakharov et al., 2019; Ha et al., 2020; Zakharov et al., 2020] have been proposed to control head poses and facial expressions with conditional generative adversarial networks. ReenactGAN [Wu et al., 2018] feeds an adapted facial boundary into a target-specific decoder for generating the reenacted target face. X2face [Wiles et al., 2018] first generates a frontalised source face given an image set and then further transforms it to ensure desired pose and expression. However, requiring specific decoders [Wu et al., 2018] or an image set [Wiles et al., 2018] is not convenient in real-world applications and thus one shot head synthesis has drawn much attention [Siarohin et al., 2019b; Yao et al., 2021] recently. Several attempts [Siarohin et al., 2019a; Siarohin et al., 2021] generate dense flow fields from sparse keypoints detected from the driving image. After that, the input image is warped and refined to produce animated results. Unfortunately, synthesizing with a single source image is challenging especially for extreme poses and expressions.

3D Model-based Head Synthesis. 3D morphable models provide disentangled representations in respect of identity, shape, and texture of facial images, serving as plausible priors in many head synthesis algorithms [Ren et al., 2021; Wang et al., 2021b]. PIRenderer [Ren et al., 2021] and HeadGAN [Doukas et al., 2021] both utilize 3D face representations to compute dense flow fields for spatially transforming the source image and then generate a photorealistic output. Instead of using an explicit 3D graphics model, [Wang et al., 2021a] represents motion information with a 3D keypoint representation and warps source image feature in 3D space to handle extreme poses and expressions. However, these methods are mainly designed for 2562 resolution and could not preserve detailed facial appearance.

StyleGAN-based Head Synthesis. In recent years, stylebased generators [Karras et al., 2019] have greatly boost the performance of high-quality image generation and applied in a large variety of applications, including face restoration [Yang et al., 2021] and portrait attribute editing [Chu et al., 2020]. StyleRig [Tewari et al., 2020b] trains a translation network between the 3DMM's semantic parameters and StyleGAN's latent codes, thus can control over face pose and expressions of synthetically created StyleGAN images effectively. Furthermore, PIE [Tewari et al., 2020a] embeds real portrait images in the latent space of StyleGAN for intuitive editing. However, due to the limited capacity of latent codes and lack of finetune, the results are unfaithful to source image's appearance. In contrast, our HifiHead generates highresolution dense flow fields to transform the hierarchical representations of the source image and produces satisfactory results confirming to desired appearance, pose and expression.

3 Methodology

The overall architecture of HifiHead is depicted in Figure 2a. Given a source image Is and a driving image Id, the proposed HifiHead can transfer the expressions, pose and gaze of Id to the source person while preserving the appearance attributes of Is, such as identity, lighting and textures. In this section, we firstly introduce the 3D face descriptors. Then, we illustrate the modules of HifiHead in detail. Finally, we present the model objectives used to train our network.

3.1 3D Face Descriptors

Following the practice of D3DFR [Deng et al., 2019b], we leverage a deep neural network (ResNet50) to predict 3DMM coefficients from the input images:

The output is a vector z = (α, β, δ, γ, p, τ) ∈ R261 , where α ∈ R80 , β ∈ R64 , δ ∈ R80 , γ ∈ R27 , p ∈ R6 and τ ∈ R4 represent the coefficients of the identity, expression, texture, illumination, pose and gaze. The original 3DMM can not control gaze directions. We add gaze coefficient which is defined as the normalized direction vector from the center of the eye to the pupil. The desired synthesized result should maintain the appearance related attributes of Is while having the motion related attributes of Id. Therefore, the 3D face descriptors of the desired result are expressed by zdes = {αs, βd, δs, γs, pd, τd}.

Next, we compute the 3D face shape S and the albedo texture T with 3DMM,

where ¯S and ¯T denotes the mean face shape and albedo texture. Bid,Bexp and Bt are the bases of identity, expression and texture. The reconstructed 3D face is further projected onto the 2D image plane with a differentiable renderer according to the predicted illumination γs and pose pd,

Please refer to [Deng et al., 2019b] for more details.

In summary, given a source image Is and a driving image Id, the 3D face descriptors of the desired result is defined as zdes = {αs, βd, δs, γs, pd, τd}. The 3D face descriptors together with the rendered 3D face image I3d are used to condition the image generation.

3.2 HifiHead Generative Network

We carefully modify the StyleGAN2 generative architecture for effective neural head synthesis and editing.

Figure 2: The overall architecture of our method.

Input. As shown in Figure 2a, the 3D face descriptors are mapped to the latent space w ∈ W of StyleGAN2,

The mapping network depth is 3 as [Karras et al., 2021]. In order to produce faithful appearance of the source identity, we condition the generative model on the multi-scale spatial features extracted from the source image Is. Besides, we also extract spatial features from I3D to provide target motion information in addition to the 3D face descriptors,

The original StyleGAN2 requires noise inputs in each block. Instead, we replace that with spatial features F i src and F i 3d .

Architecture. Directly concatenating F i src and F i 3d is not reasonable since the expression and pose are mismatched spatially. It has been proven more effective to align the source image with the desired pose and expression through spatial deformations [Ren et al., 2021]. Different from existing works [Ren et al., 2021; Doukas et al., 2021] which both train a warping network and a refine network separately in two stages, HifiHead network estimates the spatial deformations and generates the final result simultaneously in one single stage. With this in mind, we modify the original StyleGAN2 generative block such that each block has two output layers. The tRGB layer and tXY layer convert the high-dimensional features to RGB output and dense flow, respectively. The detailed structure will be illustrated in the next section.

To summarize, the HifiHead full generative network G contains the following trainable parts, the spatial feature encoder F s enc and F 3D enc , the mapping network Fmlp and the generative blocks Fgen. Given the source image Is, the 3D face descriptors zdes and the rendered 3D image I3D, the HifiHead network generates the dense flow prediction ˆ f and image prediction ˆ I as follows,

𝒊

𝑭𝒔𝒓𝒄

𝑓 !"#$%&'( 3.3 Feature Alignment and Refinement Block

warp The detailed feature alignment and refinement (FAR) block structure is illustrated in Figure 2b. The i-th FAR block takes six inputs, including I i , f i , F iGAN , F i s , F i 3d and w. The RGB output I i , dense flow f i and the high-dimensional generative features F iGAN are generated by the previous block. F i s and F i 3d denote the corresponding spatial features extracted from Is and I3d. w represents the latent code.

First, the source feature F i src is warped to ¯ F i src according to the predicted flow filed f i . As shown in Figure 3, we visualize the predicted flow field at scale 256 and scale 512. The warped source feature aligns better with the driving image. Then, the generative feature FGAN is updated as follows:

where the operation StyleConv denotes the style convolution in StyleGAN2. The definition of "Mod" and "Demod" can be found in [Karras et al., 2020].

The RGB output I and dense flow f are updated with upsampling and skip connections. We use bilinear filtering in all up operations. The updated RGB output and dense flow are formulated as:

where tRGB and tXY are both 1 × 1 convolutional layers.

3.4 Model Objectives

Previous methods [Siarohin et al., 2019a; Wang et al., 2021a] sample two frames from the same video, one as the source image Is and the other as the driving image Id. Id also serves as the ground truth (same-identity). We observe that their motion transfer performance deteriorates when the source and driving images are different persons (cross-identity). However, cross-identity motion transfer has a wider range of practical applications. Motivated by this observation, our training data is divided into two categories:

Same-identity Data. The widely-used L1 loss L1 and perceptual loss Lp [Johnson et al., 2016] are adopted as our reconstruction loss. Adversarial loss Ladv is inherited from StyleGAN2. We also construct the 3D mesh ˆS of the generated image ˆ I. The mesh loss Lmesh is introduced to minimize the vertices' distance between ˆS and the target mesh S, formulated by Equation 2. The identity loss LID is to preserve identity during motion transfer. The pretrained face recognition model CurricularFace [Huang et al., 2020] is adopted to extract deep identity features from Is and ˆ I.

The overall model objective for same-identity training data is a combination of the above losses:

Cross-identity Data. We randomly sample two frames from different videos. The reconstruction loss is not applicable since there is no ground-truth. Ladv is still adopted since it is helpful in producing realistic results. In addition, Lmesh is used to encourage the generated image to have the same expression and pose as the driving image. LID preserves the identity information from the source image.

The overall model objective for cross-identity training data is defined as follows:

In a mini-batch, the ratio of same-identity data and crossidentity data is empirically set to 1 : 1.

4 Experiment

4.1 Datasets and Implementation

Training Datasets. We utilize the VoxCeleb [Nagrani et al., 2017] dataset, which consists of around 20K talkinghead videos, to train our HiFiHead network. The cropped videos are then resized to 512×512. A total of 17, 927 training videos and 491 testing videos are obtained.

identity and more accurate eye movements, compared with other SOTA methods.

Implementation Details. We re-train the D3DFR [Deng et al., 2019b] to predict 3DMM and gaze coefficients. The StyleGAN2 is pretrained on FFHQ [Karras et al., 2019] dataset at resolution 512×512. We clarify that we use the pretrained weights of StyleGAN for faster convergence when training HifiHead on VoxCeleb. However, similar result can still be achieved just with more iterations if trained from scratch. The spatial feature encoder contains 7 down-sample convolutional layers. The learning rate is set to 0.001 for all trainable parameters. The batch size is set to 32. It takes around 2 days to train HifiHead with 8 Tesla V100 GPUs.

EvaluationMetrics. PSNR and LPIPS [Zhang et al., 2018] are adopted to evaluate the reconstruction error. The motion transfer accuracy is measured by Exp, Angle and Gaze, which calculate the average Euclidean distances of expression, pose and gaze coefficients between the generated and target images. Identity similarity (ID) is measured by the cosine distance in deep feature space. For fair comparison, another popular face recognition model ArcFace [Deng et al., 2019a] is used to extract identity features. FID [Heusel et al., 2017] is also reported to measure visual quality.

4.2 Talking-headMotion Transfer

We compare our model with three state-of-the-art methods, including FOMM [Siarohin et al., 2019a], one-shot neural talking-head synthesis (NTHS) [Wang et al., 2021a] and PIRenderer [Ren et al., 2021]. The official released models of FOMM and PIRenderer are adopted in the experiments. NTHS is re-implemented following the implementation details provided by the paper.

Same-identity Reconstruction. We first compare image synthesis results where the source and driving images are of the same person. The quantitative evaluation is shown in Table 1. It can be seen that our HifiHead obtains comparable PSNR to other competing methods, but achieves better results on LPIPS and FID, which are better measures than PSNR for the face image perceptual quality. HifiHead also obtains the lowest Exp, Angle and Gaze scores, showing that our results have more accurate head motions and eye movements. In addition, HifiHead also preserves better identity. Figure 4 shows the qualitative comparisons. Our method can reproduce the driving images more faithfully.

Cross-identity Motion Transfer. Cross-identity motion transfer has a wider range of practical applications than sameidentity reconstruction. Besides Voxceleb, we randomly sample 1K images from the FFHQ [Karras et al., 2019] and CelebaHQ [Karras et al., 2017] dataset as the source images to compare the generalization capability with other methods. The quantitative results are presented in Table 2. Our method achieves superior performance on all metrics for all datasets. One possible explanation is that the competing methods [Siarohin et al., 2019a; Wang et al., 2021a] describe motions with person-specific sparse keypoints. The accuracy is reduced when driven by cross-identity images. In comparison, our generated results are conditioned on explicit welldisentangled 3DMM coefficients and rendered 3D images, which is less sensitive to driving subjects. Furthermore, our method manages to hallucinate realistic face details thanks to the carefully designed generative block and fine-tuning strategies. The qualitative comparisons are shown in Figure 5.

Target 3D PIRender

Ours

Figure 7: Intuitive editing with 3D control.

To further demonstrate the ability of HifiHead in preserving the identity information, we randomly sample 200 source and driving pairs. Based on these pairs, the corresponding identity similarity distribution is visualized in Figure 6, in which HifiHead significantly outperforms SOTA NTHS with 98% samples achieving higher identity similarity.

4.3 Intuitive Editing with 3DControl

In this experiment, we intuitively modify the 3DMM representations to generate images with different motions. Although many methods have been proposed for face editing, few of them are capable of intuitively changing face expressions and poses. We compare to the state-of-the-art method PIRenderer, which also achieves editing via a 3DMM.

We randomly select 100 source images and 10 target expressions and poses, which totally results in 1K editing images. We do not edit eye movements as PIRenderer does not support such editing. Table 3 shows that our method achieves better results. The qualitative comparisons are presented in Figure 7. From the first row, we can see that our HifiHead can generate realistic facial details, such as hair and eyelashes. In the second row, PIRenderer can not retain the identity information from the source image. In comparison, the identity is well-maintained in our generated results.

4.4 Comparison with Dynamic NeRF

NeRF-based methods require video sequence as training data for each specific person (e.g., Dynamic NeRF [Pumarola et Src Ours w/o LID

Ours

Figure 9: Ablation study on flow field and identity loss.

al., 2021] uses 5,000 images), while HifiHead drives talking heads using only one source image and can be applied to any person. Beside, Dynamic NeRF can not model eye blinks and eye movements, as claimed in their paper. Figure 8 illustrates the comparison with the case reported in Dynamic NeRF.

4.5 Ablation Studies

To evaluate the effectiveness of our proposed HifiHead, we conduct ablation study on two variants of our method. Variant A (w/o flow) represents removing the flow field prediction branch. The source features Fsrc are directly concatenated to the features in the GAN block. Variant B (w/o LID) denotes removing identity loss during training. Figure 9 presents the visual comparison between the variants and our full model. Without explicit dense flow prediction, model A can not generate accurate motions. On the other hand, Model B can not preserve the identity effectively. In comparison, our full model can generate much better results.

5 Conclusion

In this paper, we present a high fidelity neural head synthesis framework, termed HifiHead, to produce photo-realistic results with high quality appearance of the source image and accurate target motion of the driving image. Extensive experiments show that our method can not only achieve better same-identity reconstruction, but also generalize well to cross-identity motion transfer, significantly outperforming state-of-the-art competitors. In addition, benefiting from semantically meaningful 3DMM parameters, our proposed HifiHead allows intuitive control and editing for users.