HifiHead: One-Shot High Fidelity Neural Head Synthesis with 3D Control

HifiHead: One-Shot High Fidelity Neural Head Synthesis with 3D Control

Feida Zhu, Junwei Zhu, Wenqing Chu, Ying Tai, Zhifeng Xie, Xiaoming Huang, Chengjie Wang

Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence
Main Track. Pages 1750-1756. https://doi.org/10.24963/ijcai.2022/244

We propose HifiHead, a high fidelity neural talking head synthesis method, which can well preserve the source image's appearance and control the motion (e.g., pose, expression, gaze) flexibly with 3D morphable face models (3DMMs) parameters derived from a driving image or indicated by users. Existing head synthesis works mainly focus on low-resolution inputs. Instead, we exploit the powerful generative prior embedded in StyleGAN to achieve high-quality head synthesis and editing. Specifically, we first extract the source image's appearance and driving image's motion to construct 3D face descriptors, which are employed as latent style codes for the generator. Meanwhile, hierarchical representations are extracted from the source and rendered 3D images respectively to provide faithful appearance and shape guidance. Considering the appearance representations need high-resolution flow fields for spatial transform, we propose a coarse-to-fine style-based generator consisting of a series of feature alignment and refinement (FAR) blocks. Each FAR block updates the dense flow fields and refines RGB outputs simultaneously for efficiency. Extensive experiments show that our method blends source appearance and target motion more accurately along with more photo-realistic results than previous state-of-the-art approaches.
Keywords:
Computer Vision: Neural generative models, auto encoders, GANs  
Computer Vision: 3D Computer Vision
Computer Vision: Applications