Talking Face Generation by Conditional Recurrent Adversarial Network

1 Introduction

The talking face generation problem aims to synthesize naturally looking talking face video provided with a still facial image and a piece of audio speech. Aside from being an interesting topic from a research standpoint, it has a wide-range of applications, including, for example, video bandwidth reduction [Suwajanakorn et al., 2017], face animation, and other entertainment applications.

Nevertheless, the talking face generation problem faces extreme challenges as can be summarized from the following three perspectives. First, video generation is, in general, more challenging than still image generation since human visual system is more sensitive to temporal discontinuities in videos than spatial discontinuities in images. Second, audio speech to video generation poses extra rigid requirement on the accuracy of lip synchronization. Third, due to the large variations in human pose, talking speed and style, obtaining a video generation model with generalization capacity for unseen audio and face images is quite difficult.

∗ equal contribution, † corresponding author.

Most existing works simplify the video generation problem as a temporal-independent image generation problem, e.g., [Chung et al., 2017; Karras et al., 2017; Zhang et al., 2017], where the temporal dependency in content (i.e., face) is largely left out and the coarticulation effect cannot be adequately modeled. Other works do model the temporal dependency for smooth results. For example, [Suwajanakorn et al., 2017] modeled the dynamics of audio features through recurrent neural network (RNN). By measuring the similarity between the given probe audio feature and the gallery audio feature set which are extracted from the source video set, they can find the best matching mouth region. The matched mouth sequence and the target video are then re-timed and synthesized into the output video. Although the result seems very promising, it only works for a given person which largely restricts the generalization capability. In addition, this method only models the lip and mouth region without considering the expression or head pose variations as a whole.

Compared with these approaches, the proposed framework incorporates both image and audio in the recurrent unit to achieve temporal dependency in the generated video on both facial and lip movements, such that smooth transition across different video frames can be realized. We also observe that the image and audio features (or hybrid features) learned by minimizing the reconstruction error between the generated and ground truth frames are insufficient to accurately guide lip movement. This is because the reconstruction error only calculates the averaged pixel-wise distance, instead of semantically penalizing inaccurate lip movements. In order to guide the network to learn features related to semantic lip movements, a lip-reading discriminator is adopted to train the network in an adversarial manner. In addition, we deploy a spatial-temporal discriminator to improve both photo-realism and video-realism. Compared to previous approaches, no extra image deblurring or video stabilization procedure is needed in our framework. Our method can also be extended to model single person video with natural pose and expression. Instead of only using the hybrid feature to feed into the next recurrent unit, we also include the previously generated image frame such that the natural pose and expression of talking face can be intrinsically modeled. Our contributions are thus summarized as follows:

• We propose a novel conditional recurrent generation network that incorporates both image and audio in the recurrent unit for temporal dependency such that smooth transition can be achieved for both lip and facial movements.
• We design a pair of spatial-temporal discriminators for both image-realism and video-realism.
• We construct a lip-reading discriminator to boost the accuracy of lip synchronization.
• We extend the network to model the natural pose and expression of talking face on the Obama Dataset.

2 RelatedWork

2.1 Speech Face Animation

In recent years, several new developments have been reported, e.g., [Suwajanakorn et al., 2017] trained the lip model for only one person, i.e., President Obama. The trained model, however, is difficult to adapt to other people due to the rigid matching scheme used and the large variation in mouth movement. [Karras et al., 2017] learned a mapping between raw audio and 3D meshes by an end-to-end network. Since this method aims to generate 3D mesh animation, it cannot capture the tongue, wrinkles, eyes and head motion which are essential for image level face animation. [Taylor et al., 2017] learned the mapping between audio and phoneme categories, then matched the best predefined lip region model according to the phoneme label.

2.2 Video Prediction and Generation

Video prediction has been widely studied in many works [Mathieu et al., 2015; Oh et al., 2015; Srivastava et al., 2015; Liang et al., 2017]. It aims to predict the future frames conditioned on previous frames. With the success of GAN [Goodfellow et al., 2014], a few works have attempted to generate videos through the adversarial training procedure. [Vondrick et al., 2016] decomposed video generation into motion dynamic and background content generation. However, it is restricted to generate only fixed length videos. [Tulyakov et al., 2017; Villegas et al., 2017] disentangled the video generation problem as content and motion generation. By fixing the content representation and changing the motion latent variables, the video can be generated with motion dynamic under the same content. However, with the unconditional setting, the generated video suffers from low-resolution and fixed short length issues. To model the temporal consistency, Vid2Vid [Wang et al., 2018] proposed a sequential generation scheme where current frame depends on previously generated frame(s).

2.3 Concurrent Works

Concurrent with our paper, there have been two recent closely related works [Chen et al., 2018; Zhou et al., 2019]. [Chen et al., 2018] focuses on generating the lip region movement. However, keeping the identity across different frames while preserving the video-realism is challenging. Even if the generated lip region images can be blended into the whole face image, the noticeable inconsistency and the existence of those unrealistic static regions will still affect the video-realism. Compared to [Chen et al., 2018], [Zhou et al., 2019] designed different network architectures as well as loss functions such that the whole face is modeled. Nonetheless, the video is generated by stacking sequence of independently generated frames, thus the temporal dynamic is not well modeled which has caused strong inconsistency between frames and quite noticeable "Zoom in and out" effect. In this case, a postprocessing of "video stabilization" has to be applied to generate satisfactory results.

3 ProposedMethod

3.1 ProblemFormulation

We hypothesize that the audio to video generation can be formulated as a conditional probability distribution matching problem. In this case, the problem can be solved by minimizing the distance between real video distribution and generated video distribution in an adversarial manner [Goodfellow et al., 2014]. Let A ≡ {A1,A2, · · · ,At, · · · ,AK} be the audio sequence where each element At represents the audio clip in this audio sequence. Let I ≡ {I1, I2, · · · , It, · · · , IK} be the corresponding real talking face sequence. And I ∗ ∈ I is the identity image which can be any single image chosen from the real sequence of images. Given the audio sequence A and one single identity image I ∗ as condition, the talking face generation task aims to generate a sequence of frames ˜ I = { ˜ I1, ˜ I2, · · · , ˜ It, · · · , ˜ IK}, so that the conditional probability distribution of ˜ I given A and I ∗ is close to that of I when given A and I ∗ , i.e., p(˜ I|A, I ∗ ) ≈ p(I|A, I ∗ ).

Generally speaking, there have been two schemes developed for conditional video generation, namely, frame-toframe (Fig. 1a) and sequential frame generation as used in Vid2Vid [Wang et al., 2018] (Fig. 1b). The frame-to-frame scheme former simplifies the video generation problem into image generation by assuming the i.i.d between different frames. On the other hand, the sequential scheme generates the current frame based on previously generated frame to model the short term dependency.

For the talking face generation problem in specific where only the audio sequence and one single face image are given, it requires the generated image sequence to 1) preserve the identity across a long time range, 2) have accurate lip shape corresponding to the given audio, and 3) be both photo- and video-realistic. The video generated by the frame-to-frame scheme tends to exhibit jitter effect because no temporal dependency is modeled. The sequential generation scheme cannot preserve the facial identity in long duration because only short term dependency is modeled. To solve these issues, instead, we propose the so-called recurrent frame generation scheme, as shown in Fig. 1c, where the next frame is generated not only depending on the previously generated frames but also the identity frame to preserve long term dependency.

As illustrated in Fig. 1c, another problem with this modeling scheme is that although it considers the image temporal dependency, the audio temporal relation is still ignored. In order to solve these problems, we propose a hybrid recurrent feature generation modeling scheme to capture both the visual and audio dependency over time. We use a recurrent neural network to ingest both the image and audio sequential signal and generate the image sequence directly from a decoder network. We will explain the detailed design in the following section.

3.2 Conditional Recurrent Video Generation

In order to map independent features to a sequence of correlated ones, we apply the recurrent unit on the hybrid features to enforce the temporal coherence on both image and audio signals. Our proposed conditional recurrent video generation network is illustrated in Fig. 2. The inputs to the network are collected by the following feature extraction modules.

Audio Feature Extraction

Given an entire audio sequence A, we use a sliding window to clip the audio file into several audio segments. In Fig. 2, each At represents the Mel-Frequency Cepstral Coefficients (MFCC) features of each audio segment. We further feed each audio MFCC feature At into an audio encoder EA to extract the audio feature zA t = EA(At).

Image Feature Extraction

Frames that have the corresponding lip shapes are extracted according to the starting and ending time of each audio segment At. There are usually multiple frames correspond to one audio segment, where we simply use the middle one as the image It with the corresponding lip shape. The identity image I ∗ can be randomly selected from I. The image feature zI is calculated by an image encoder as zI = EI(I ∗ ).

A series of audio feature variables denoted as zA = [zA 0 , zA 1 , · · · , zA t , · · · , zAK ] and the image feature zI are concatenated to generate a hybrid feature where both the face and audio information are incorporated. The concatenated feature is later fed into the decoder network Dec to generate the target image with desired lip shape while preserving the same identity using the reconstruction loss Lrec.

where ˜ It = Dec(zI , zA) = G(A, I ∗ ) and It are the generated and ground truth video frames, respectively.

3.3 Adversarial Learning

Lip-Reading Discriminator

The reconstruction error alone is insufficient to accurately guide the lip movement during the training stage because it only calculates the averaged pixel-wise distance, instead of semantically penalizing the inaccurate lip movements. To solve this problem, we use a lip-reading model as a semantic lip guidance, i.e., a lip-reading discriminator, to update the generator in an adversarial manner. The objective function for updating the lip-reading model Dl is shown in Eq. 2.

where y is the real word label, the output of Dl is the predicted probability. Ii and ˜ Ii are the real and fake frame sequence where i = 1, 2 · · · ,K. By minimizing Ll, the predictions of fake frame sequences through Dl is forced to be misclassified, while the predictions of real frame sequences are pushed toward their true labels.

Spatial-Temporal Discriminator

Both image discriminator and video discriminator are adopted to improve the image quality as well as temporal realism. Although the functionality of the two discriminators overlaps in enhancing image quality, we demonstrate in Sec. 4.3 that the image discriminator is more effective in single image enhancement because it focuses relatively more on each individual frame. Meanwhile, the video discriminator helps achieve smooth transition between frames.

The image discriminator DI aims to generate realistic images. The corresponding objective is expressed in Eq. 3,

where pI and p˜ I denotes the distribution of real images in the training videos and the generated images, respectively.

The video discriminator DV works on a sequence of images/frames to mainly improve the smoothness and continuity between generated image sequences. Eq. 4 shows the objec-

Finally, the total loss for updating our generation network G is

where λI , λV , and λl are weighting parameters for the corresponding loss functions, respectively.

4 Experimental Results

We evaluate the effectiveness of the proposed method on three popular datasets (Sec. 4.1), and demonstrate our advantages over the state-of-the-art works. Specifically, the evaluation is conducted in terms of image quality, lip movement/shape accuracy, and video smoothness. Both qualitative (Sec. 4.3) and quantitative (Sec. 4.4) studies demonstrate the superior performance and generality of the proposed method in generating talking face videos 1 .

4.1 Datasets

We use TCD-TIMIT [Harte and Gillen, 2015], LRW [Chung and Zisserman, 2016], and VoxCeleb [Nagrani et al., 2017] in our experiments. TCD-TIMIT is built for audiovisual speech recognition, where the sentences contain rich phoneme categories, and the videos are captured under wellcontrolled environment. LRW is collected from the real world accompanied by truth labels (words), and the videos are short, i.e., lasting only a few seconds. VoxCeleb also contains real world videos with large variation in face pose and occlusion/overlap of faces and speech/audio, and longer duration than LRW. For the LRW and VoxCeleb datasets, we filter out those video segments with extreme facial poses or noisy audios in order to generate more stable video result and facilitate the training process. For a fair comparison and performance evaluation, we split each dataset into training and testing sets following the same experimental setting as previous works [Chung et al., 2017; Chen et al., 2018; Zhou et al., 2019].

4.2 Experimental Setup

Network Inputs

We extract the video frames to make them synchronized with audio segments. For the input/ground truth images, face regions are cropped from the videos and resized to 128 × 128. For the audio inputs, we try different window sizes for MFCC feature and find that 350ms gives the best result. For the lipreading discriminator, we only feed the mouth regions in order to avoid other interference, such as facial expressions and large head movements.

Network Architecture

The audio encoder EA, image encoder EI , image discriminator DI , and image decoder Dec are constructed by convolutional or deconvolutional networks. To capture the spatialtemporal information, we use 3D convolution to build the video discriminator DV . In order to preserve the identity, especially for unseen faces during the training, we follow the idea of U-Net to add more low level features to the image decoder Dec.

Training Scheme

We select ADAM [Kingma and Ba, 2014] as the optimizer with α = 0.0002 and β = 0.5 in the experiment. We first Corresponding ground truth frames train our network without discriminators for 30 epochs, and then add DI, DV , Dl to finetune the network for another 15 epochs. Dl is initialized from a pre-trained lip-reading model. The weights for LI, LV , and Ll are 1e-3, 1e-2, and 1e-3, respectively. Since we don't have word/sentence labels for VoxCeleb and TCD-TIMIT dataset, we only apply Ll on LRW samples during the training.

1 Code and More results: https://github.com/susanqq/Talking Face Generation

4.3 Qualitative Evaluation

Comparison with Other Methods

We qualitatively compare the proposed method with three related works, i.e., [Chen et al., 2018], [Chung et al., 2017] and [Zhou et al., 2019] in this section, as shown in Fig. 3. The input audio is saying "high education", which is randomly selected from the LRW testing set. The corresponding ground truth frames are listed in the first row, and the input faces are shown in the first image column. We feed the same input audio and faces to different methods for fair comparison. Note that [Chen et al., 2018] only works on the mouth region instead of the whole face.

As visualized in Fig. 3, the frames generated by the proposed method present higher visual quality, i.e., sharper skin texture, more realistic winkles and clearer details (e.g., teeth) than all other methods (please zoom in for better visualization), which demonstrates the effectiveness of using spatialtemporal discriminator as in our framework. The second observation is that we outperform other methods by generating sharper and more discriminative mouth shapes. One reason is due to the application of the lip-reading discriminator. Because of the usage of the lip-reading discriminator, the generator is forced to generate more accurate mouth shapes in order to predict the correct word label. Without such semantic guidance on the lip movement, the generated mouth shapes usually present less expressive and discriminative semantic cues, such as the comparison results listed in Fig. 3 from [Chung et al., 2017; Zhou et al., 2019].

In addition, video frames generated by [Chung et al., 2017] and [Zhou et al., 2019] contain inter-frame discontinuities and motion inconsistency which are more obvious in the video (provided in the supplement) than in Fig. 3. The proposed method reduces these discontinuities by performing the recurrent generation to model the temporal dynamics, which has not been considered in other methods.

Ablation Study

To compare different generation schemes and analyze the effectiveness of different loss functions in our method, we carry out extensive ablation studies as follows.

Effectiveness of different losses in the proposed method (Eq. 5) is compared and demonstrated in Fig. 4, where Lr, LI, LV , Ll represent reconstruction loss, image adversarial loss, video adversarial loss, and lip-reading adversarial loss, respectively. We use the results (first row) produced by only using Lr as the baseline. From Fig. 4, we can see that after adding the image adversarial loss LI , the generated images present more details, e.g., the teeth region becomes much clearer. Adding the video adversarial loss Lr,I,V further sharpens the images (see the teeth region as well) and smooth out some jittery artifacts between frames. Finally, the addition of lip-reading discriminator helps achieve more obvious lip movement. Although the recurrent network in our framework also aims at maintaining the temporal consistency, it plays a more important role on the global video smoothness such as avoiding pose inconsistency (e.g., zoom-in and zoomout effect).

Effectiveness of different schemes is compared in Fig. 5. We compare the recurrent generation (bottom block) with other two existing schemes, i.e., sequential generation (top block) and frame-to-frame generation (middle block) as introduced in Sec. 3.1. Obviously, the sequential generation scheme fails to preserve the identity while the frame-to-frame scheme exhibits large variance between adjacent frames as illustrated by the optical flow. The gray-scale map represents the motion intensity map 2 which is calculated by averaging the optical flow for the whole sequence, where brighter pixels illustrate larger variation between adjacent frames. Compared with frame-to-frame generator, the recurrent scheme preserves the identity information well and achieves the smooth flow between frames, i.e., most movements are around the mouth area.

4.4 Quantitative Evaluation

The same as current related works [Chung et al., 2017; Zhou et al., 2019], we use PSNR and SSIM to evaluate the image quality. To measure the lip movement accuracy, we also use lip-reading accuracy [Chung and Zisserman, 2016] and Landmark Distance Error (LMD) [Chen et al., 2018] as the measuring criteria to understand the lip movement from the semantic level and pixel level respectively. Note that [Chen et al., 2018] is excluded in our quantitative study because it only focuses on the lip region which would not be a fair comparison.

We list quantitative evaluation results as well as the comparison with other methods on the LRW dataset in Table 1. Our recurrent video generation framework has demonstrated better performance over other frame-by-frame generators.

We also list the result of our method using different loss functions. Our key observation is that by applying discriminators, especially the image and video discriminators, both the PSNR and SSIM values are negatively affected, though they greatly improve the visual quality by adding more details which may not appear in the original images. The reason for this phenomenon is that both PSNR and SSIM calculate pixel level differences which cannot well reflect the visual quality. This phenomenon has been studied in many super-resolution related works. Similarly, we observe that the best LMD value is achieved by using only the reconstruction loss. We argue that LMD cannot precisely reflect the lip movement accuracy mainly from two aspects. First, for the same word, different people may pronounce it in different ways, e.g., very exaggerated mouth opening range. The generated samples are not necessarily to be the same way as the ground truth. Second, this may be caused by the landmark detection error especially for non-frontal faces. In other words, PSNR, SSIM and LMD cannot accurately reflect whether the generated lip movement is correct or not.

2 Assume (u, v) = opticalflow(I1, I2) is the optical flow between two continuous frames, the motion intensity for each pixel is calculated by u2 + v2 .

For better evaluation of the lip sync result, we use thestate-of-the-art deep lip-reading model which is trained on real speech videos to quantify the accuracy. Computer vision based lip-reading models have been used to aid correction of lip movements to improve human's pronunciation [Wei, 2014], and thus should be an ideal metric to justify the authenticity of the generated speech videos. We find with lipreading discriminator, the lip-reading accuracy is improved from 18.5% and 38.0% to 36.2% and 63.0% on top1 and top5, which approaches the accuracy on real videos (60% and 80% top1 and top5).

4.5 User Study

We further conduct the user study for more subjective evaluation.We choose Amazon Mechanical Turk (AMT) as the user study platform and all the workers are required to be from either "UK" or "USA". In the study, workers are asked to perform pair-wise comparison between videos generated by [Chung et al., 2017], [Zhou et al., 2019], and ours. Human evaluation is very subjective and different people may focus on different perspectives. Therefore we give clear user instructions, and ask people to evaluate the result from three perspectives: whether the generated video has good smoothness like a real video (realistic of video); whether the lip shape is accurate and synchronized well with the audio (lip movement accuracy); and whether the image frames are blurred or with artifacts (image quality). In total, each worker is asked to evaluate 60 test samples from the VoxCeleb dataset and the LRW dataset, each of which has been used to generate 3 videos by three compared methods. For each test sample, videos generated by our method and a compared method are provided to workers, and they are asked to select the better video according to the instruction. Each sample is evaluated by 10 different workers. We summarize the result and list it in Table 2.

As shown in Table 2, our method clearly outperforms recent approaches [Chung et al., 2017; Zhou et al., 2019]. In specific, our method outperforms the other two methods on lip movement accuracy which is also consistent with the lipreading model result in Table 1. In addition, our framework is superior on the video realistic metric as well as image quality since other methods suffer from lots of motion artifacts like pose discontinuity and unstable face sizes between frames.

5 Study on Natural Pose and Expression

One potential problem for the proposed structure and other concurrent works is that it is difficult to generate the talking face video with natural poses and expressions using 2D method. We observe that instead of only using the hybrid features as inputs to the next recurrent unit, we also include the previously generated image frame such that the natural pose and expression of talking face can be intrinsically modeled. This modification actually integrates the sequential generation scheme and our recurrent generation scheme. Recall that the sequential generation scheme alone will cause changing face issue when generalized to different people, but will not be a problem for single-person video generation. We train this modified neural network on the Obama dataset [Suwajanakorn et al., 2017]. As shown in Fig. 6, our generated video exhibits accurate lip shape and natural pose change.

6 Conclusion, Limitations and FutureWork

We presented a new conditional adversarial network for talking face video generation. Our framework mainly utilizes the recurrent adversarial network to capture the temporal dependency of sequential audio features and image information simultaneously. Furthermore, the framework incorporates three discriminators to further improve the image quality, video realism and lip movement accuracy in an adversarial training manner. Our extensive experimental results obtained from public constrained and unconstrained data demonstrated the superiority over state-of-the-arts under different performance metrics.

One of the areas that deserve further investigation is the design of a true end-to-end generation framework, where the raw audio file instead of the MFCC features is used as input to the network. The other area is the incorporation of a sentence-level lip-reading discriminator for guiding better lip motion generation. To improve image quality, the superresolution [Zhang et al., 2019] work can be incorporated.