SyncAnimation: A Real-Time End-to-End Framework for Audio-Driven Human Pose and Talking Head Animation

1 Introduction

In recent years, audio-visual synthesis techniques have garnered significant attention, with audio-driven realistic avatar generation emerging as a key research focus. Over the past few years, many researchers have employed GAN- or SD-based deep generative models to tackle this task [Prajwal et al., 2020; Zhang et al., 2023b; Zhong et al., 2023; Zhang et al., 2023a; Xu et al., 2024; Wang et al., 2024; Chen et al., 2024; Xie et al., 2024; Tan et al., 2025]. Among them, SD-based models, leveraging model parameters and large-scale datasets, can generate fully animated avatars from a single reference image. However, their reliance on largescale and diverse individual datasets, coupled with overwhelming computational and time costs, limits their applicability in real-time scenarios such as live streaming or video conferencing, where high fidelity and real-time rendering are essential [Guo et al., 2024; Hu, 2024].

∗Corresponding author.

Recently, Neural Radiance Fields (NeRF) have been applied to audio-driven talking avatars [Mildenhall et al., 2021; Guo et al., 2021; Li et al., 2023; Ye et al., 2023b; Kim et al., 2024; Peng et al., 2024; Ye et al., 2023a]. By associating NeRF with audio either end-to-end or through intermediate representations, these methods enable the reconstruction of personalized talking avatars with impressive synthesis quality and inference speed. However, existing approaches, such as ER-NeRF [Li et al., 2023] and Synctalk [Peng et al., 2024], focus primarily on achieving precise synchronization between lip movement and audio, given strong correlation. Nevertheless, they have yet to tackle the mismatch between audio and head poses, as well as the challenging association between audio and facial expressions [Zhang et al., 2023a], ultimately reducing the realism of the generated avatar.

In this paper, we highlight the importance of generating realistic talking avatars driven by audio, focusing on identity consistency and detail preservation, while facing three critical challenges that require further attention, shown in Fig. 1: (1). Pose inconsistency with audio: Generate identical and fixed poses across different inferred audios (derived from original video frames), possibly even exhibiting exaggerated head movement in silent segments. (2). Expression inconsistency with audio: Insufficient attention is given to facial animation elements beyond lip-syncing, such as eyebrow movement and blinking, which are crucial to conveying natural expressions and emotional depth, resulting in unnatural and stiff animations. (3). Loss of paste-back ability in audio-driven pose method: Only the head is generated, and the edges of the body are generated with changes, causing the torso to be displaced and therefore cannot be attached back to the origin torso. As mentioned above, audio-driven pose generation methods cannot achieve upper-body generation, and the lack of audio-driven expression generation results in inconsistent expressions. Thus, a fully generative NeRF-based approach with strong audio correlation is essential for achieving realistic talking avatars.

To address these critical challenges, we propose SyncAnimation, a NeRF-based framework focused on audio-driven rendering of upper body and head . This framework integrates three core modules: the AudioPose Syncer and AudioEmotion Syncer, which enable stable, precise, and controllable mappings from audio to head poses and facial expressions, and the High-Synchronization Human Renderer, which ensures seamless integration of head motion and upper-body movement without post-processing. Together, these modules form a unified solution for generating dynamic, expressive, and highly audio-synchronized avatars. In summary, themain contributions of our work are as follows:

• We propose SyncAnimation, a rendering framework for audio-driven expression and upper body generation. This framework generates an avatar that is highly consistent with the audio and displays a diversity of actions, while supporting both one-shot and zero-shot inference.
• We introduce the Audio2Pose and Audio2Emotion modules to support end-to-end efficient training, enabling high-precision poses and expression generation, and progressively generating audio-sync upper body, head, and lip shapes.
• Our algorithm achieves 41 FPS inference on an NVIDIA RTX 4090 GPU, and to our knowledge, this is the first real-time audio-driven avatar method capable of generating audio-sync upper body movement and head motion.
• Extensive experimental results show that SyncAnimation successfully generates realistic avatars with the same scale as the original video and significantly outperforms existing state-of-the-art methods in both quantitative and qualitative evaluations.

2 RelatedWork

2.1 Paste-Back Generation

GAN- and NeRF-based methods are representative of pasteback generation. Among them, GAN-based talking head synthesis has primarily focused on generating video streams for the lip region, creating new visual effects for talking head avatar[Cheng et al., 2022; Zhang et al., 2023b; Zhong et al., 2023; Tan et al., 2025]. For example, Wav2Lip [Prajwal et al., 2020] introduced a powerful lip-sync discriminator to supervise lip movement and penalize mismatched mouth shapes. IP-LAP [Zhong et al., 2023] proposed an audio-to-landmark generator and a landmark-to-video model, using prior landmark and appearance information to reconstruct lips from a reference image. Recently, EdTalk presented an effective disentanglement framework, using orthogonal bases stored in a dedicated library to represent each spatial component for efficient audio-driven synthesis. With the rise of NeRF, earlier works [Guo et al., 2021; Ye et al., 2023b; Li et al., 2023; Peng et al., 2024] have integrated NeRF into the task of synthesizing talking heads, using audio as the driving signal. For instance, AD-NeRF [Guo et al., 2021] was the first to render both the torso and head but suffered from poor generalization, and the synthesized lip movement sometimes appeared unnatural. ER-NeRF [Li et al., 2023] innovatively introduced tri-plane hash encoders and a region attention module, advocating a fast and precise lip-sync rendering approach. Geneface [Ye et al., 2023b] and SyncTalk [Peng et al., 2024] generated a generalized representation based on extensive 2D audiovisual datasets, ensuring synchronized lip movement across different audios. The GAN-based methods map audio to lip-sync but paste other parts, causing blurry lips (Fig.5). In contrast, NeRF-based methods perform fullface synthesis but fail to synchronize audio with facial expressions, head poses, and upper-body movement.

2.2 Joint Generation

Leveraging the fundamental principles of text-to-image diffusion models, recent advances in video generation based on diffusion techniques have shown promising results [Xu et al., 2024; Wang et al., 2024; Chen et al., 2024]. VExpress [Wang et al., 2024] effectively links lip movement, facial expressions, and head poses through progressive training and conditional dropout operations, enabling precise control using audio. Hallo [Xu et al., 2024] adopts a hierarchical audio-driven visual synthesis approach, achieving lip synchronization, expression diversity, and pose variation control. EchoMimic [Chen et al., 2024] employs a novel training strategy that incorporates both audios and facial landmarks for avatar synthesis. The SD-based methods generate audiodriven talking avatars with poses and facial animations. However, large-scale training often results in poor resemblance to the original individual and comes with high computational costs, making real-time applications like live streaming and video conferencing challenging. For instance, generating a one-minute video can take up to half an hour. Additionally, these methods struggle with audio-motion mismatches when handling out-of-domain audio, further limiting their suitability for real-time use.

3 Method

In this section, we propose SyncAnimation, a generative, audio-driven model for creating avatars with dynamic head motion and upper-body movement in Fig. 2. The framework includes three main components: (1). the AudioPose Syncer for accurate audio-to-head pose mapping in Sec. 3.1, (2). the AudioEmotion Syncer for controllable, audio-driven facial expressions in Sec. 3.2, and (3). the High-Synchronization Human Renderer for seamless upper-body generation without head pasting in Sec. 3.3.

3.1 AudioPose Syncer

Audio-aware Poses Generation. Learning a model to control head motion from audio is challenging due to two main issues: (1). The rotation matrix must be orthogonal with a determinant of 1, but direct prediction can lead to nonorthogonal matrices, causing errors in rotation and pose reconstruction. (2). There is inherent ambiguity between audio and motion, leading to jitter or frame skipping in the generated avatar. We propose AudioPose to address these issues, as shown in Fig.3. Considering the difficulty in predicting orthogonal rotation matrices with a high-dimensional compact representation, we transform the task into predicting three euler angles (roll α, pitch β, and yaw γ). Given rotation matrix R, the conversion formula is:

Figure 4: Audio2Emotion reconstructs 3DMM expression offsets from audio and monocular input. To address weak expressionaudio correlation and blinking periodicity, we replace the diversity template with a conditional VAE guided by time Tτ and contextdependent audio features Aseq.

Rij denotes the element in the i-th row and j-th column of the rotation matrix R. The euler angles e = (α, β, γ) and the translation t = (x, y, z) jointly define the head poses, controlling its orientation and position.

The non-linear transformation of the rotation matrix introduces new challenges, such as gimbal lock (β ≈ ±90◦ ) causing non-invertibility, and pose variations influenced by euler angles and translations within narrow ranges. In real-world videos, head poses are confined to small ranges, not extending to extreme values like ±90◦ or ±180◦ . This insight is incorporated into training to address these challenges. Specially, the prediction range is restricted from (e, t) ∈ (-∞,+∞) to (∆e,∆t) ∈ (¯e±∆, ¯t ±∆), where ¯e and ¯t are the average poses, and ∆ defines the deviation. To ensure consistency, outputs are converted into a normalized distribution ( ¯ ∆e, σ∆e), standardizing the deviation range as follows:

To address the ambiguity between audio and pose in issue 2, we introduce two conditional vectors Spose and Dpose. The overall Audio2Pose prediction pipeline is as follows:

Where F(·) represents the pose generation model. Both Dpose and Spose are conditional vectors, with Spose enhancing pose stability and Dpose guiding diversity. For g(a), we use FaceXHuBERT [Haque and Yumak, 2023] as the audio encoder, as it is more suited than pre-trained models like DeepSpeech and wav2vec [Hannun et al., 2014; Baevski et al., 2020], which lack person-specific information.

Audio-only poses prediction (a → Off (e)aud , Off (t)aud ) suffers from ambiguity. To address this, we introduce an conditional vector Spose to enhance pose stability by providing reference poses, reducing the solution space from oneto-many to many-to-one. During training, the regression loss updates Spose, which is adaptively adjusted to resolve ambiguities. Specifically, at the τ-th frame, gaussian noise is added to the poses of the (τ - 1)-th frame, which is then encoded through Multilayer Perceptrons (MLPs) to generate Spose.

where Off (eτ-1) and Off (tτ-1) represent the pose information of the (τ -1)-th frame, and N(µS, δS) denotes gaussian noise with mean µS and standard deviation δS.

To generate stable yet diverse poses, We further introduce a diversity-guided conditional vector Dpose, which injects uncertainty and provides diverse pose templates via Gaussian sampling. VAE backpropagation improves convergence and accuracy over single audio regression (Exp. 4.5). To normalize the Dpose vector space, we apply the KL-divergence loss:

where N(µD, δ2D ) ∈ RD is the latent distribution, and N(0, I) is the standard normal distribution. For the poses generation model F(·), we use the U-Net architecture [Qian et al., 2021; Wang et al., 2024], which extracts audio features and incorporates Dpose and Spose. The U-Net generates a shared feature representation, which is then processed by MLPs to predict rotation Off (e)aud and translation Off (t)aud . Reconstruction loss Lreg is applied to align the generated poses with the ground truth.

Therefore, the final poses generation loss is:

where λKL and λreg are the weights for the VAE and regression loss terms, respectively.

Audio-guided Upper-body Generation. We generally follow the rendering process in preceding work [Guo et al., 2021; Li et al., 2023]. Originated from the representation of neural radiance field, the implicit function is defined as FNeRF : (x, d) → (c, σ), where x = (x, y, z) refers to 3D spatial location and d is the viewing direction. The output c and σ determine color value and corresponding density.

To render high-fidelity and realistic scene efficiently, we employ a 2D-multiresolution hash encoder introduced by [Li et al., 2023]. A specific encoder designed for position x = (x, y, z) projected on plane AB is defined as HAB : (a, b) → hAB ab , where a and b are projected coordinates, while hAB ab ∈ RLD represents L levels of features with D dimensions.

To ensure dynamic upper-body generation and consistency between head motion and upper body movement, the audiopredicted Off (e)aud and Off (t)aud are recovered to space representations eaud and taud via inverse normalization. Subsequently, three trainable coordinates ˜X are deformed as rendering conditions. The implicit function is defined as:

where Ht denote the hash encoder. During training, we optimize the upper-body model by minimizing the error between ˆ C(r) and genuine pixel color C(r) through:

3.2 AudioEmotion Syncer

Lifelike Expression Prediction. Audio-driven research often focuses on lip-sync, neglecting other facial expressions with weaker audio correlation [Tian et al., 2025], leading to unnatural expressions. We use Arkit face blendshapes, semantically rich 3D coefficients, to model the upper face region b [Peng et al., 2023]. However, for the eyes, b reaches extreme values (0 or 1) during full eye closure or opening. To enable full eye closure, we predict the offsets Off (b) relative to the average ¯b and incorporate stability constraints Sexp and diversity guidance Dexp (Fig.4).

Blinking relies on temporal patterns, making it hard to model and often causing unnatural behavior like missing blinks. To address this, we enhance the VAE within the diversity-guided Dexp to capture temporal dependencies by incorporating time features Tτ and sequential audio Aseq, enabling the CVAE to generate Dexp with strong temporal correlations.

where Tτ is encoded via sinusoidal functions and processed by an MLP. Aseq is derived by integrating audio data from n neighboring frames {aτ-n, . . . , aτ+n}, and processed through stacked convolutional layers to generate temporally contextualized representations that ensure dynamic expression continuity. Given the weak audio-expression correlation, we enhance the expression reference by modifying the input to the stability vector Sexp. Specifically, the previous expression bτ-1 and gaussian noise N(µS, δS) are replaced by the current frame's expression bτ .

For the audio-driven expression model, we optimize Lexp using the following loss function:

where λKL and λreg are weights applied to the CAVE and regression loss terms.

Dynamic Head Rendering. We generate realistic head poses using a technique similar to Sec.3.1. The audiopredicted Off eaud, Off taud, and Off baud are converted to eaud, taud, and baud in the original space via inverse normalization. Due to the complexity of head generation, we use three 2D hash encoders, each operating on a specific plane. The final geometric feature is obtained by concatenating the outputs from all three planes. Predicted coefficients control expressions, and audio features enhance head motion prediction. The implicit function is then defined as:

Here, ah is the audio representation extracted by Hubert [Hsu et al., 2021], distinct from the one used for predicting poses and expressions due to different task requirements. The corresponding loss Lhead aligns with Lupper-body.

3.3 High-Synchronization Human Renderer

Facial-aware attention. To better utilize audio and expression information in different facial regions, we apply a channel-wise attention mechanism [Li et al., 2023]. From the hash encoders' output, we obtain attention weights for audio and expression using two MLPs: Attnaud and Attnexp.

By applying hadamard product we attain region-aware features ah,x = ah,x ⊙ vaud and bout,x = bout ⊙ vexp. Such operation makes sure model can explore useful information in a disentangled way and thus increase rendering quality.

Fine Lip Optimization. Though audio feature benefits holistic head rendering, we seek to manually augment attention weight on lip region due to more relevance. By utilizing mask technique [Peng et al., 2024], we lower the attention weight out of lip area. Meanwhile, we use LPIPS loss focused on lip zone to gain finer result. The process is expressed as:

4 Experiments

4.1 Implementations

Dataset. For fair comparison, the experimental dataset was obtained from publicly available video collections in [Guo et al., 2021; Li et al., 2023; Peng et al., 2024] and HDTF [Zhang et al., 2021]. We collected well-edited video sequences in English, French, and Korean, with an average of 6665 frames per video at 25 FPS. Each raw video was standardized to 512×512, with the a center portrait.

Quantitative Evaluation Metrics We evaluate our method using widely adopted metrics. For image quality, we employ full-reference metrics, including Peak Signal-to-Noise Ratio (PSNR) [Hore and Ziou, 2010], Learned Perceptual Image Patch Similarity (LPIPS) [Zhang et al., 2018], Structural Similarity Index Measure (SSIM) [Wang et al., 2004], and Frechet Inception Distance (FID) [Heusel et al., 2017]. In terms of lip and face synchronization, we utilize landmark distance (LMD) to measure the synchronicity of facial movement [Chen et al., 2018], and Action Unit Error (AUE) to evaluate the accuracy of facial expressions [Baltruˇsaitis et al., 2015]. Furthermore, we introduce Lip Sync Error Confidence (LSE-C), consistent with Wav2Lip, to evaluate the synchronization between lip movement and audio [Prajwal et al., 2020].For the diversity of head motion generated, a standard deviation of the embeddings of head motion features is extracted from the generated frames using Hopenet [Ruiz et al., 2018]. For eye blink detection, The Eye Aspect Ratio (EAR), calculated based on the positions of facial landmarks around the eyes, is utilized to evaluate the naturalness and accuracy of generated eye movement [Soukupova and Cech, 2016].

Comparison Baselines. We compare SyncAnimation with two SOTA methods for lip synchronization and motion generation. For lip synchronization, we include GAN-based methods like Wav2Lip [Prajwal et al., 2020], DInet [Zhang et al., 2023b], and IP-LAP [Zhong et al., 2023], as well as NeRFbased methods such as SyncTalk [Peng et al., 2024], ADNeRF [Guo et al., 2021], ER-NeRF [Li et al., 2023], and GeneFace++ [Ye et al., 2023a]. For motion generation, we compare SyncAnimation with three large-scale stable diffusion models: V-Express [Wang et al., 2024], Hallo [Xu et al., 2024], and EchoMimic [Chen et al., 2024].

Implementation Details. We train SyncAnimation with separate steps for upper-body generation, head rendering, and lip-audio synchronization, using 15k, 12k, and 4k steps, respectively. Each iteration samples 2562 rays and employs a 2D hash encoder with parameters L = 14 and F = 1.

The AdamW optimizer is used, with learning rates set to 0.01 for the hash encoder and 0.001 for the Audio2Pose and Audio2Emotion modules. The total training time is approximately 4 hours on an NVIDIA RTX 4090 GPU.

4.2 Quantitative Evaluation

Compare with baseline. We first compare one-shot SyncAnimation (SyncAnimation-One) with several state-of-theart methods using the first frame and test audio for driving. As shown in Tab.1, SyncAnimation-One outperforms GAN-, NeRF-, and SD-based approaches. (1). In terms of image quality, SyncAnimation-One achieves superior results in PSNR, LMD, and FID, indicating better detail preservation. (2). For synchronization, SyncAnimation-One surpasses other methods on LMD and AUE, demonstrating excellent audio-lip synchronization. Although SyncScore lags behind GAN-based methods due to SyncNet training loss, it still outperforms NeRF-based methods. (3). SyncAnimationOne significantly outperforms GAN-, NeRF-, and even SDbased methods and achieves the lowest EAR error.

Our framework supports both one-shot and zero-shot inference. In zero-shot mode (SyncAnimation-Zero), reference poses and blendshapes are replaced with gaussian noise. As shown in Tab.1, SyncAnimation-One (using the first frame as reference) and SyncAnimation-Zero (using gaussian noise) perform similarly across all metrics and outperform other methods, demonstrating that SyncAnimation operates independently of reference inputs for identity or information, relying solely on audio cues for dynamic rendering.

4.3 Qualitative Evaluation

In the previous section, we quantitatively demonstrated the superiority of SyncAnimation. In this section, we visually assess the quality of generated frames by comparing GAN-, NeRF-, and SD-based methods with SyncAnimation, as illustrated in Fig.5. Unlike other methods that render the upper body but fail to synchronize movement with audio, SyncAnimation excels by producing audio-synced upper-body movement. Additionally, GAN- and NeRF-based methods struggle with upper facial expressions during one-shot inference. Although SD-based methods generate diverse facial expressions, their reliance on large-scale training often fails to capture fine details like eyes and lips, causing issues such as asymmetry, overexposure, missing teeth, unfocused eyes, and unnatural lip closures. In contrast, SyncAnimation preserves the subject's identity with superior fidelity and resolution, accurately reproducing subtle actions like blinking and eyebrowmovement, thanks to the AudioEmotion Syncer. For a more comprehensive comparison, we recommend watching the supplementary video.

4.4 Upper-body Scaling Expansion

In real-time applications (e.g., news broadcasting and live teaching), individuals often appear in upper-body. Existing methods [Ye et al., 2023a; Peng et al., 2024] often paste the rendered head onto the original torso, resulting in unnatural avatars and limiting applicability. There is a growing demand for direct upper-body rendering that is highly synchronized with audio.

We depart from the conventional approach of restricting upper-body scale and pasting back to the original frame. SyncAnimation directly renders upper-body avatars driven by audio, progressively increasing the upper body proportion in the rendered images. Image quality and audio consistency metrics are shown in Tab.2. As the upper-body scale increases,

overall image quality improves, but lip-sync consistency and head motion diversity decline. This suggests that while the upper body is easier to render, a higher upper-body proportion improves image quality, but reduces focus on the head. The EAR remains stable due to the periodic time features in Sexp from Audio2Emotion. As shown in Fig.6, we scale the rendered avatar to match the original video's size, demonstrating that SyncAnimation can generate upper-body avatars with strong audio correlation and natural poses.

4.5 Ablation Study

In this section, we report ablation studies under the joint generation setting to validate the effectiveness of our major contributions from two perspectives. Additionally, to demonstrate the adaptability of the SyncAnimation to Outof-Domain (OOD) audio, we evaluate the impact of different backbones on rendering structures under external audio conditions. The results are presented in Tab.3 and Fig.7.

Effectiveness of Dpose and Spose

The Audio2Pose model is key to generating stable and natural free-form poses. To demonstrate the effectiveness of the diversity-guided conditional vector Dpose and stabilityguided vector Spose, we evaluate the density metric in static and dynamic audio, as well as the absolute error at final convergence. As shown in Tab. 3, incorporating Dpose and Spose either individually or together consistently produces natural avatar. However, the absence of Spose leads to unstable head generation with frequent jittering, resulting in an increased density metric. On the other hand, excluding Dpose results in larger absolute poses errors, and the diversity in silent segments becomes excessively high, making it difficult to achieve the desired small-scale head motion.

Effectiveness of Dexp and Sexp

In the Audio-driven talking avatar, we evaluate the impact of the proposed Dexp and Sexp on facial expressions, specifically focusing on blinking behavior. The evaluation metric is the EAR (Eye Aspect Ratio) calculated for each frame, along with line plots of blendshapes for the left and right eyes predicted by the Auido2Pose model. As shown in Fig.7, when Dexp or Sexp is not incorporated, the rendered image exhibits either a single blink over the 22-second driven audio or incomplete eye closures despite EAR fluctuations. Under the combined effect of Dexp and diversity-guided Sexp, periodic fluctuations are guided by Dexp, while Sexp accentuates each peak (as illustrated in the comparison between (b) and (c)), resulting in more natural and rhythmic blinking motion.

5 Conclusion

In this paper, we propose SyncAnimation, which aims to achieve three key objectives for modern talking avatars: generalized audio-to-head poses matching, consistent audioto-facial expression synchronization, and jointly generative upper-body avatars. Our framework integrates AudioPose Syncer, AudioEmotion Syncer, and High-Synchronization Human Renderer modules. These components enable the generation of stable synchronized poses, upper-body movement, and realistic expressions while preserving subject identity, even from monocular or noisy inputs. Extensive experiments demonstrate that our method achieves the three goals of modern talking avatars and outperforms existing methods.

Acknowledgements

Yujian Liu and Shidang Xu have made equal contributions to SyncAnimation.