ExpTalk: Diverse Emotional Expression via Adaptive Disentanglement and Refined Alignment for Speech-Driven 3D Facial Animation

1 Introduction

Speech-driven 3D facial animation aims to generate natural and realistic 3D facial animation highly aligned with the corresponding speech. With the rapid development of deep learning, existing speech-driven 3D facial animation methods efficiently generate relatively accurate facial expressions of spoken language through lip synchronization [Cudeiro et al., 2019; Fan et al., 2022; Peng et al., 2023a; Xing et al., 2023; Fan et al., 2025]. Thanks to significant performance, these methods exhibit promising potential for application across a spectrum of fields, including digital avatars, virtual reality, interactive entertainment, and online meetings [Morishima, 1998; Tanaka et al., 2022].

* Corresponding authors.

Despite significant progress, many existing methods mainly focus on high-quality synchronization of facial language expressions, but overlook the fine-grained generation of facial emotional expressions. Such subtle facial emotions are another core component of facial expressions apart from facial spoken language [Murray and Arnott, 1993], playing a crucial role in daily interactions. Recognizing this, some methods [Danˇeˇcek et al., 2023; Peng et al., 2023b] attempt to utilize a single emotional aspect in speech to aid in synthesizing the emotion in a complete facial animation. However, in the real world, human emotions are complex and diverse, which results in multiple emotions present in each complete speech needing to be reflected in the facial expression. For example, upon receiving an unexpected gift, individuals typically experience a moment of surprise quickly transitioning into happiness, a shift synchronously reflected in their speech and facial emotions, as shown in Figure 1. Rigid facial expressions aligned with the spoken language in speech but failing to fully match diverse speech emotions may potentially trigger the uncanny valley effect [Mori et al., 2012].

To tackle the above issues, we propose a method for 3D facial animation generation, which decouples spoken language and emotions from the original facial expressions, followed by detailed refinements based on the corresponding speech information to produce new expected facial expressions. Our method has two goals: (1) Facial Expression Decoupling.

The facial expressions of spoken language and emotions are deeply fused. When individually refining them based on the corresponding information in speech, they tend to mutually interfere each other. Especially, when dealing with complex emotional transitions requiring detailed refinements, such adverse interference may be particularly significant. Thus, before detailed refinements, it is essential to decouple spoken language and emotions in facial expressions. (2) Facial Expression Refinement. When transferring facial expressions of spoken language and emotions, it is essential to make subtle refinements to numerous facial vertices one by one. This demand becomes more pronounced, particularly when swiftly transitioning between emotions. For this demand, we design deep damage to the facial features and deep reconstruction techniques to refine facial expressions.

Technically, we introduce the ExpTalk model for speechdriven 3D facial animation to achieve each objective, which consists of two target designs: (1) Adaptive Disentanglement VAE (AD-VAE). This module decouples the original facial expressions into two types of quantization codebooks, encapsulating spoken language and emotion priors. Leveraging extracted speech emotions, we adopt fine-grained contrastive learning to drive the training of AD-VAE for adaptive encodings disentangling. (2) Refined Alignment Diffusion (RAD). This module first utilizes the noise-injection encoder to make feature-dimension compression and iterative perturbations for facial priors extracted by AD-VAE. Later, RAD extracts speech features aligned with facial priors from multiple dimensions. With the refined speech features, we completely reconstruct facial priors to produce expected facial expressions synchronized with speech.

Our contributions are as follows:

• To the best of our knowledge, we take the early exploration of the emotion-shift problem in speech-driven 3D facial animation and propose ExpTalk to decouple the emotions from facial expressions, enabling fine-grained refinement of facial emotions based on speech emotion.
• We propose Adaptive Disentanglement VAE to decouple the facial features via contrastive learning, preventing mutual interference in the subsequent refinement of each facial prior. We propose Refined Alignment Diffusion to iteratively damage and reconstruct facial priors to align with corresponding speech.
• Extensive experiments prove the effectiveness of our ExpTalk model, demonstrating its potential and effectiveness in practical applications.

2 RelatedWork

2.1 Speech-Driven 3D Facial Animation

Recent work on speech-driven 3D facial animation methods [Cudeiro et al., 2019; Peng et al., 2023a] has achieved significant progress in generating highly synchronized lip animations based on speech. FaceFormer [Fan et al., 2022] adopts a transformer-based [Vaswani, 2017] model to capture the relevant speech information, autoregressively generating continuous facial animations. CodeTalker [Xing et al., 2023] leverages VQ-VAE to learn a discrete code space and employs a temporal autoregressive model to sequentially synthesize facial animations from input speech signals. UniTalker [Fan et al., 2025] introduces a unified multihead model to address annotation inconsistencies, enabling 3D facial animation across multi-domain datasets. However, these methods fail to adequately consider the characteristics of speech in conveying emotions, resulting in facial animations that still lack expressiveness.

To address this, recent studies [Kim et al., 2024; Xie et al., 2025] have incorporated emotional information as an additional condition to create emotional facial animations. EmoTalk [Peng et al., 2023b] utilizes an emotion disentangling encoder with a cross-reconstruction loss to decouple emotion and content from speech, followed by an emotionguided multi-head attention decoder to generate facial animations. EMOTE [Danˇeˇcek et al., 2023] employs a temporal variational autoencoder to learn motion priors and uses annotated emotional vector labels with a transformer encoderdecoder structures for animation generation. Despite these advances, these methods still struggle with refined alignment with real speech emotions.

2.2 Probabilistic Mapping of DiffusionModels

Recent advances in probabilistic modeling have enabled more effective cross-modal alignment in various domains [Li et al., 2023a; Ji et al., 2023; Li et al., 2023b], among which diffusion models [Ho et al., 2020; Song et al., 2020] stand out due to their capability to model complex data distributions by iteratively adding noise and denoising. The extension to human-driven tasks [Tevet et al., 2023; Zhu et al., 2024] has motivated exploration in speech-driven 3D facial animation, where they outperform traditional methods in generating diverse expressions. FaceDiffuser [Stan et al., 2023] first applied the diffusion framework to facial animation using audio features as conditions and employing a GRU decoder for animation generation. DiffSpeaker [Ma et al., 2024] enhanced diversity of facial animations by integrating a transformerbased architecture with designed bias. Building on this, our work aims to leverage the iterative perturbation of diffusion models to globally capture high-dimensional facial feature distributions, enabling more refined cross-modal mapping.

3 Method

We aim to generate 3D facial animations synchronized with the given speech, which contains diverse emotions. The task can be formulated as follows: let A1:T = (a1, . . . , aT ) denote the input speech sequence corresponding to T video frames, where each at ∈ RD contains D samples. Let V1:T = (v1, . . . , vT ) denote the facial vertices sequence of length T, with each frame represented by V vertices, where vt ∈ RV×3 . Our goal is to generate facial vertices sequence V1:T with emotional variations based on speech A1:T .

As shown in Figure 2, ExpTalk follows a two-stage pipeline. In the first stage, we introduce Adaptive Disentanglement VAE (AD-VAE) to reconstruct the facial vertices sequence. Leveraging fine-grained contrastive learning with a pre-trained speech emotion feature extractor, this module adaptively disentangles the spoken language and emotional facial priors, which are then quantized into two subcodebooks for the reconstruction of the facial vertices sequence. In the second stage, we propose Refined Alignment Diffusion (RAD), which refines facial expressions based on speech information by integrating speech-guided and emotion-guided conditional attention mechanisms to denoise the noise-injected encodings from encoders of AD-VAE.

During inference, the model ExpTalk takes the speech sequence A1:T and speaker identity p as inputs. By iteratively denoising random noise, the model generates the facial animation ˆ V1:T = (ˆ v1, . . . , ˆ vT ). Formally,

where Θ denotes the learnable parameters of the model.

3.1 Adaptive Disentanglement VAE

In facial expressions, spoken language and emotions may be deeply fused, leading to mutual interference when separately refining them. The mutual interference may be more significant, especially when refining more diverse emotions. Inspired by [Zhang et al., 2020; Wu et al., 2024], Adaptive Disentanglement VAE (AD-VAE) is designed to disentangle facial expressions into two independent priors: spoken language and emotion. By embedding them into distinct quantization sub-codebooks, AD-VAE ensures their independence in the latent space. This disentanglement process is driven by fine-grained contrastive learning, which leverages speech emotion features as a supervisory signal to adaptively disentangle facial priors.

Model Design

AD-VAE consists of a motion encoder, an emotion encoder, and a facial vertices decoder. The motion and emotion encoders share the same structure and are designed to extract spoken language and emotional priors, respectively. Both encoders utilize spiral convolution operators to extract spatial features of facial vertices and temporal attention to capture the temporal dependencies within the vertices sequence. This design effectively integrates both the spatial correlations among vertices and the temporal dependencies of the sequence. The spiral convolution operator [Bouritsas et al., 2019; Gong et al., 2019] is a graph-based convolution that samples vertices in the spatial domain to capture local correlations. Given a convolution center v, the local neighborhood is described by k-ring(v), the set of vertices exactly k steps away from v, and k-disk(v), the union of all i-ring(v) for i ≤ k. They are formally defined as:

where N(·) denotes the set of neighboring vertices of a given vertex. The spiral sequence is defined by concatenating several k-ring(v) until reaching a predefined length. Using these sequences, the spiral convolution operator encodes local vertices information via a fully connected layer. Temporal attention is then applied to the encoded features, generating motion encoding zm ∈ RT×N×C and emotion encoding ze ∈ RT×N×C , collectively denoted as z∗.

To ensure independence between the encodings, a pretrained speech emotion feature extractor, emotion2vec [Ma et al., 2023], is utilized to extract emotional features from the corresponding speech. During training, contrastive learning is applied to enhance the alignment between the facial emotion encoding and speech emotional features while reducing the similarity between the facial motion encoding and speech emotional features, achieving encoding disentanglement [Li et al., 2024].

After extracting the encodings, the corresponding vectors ei in the sub-codebook E∗ are queried using a nearestneighbor search operation for quantization:

where ∥ · ∥22 denotes the L2 norm.

For decoding, the facial vertices decoder adopts a structure symmetric to the encoder. The quantized encodings are concatenated and mapped to their original dimensions, followed by temporal attention and spiral convolution operators to reconstruct the facial vertices sequence ˆ V1:T .

Loss Functions

To fully train our AD-VAE, we introduce carefully designed loss functions, which consist of three parts: reconstruction loss Lrec, quantization loss Lqua, and contrastive loss Lcon.

The reconstruction loss Lrec ensures that the model can accurately reconstruct the sequence. This loss is computed as the L1 loss between the ground truth facial vertices sequence and the predicted sequence:

where T denotes the sequence length, and ∥ · ∥1 denotes the L1 norm used for distance calculation.

The quantization loss Lqua is used to constrain the proximity between the unquantized latent encodings and the codebook vectors. This loss is defined as the weighted sum of codebook loss term and commitment loss term proposed from VQ-VAE [Van Den Oord et al., 2017]:

where sg(·) denotes the gradient-stop operation.

The contrastive loss Lcon guides the disentanglement of motion and emotion encodings. To achieve this, a pretrained speech emotion feature extractor is introduced to extract speech emotion features e ∈ RT×K , where K is the feature dimension. Emotion encodings of corresponding frame are paired with speech emotion features as positive samples, while motion encodings serve as negative samples. The loss is defined using InfoNCE [Oord et al., 2018]:

where τ denotes the temperature, and sim(·, ·) denotes the cosine similarity, defined as sim(a, b) = a⊤b ∥a∥∥b∥ .

The total loss for AD-VAE LADV is given by:

where λqua and λcon are hyperparameters.

3.2 Refined Alignment Diffusion

After decoupling spoken language and emotion priors of facial expressions, achieving seamless synchronization and expressiveness in facial animations requires precise crossmodal alignment with the corresponding speech information. To realize this objective, we design Refined Alignment Diffusion (RAD), a diffusion-based denoising network that iteratively refines spoken language and emotion priors of facial expressions to align with the corresponding speech information. We attempt to preserve high-frequency details and enable realistic, emotionally consistent facial animations synchronized with speech in the alignment process.

During training, the denoising network of RAD f(·) takes the noise-injected encodings znm and zn e , conditioned on the speech A1:T , speaker identity p, and diffusion step n, and directly predicts the clean encodings ˆ z0m and ˆ z0 e ,

In the inference phase, the high-noise version distributions of the motion and emotion encodings at diffusion step N, p(zN m , zN e ), are sequentially converted to the low-noise version distributions through a Markov chain, until the clean encoding distributions p(z0m, z0 e ) are achieved:

where p(zN m ) ∼ N(0, I) and p(zN e ) ∼N(0, I) represent the initial distributions of motion and emotion encodings, modeled as standard normal distributions N(0, I) with a mean of 0 and an identity covariance matrix I. Subsequently, they are passed into the decoder of the trained AD-VAE to reconstruct the facial vertices sequence.

Model Design

RAD consists of encoders, conditional embeddings, and a backbone conditional transformer decoder. We use WavLM [Chen et al., 2022] as the pre-trained speech encoder Ea(·) to extract speech features from A1:T and emotion2vec [Ma et al., 2023] as the pre-trained speech emotion encoder Ee(·) to extract emotion features. Both features are linearly transformed to match the dimensions required for the diffusion process. The noise input zn = [znm, zn e ], which concatenates the motion and emotion noise-injected encodings, is processed by the noise encoder En(·), a linear layer that compresses high-dimensional noise to retain critical motion and emotion information.

The speaker identity embedding Ep(p) and diffusion step embedding Et(n) are introduced as conditional embeddings into the model to enhance the model's capacity for capturing speaker style [Thambiraja et al., 2023; Yu et al., 2024] and diffusion step information.

Inspired by DiffSpeaker [Ma et al., 2024], we adopt a conditional transformer decoder as the backbone for denoising. This decoder takes speaker identity p and diffusion step n as conditions, and incorporates speech-guided and emotionguided conditional attention to progressively remove noise.

Specifically, the network consists of masked conditional self-attention, masked conditional cross-attention, and a feedforward network. The query Q in the self-attention comes from the noise input encoding En(zn ) ∈ RT×N , while the keys K and values V are obtained by concatenating the speaker identity embedding Ep(p), diffusion step embedding Et(n), and the noise input encoding En(zn ). Based on prior experience [Fan et al., 2022; Ma et al., 2024], we apply a symmetric temporal period mask Ms(i, j) to restrict attention to a small range within the current frame, while also considering the speaker identity and diffusion step information:

where p denotes the temporal period, which corresponds to the frame rate. The operator ⌊·⌋ represents the floor function.

The speech-guided and emotion-guided conditional crossattention mechanisms share an identical structure. Their keys K and values V are constructed by concatenating the speaker identity encoding Ep(p), the diffusion step encoding Et(n), and either the speech features Ea(A1:T ) or the emotion features Ee(A1:T ). To ensure strong frame-level correlations between the speech and emotion features and their corresponding encodings, we apply an alignment mask Mc(i, j):

where k denotes the frame-level alignment window size, controlling the range of temporal correlation between frames.

After multiple layers of the conditional transformer decoder, ˆ z0m and ˆ z0 e are dimensionally recovered by the latent decoder Dn(·), which is implemented as a linear layer.

Loss Functions

The loss for RAD LRAD is defined to measure the difference between the predicted encodings after the denoising process and the ground truth encodings. We use the Huber loss to quantify this difference, as follows:

where Ep(znm,zn e ) is the expectation over the noise encodings distribution, and ∥ · ∥H denotes the Huber loss [Huber, 1992].

4 Experiments

4.1 Experimental Settings

Datasets. In our experiments, we employ the 3D-MEAD dataset [Wang et al., 2020; Danˇeˇcek et al., 2023], which provides high-quality facial expression data for training and evaluating emotion-driven facial animation methods. Each 3D facial mesh is registered to the FLAME topology [Li et al., 2017], with 5023 vertices. The dataset includes speech and 3D reconstruction data from 46 English-speaking subjects across eight emotion categories: Neutral, Happy, Sad, Surprise, Fear, Disgust, Anger, and Contempt, with three intensity levels for each emotion except Neutral. Facial vertices sequences are sampled at 25 frames per second. To introduce emotional variations, we randomly concatenate the speech and facial vertices data. Due to the absence of official data configuration, we select 10 subjects for training and 3 unseen subjects for testing to ensure fairness.

To further evaluate our model, we employ the VOCASET dataset [Cudeiro et al., 2019], which includes 480 speech samples and 3D facial reconstruction data from 12 subjects. The facial animations are originally sampled at 60 frames per second. We downsample them to 25 frames per second and randomly concatenate the speech and facial vertices data to align with the same configuration as the 3D-MEAD dataset. We follow the official data split for fair comparisons.

Implementation Details. For the training of ExpTalk, we use the Adam optimizer [Kingma, 2014] in both stages. ADVAE is trained for 100 epochs with a batch size of 1 and a learning rate of 1 × 10-3 , taking approximately 10 hours. RAD is also trained for 100 epochs with a batch size of 1 and a learning rate of 2 × 10-5 , taking about 12 hours. During inference, we use 50 steps of the DDIM [Song et al., 2020] sampler. Our framework is implemented with PyTorch and runs on an Nvidia RTX 4090 GPU.

Compared Baselines. We extend several state-of-the-arts as baselines for a comprehensive comparison: (1) nonemotion-driven methods: FaceFormer [Fan et al., 2022], CodeTalker [Xing et al., 2023], FaceDiffuser [Stan et al., 2023], and UniTalker [Fan et al., 2025]. (2) emotion-driven methods: EMOTE [Danˇeˇcek et al., 2023], EmoTalk [Peng et al., 2023b], and DEEPTalk [Kim et al., 2024].

4.2 Quantitative Evaluation

We follow the evaluation metrics used in CodeTalker [Xing et al., 2023] and FaceDiffuser [Stan et al., 2023], including Mean Vertex Error (MVE), Lip Vertex Error (LVE), and Upper-Face Dynamics Deviation (FDD). To evaluate the accuracy of facial emotion representation, we also adopt Emotional Vertex Error (EVE) from EmoTalk [Peng et al., 2023b].

MVE measures overall accuracy via the Euclidean distance between generated and ground-truth facial vertex sequences. LVE measures accuracy in the lip region, reflecting lip synchronization, while EVE focuses on the upper face (e.g., forehead, eyes), capturing emotional expressiveness. FDD evaluates dynamic consistency by comparing the standard deviation of upper facial movement between generated and ground-truth sequences.

Table 1 presents the quantitative evaluation results. On the 3D-MEAD dataset, ExpTalk demonstrates outstanding performance across all metrics, showcasing its capability to generate emotionally expressive 3D facial animations. On the VOCASET dataset, ExpTalk leads in most metrics, although its MVE score is slightly lower than comparison models. This can be attributed to the characteristics of the VOCASET dataset, where speech samples are relatively short, and the dynamic range of vertex movements is limited. Such dataset distribution may place greater emphasis on static regions, which are less critical for evaluating dynamic emotional expressions. Despite this, ExpTalk excels in metrics such as LVE and EVE, which focus on dynamic facial regions, validating its ability to generate expressive and emotionally consistent facial animations aligned with speech.

4.3 Qualitative Evaluation

To further validate our method, we visualize and compare facial animations driven by single-emotion and multi-emotion speech. Single-emotion speech samples are from the 3DMEAD dataset, while multi-emotion speech is synthesized by concatenating single-emotion speech clips with 1-second transitions. The results are shown in Figure 3.

For single-emotion speech, non-emotion-driven methods leave the upper facial region relatively static and lack emotional expressiveness. Emotion-driven methods often generate overly exaggerated or indistinct expressions. In contrast, ExpTalk, through decoupling and refinement, generates facial animations with better lip synchronization and emotional expressiveness, closely matching the ground truth.

For multi-emotion speech, ExpTalk achieves emotional transitions in facial expressions, such as from happy to surprise or anger to happy. By aligning facial expressions with diverse speech emotions, our model produces more natural and expressive animations than other methods.

We also visualize heatmaps of the mean and standard deviation of facial animations generated by four methods compared to the ground truth. Figure 4 demonstrates that ExpTalk achieves superior performance, with upper facial movements more closely aligning with the ground truth. Additionally, We use t-SNE[Van der Maaten and Hinton, 2008] to visualize facial encodings with and without disentanglement, as shown in Figure 5. The results indicate that disentanglement enhances the distinction of different facial emotions, further validating its effectiveness in reconstructing facial expressions.

4.4 User Study

To evaluate ExpTalk's effectiveness, we conducted a user study comparing it with CodeTalker, UniTalker, EMOTE, and EmoTalk. Using 90 speech samples from the 3D-MEAD, RAVDESS [Livingstone and Russo, 2018], and online videos, covering diverse demographics, languages, and prosodic features. We generated facial animations driven by single and multiple emotional speech. Participants rated randomly presented videos on lip synchronization, emotional expressiveness, and visual realism using a 1-5 scale. The mean opinion scores are shown in Table 2. ExpTalk outperformed compared methods across all aspects, demonstrating its ability to generate expressive and emotionally rich facial animations.

4.5 Ablation Study

We conduct ablation studies on the 3D-MEAD dataset. The results are as shown in Table 3 and Figure 5.

Effect of the Key Components. We investigate the impact of the conditional Transformer decoder and emotion2vec. Replacing the conditional decoder with a standard Transformer decoder, we add speaker identity and diffusion step embeddings to the noise input and speech features. Results show that removing the conditional decoder significantly degrades multiple metrics, primarily because the model initially relies on audio information but gradually shifts focus to speaker identity as the noise is removed. Replacing emotion2vec with wav2vec [Baevski et al., 2020] in both modules results in decreased emotional expressiveness, indicating that emotion2vec enhances fine-grained emotion modeling through its pretrained knowledge and effectively guides both AD-VAE disentanglement and RAD refinement.

Effect of the Model Architecture. We evaluate the necessity of facial feature disentanglement and the diffusion framework. Removing the emotion branch in AD-VAE leads to entanglement between emotional and non-emotional information in facial features, making it difficult for the model to clearly refine facial expressions and resulting in facial animations with ambiguous emotional characteristics. This observation is further validated by Figure 5. Replacing the diffusion framework with a Transformer decoder to predict facial priors causes a significant drop in EVE, demonstrating that the iterative denoising in the diffusion process plays a crucial role in the fine-grained alignment of facial priors.

5 Conclusion

In this work, we propose an innovative speech-driven 3D facial animation method, ExpTalk, aimed at generating more natural and emotionally expressive facial animations. The meticulously designed AD-VAE employs fine-grained contrastive learning to embed disentangled encodings into distinct quantization sub-codebooks. RAD then facilitates the refined alignment of cross-modal information between speech and facial priors, ensuring lip synchronization and emotional consistency in the generated speech-driven facial animations. Extensive qualitative and quantitative experiments validate the method's advantages on multiple benchmark datasets. We hope ExpTalk will inspire further research toward more realistic facial animation generation.

Ethical Statement

This research focuses on advancing speech-driven 3D facial animation technology, which has applications in fields such as virtual reality, digital avatars, and interactive entertainment. We commit to ensuring that the proposed methods are developed and applied responsibly, avoiding any misuse in areas such as deepfake creation or deceptive practices. Ethical considerations, including privacy and consent, are prioritized in all stages of data usage and model deployment.

Acknowledgements

This work is supported by National Natural Science Foundation of China (No. 62222211, U24A20326, 62402429), Key Research and Development Program of Zhejiang Province (No. 2024C03270), ZJU Kunpeng&Ascend Center of Excellence, Ningbo Yongjiang Talent Introduction Programme (2023A-397-G) and Young Elite Scientists Sponsorship Program by CAST (2024QNRC001).