Outstanding Orthodontist: No More Artifactual Teeth in Talking Face

1 Introduction

Audio-driven TFS [Prajwal et al., 2020; Xu et al., 2024c; Tian et al., 2025] aims to generate realistic videos of a target person given a single portrait image and an audio speech clip as input. This technology enables automatic generation of synchronized facial animations from audio input, with broad applications in virtual communication, digital entertainment [Xu et al., 2024a], and multimedia content creation. Recent methods such as TalkLip [Wang et al., 2023], SyncTalk [Peng et al., 2024], and TTSF [Jang et al., 2024] have made significant progress in both lip synchronization and the consistency of facial expressions.

∗Corresponding author.

However, a critical yet unexplored challenge remains: maintaining the temporal consistency and realism of teeth appearance under dynamic teeth occlusion during speech. Since the teeth occlusion phenomenon frequently occurs during speech, the generation of the teeth region shifts from an image editing problem to an image inpainting problem, introducing two additional technical challenges. Firstly, teeth inconsistency arises in the generated videos, as shown in Fig. 1 (a), (b), (c), and (d), with unnatural expansion or contraction of the teeth size across frames, leading to instability in the rigid structure. This issue is exacerbated by models that generate frames independently, causing inconsistent regeneration of the teeth region. Secondly, there is the problem of generation hallucination in the teeth region. As shown in Fig.1 (a), significant hallucinations appear in the lower teeth, while Fig.1 (c) shows severe degradation of teeth details and surface textures, with blurred gaps between teeth. Due to insufficient prior knowledge, models may create, remove, or distort teeth structures, resulting in unnatural textures, uneven edges, and unrealistic shapes. This issue is exacerbated by facial movements involving complex interactions between the lips and teeth. These phenomena make generated videos look obviously fake, negatively impacting viewer experience.

To address these challenges, we propose the Orthodontist Network (OrthoNet), which focuses on the teeth region of generated talking face videos to enhance their realism. Inspired by the outstanding orthodontists in aligning and stabilizing teeth, we also propose the Detail-oriented Teeth Aligner and Memory-guided Teeth Stabilizer to eliminate artifacts in the teeth region of TFS. OrthoNet introduces the Detail-oriented Teeth Aligner module to address generation hallucinations. One branch preserves fine details of the target person's teeth region using dense remap convolution (DRM-Conv), while the other adapts the teeth shapes during lip movements through polymorphic kernel convolution (PKConv). Additionally, the Memory-guided Teeth Stabilizer combines long-term memory (LTM) for global teeth features with short-term memory (STM) for fine-grained dynamics during speech, ensuring temporal consistency in the teeth region.

Our key contributions are as follows:

• To the best of our knowledge, our paper is the first to introduce and address the temporal inconsistency and artifact of the teeth region in the TFS task.
• Guided by prior information about the target person's teeth, the Detail-oriented Teeth Aligner module models the structural patterns in the dynamic talking face to eliminate generation hallucinations.
• The Teeth Stabilizer leverages the interaction between LTM and STM to maintain a stable teeth appearance, even with the frequent disturbance caused by the occlusion phenomenon.
• Our method is a plug-and-play framework that can be integrated with other Audio2Video methods to enhance generation realism, as demonstrated by extensive qualitative and quantitative experimental results.

2 RelatedWork

Audio-driven TFS makes significant strides with deep learning advancements [Tao et al., 2025]. Early methods focus on direct audio-visual mappings, with foundational works like Chen et al. [Chen et al., 2018] and Zhou et al. [Zhou et al., 2019] establishing basic frameworks for lip synchronization and expression generation.

Recent developments bring substantial improvements in synthesis quality and controllability. SyncDiffusion [Zhao et al., 2024] introduces a diffusion-based framework that achieves remarkable temporal coherence and lip-sync accuracy. HeyGen [Li et al., 2024] proposes a attention mechanism specifically designed for handling dynamic facial features during speech. Additionally, SadTalker [Zhang et al., 2023a] develops a robust emotion-aware generation pipeline that significantly enhances the naturalness of facial expressions. Similar approaches for feature enhancement and artifact detection have been explored in other domains [Tao et al., 2021].

The latest research focuses on enhancing fine-grained control and generalization [Xu et al., 2024b]. DiffTalk [Wang et al., 2024] employs a conditional diffusion model that enables precise control over facial attributes while maintaining speech synchronization. VideoReTalking [Xu et al., 2024d] achieves impressive results in cross-identity synthesis through careful disentanglement of speech content and speaker identity. VASA [Xu et al., 2024c] further advances the field by introducing a view-adaptive speaking architecture that handles varying head poses effectively.

Several works specifically target temporal consistency. TalkLip [Wang et al., 2023] introduces a temporal discriminator architecture that significantly improves lip movement stability. SpeechToFace [Zhang et al., 2024] develops an advanced temporal coherence loss that enhances the smoothness of facial transitions. EMO [Tian et al., 2025] proposes an emotional-aware framework that maintains consistency while incorporating expressive variations.

Multi-modal methods also gain prominence. AudioStyle [Chen et al., 2024a] combines audio features with style transfer techniques to achieve more personalized facial animations. StyleTalk [Liu et al., 2024] leverages style-based generation to enhance the visual quality of synthesized faces. These methods demonstrate significant improvements in generating realistic and temporally coherent facial animations.

Despite these advances, existing methods typically treat all facial features uniformly without specific consideration for rigid structures like teeth. While recent works make substantial progress in lip synchronization and overall facial animation quality, they do not address the unique challenges of maintaining teeth temporal consistency throughout the generated video sequence. This oversight often leads to temporal inconsistencies and artifacts in the teeth region, particularly under dynamic teeth occlusion during speech.

3 Method

As shown in Fig. 2, we present OrthoNet, a plug-and-play framework for teeth restoration in talking face videos that operates as a virtual orthodontist. It has two main parts: Detail-oriented Teeth Aligner for extracting precise teeth features and Memory-guided Teeth Stabilizer for maintaining stability over time. Detail-oriented Teeth Aligner uses dualbranch: DRM-Conv preserves fine-grained teeth details, and PKConv captures the overall shape of the teeth, ensuring both small parts and the whole structure are accurately modeled. Memory-guided Teeth Stabilizer has a dual-memory stream inspired by post-orthodontic treatment. LTM maintains the temporal consistency of the teeth, and STM enables teeth adaptation to speech dynamics, similar to how orthodontists use different retainers to retain strength while adapting naturally during speech.

3.1 Detail-oriented Teeth Aligner

The Detail-oriented Teeth Aligner module extracts precise teeth features through a dual-branch method, similar to the method used by orthodontists with different tools to adjust teeth position and shape.

Fine-grained Feature Extraction. Given an input feature map Fin ∈ RC×H×W , where C represents the channel dimension and H,W denote spatial dimensions, our dual-branch structure extracts comprehensive teeth features through two complementary pathways: DRM-Conv for preserving fine-grained details and PKConv for dynamic shape adaptation. The DRM-Conv branch, illustrated in Fig. 3a, preserves critical fine-grained details through a space-todepth transformation followed by specialized convolution operations:

where scale = 2 is empirically determined to maintain optimal balance between computational efficiency and preservation of critical edge details. The operation Cat(·) concatenates feature maps along the channel dimension, while Wdrm represents learnable convolution weights [Sunkara and Luo, 2022] specifically designed for detail enhancement. For handling complex teeth shapes and occlusions, the PKConv branch, depicted in Fig. 3b, employs an adaptive sampling mechanism:

where p represents the spatial position, Pn denotes the sampling pattern with n points, wk(p) are position-specific adaptive weights, and foffset(Fin,Mteeth) computes dynamic offsets [Zhang et al., 2023b] based on both input features and teeth segmentation mask Mteeth. The multi-scale features from both branches are integrated through an adaptive fusion mechanism:

where αl ∈ [0, 1] dynamically balances the contribution of local details and global structure at each scale level l, Up(·)

performs bilinear upsampling to align feature resolutions, and βl are learnable scale-specific weights that optimize the importance of each resolution level. This multi-scale fusion ensures comprehensive feature extraction across different spatial scales, crucial for accurate teeth alignment planning.

Feature Quality Enhancement. To ensure pixel-level accuracy between the generated teeth region Ig and the ground truth Igt, we employ:

This reconstruction loss [Li et al., 2023] ensures pixel-level accuracy and perceptual similarity between generated and ground truth images. Additionally, we introduce a teeth perception loss to capture structural characteristics:

where:

Here, Mt denotes the teeth region mask, φl and φh represent low-level and high-level VGG-19 features respectively, with β set to 0.7.

3.2 Memory-guided Teeth Stabilizer

To maintain temporal consistency while preserving the structural details of teeth, we propose a Memory-guided Teeth Stabilizer that integrates both long-term and short-term temporal dependencies [Xu et al., 2021], similar to how orthodontic retainers stabilize teeth after treatment, as detailed in Algorithm 1.

Long-termMemoryModule. The LTMmodule maintains a feature buffer BL ∈ R30×C storing teeth structural information from past 30 frames. Quality assessment (Qstruct) evaluates each new frame's teeth features before updating the buffer. Only high-quality frames meeting the threshold θq are retained through Top-K selection, ensuring the buffer maintains representative exemplars of teeth structure. This adaptive update mechanism prevents degradation from poorquality frames while preserving stable global features.

Short-termMemory Module. The STMmodule employs a ConvLSTM architecture to capture frame-to-frame variations in teeth appearance within a 4-frame sliding window (Sst). Through specialized PKConv-based queries, it generates hidden states (ht) that encode local temporal dependencies. This enables focused attention on relevant teeth features and smooth transitions between frames. The STM adapts to dynamic speech patterns while maintaining local consistency.

Memory Integration. The framework integrates LTM and STM features through a cross-attention mechanism. Attention weights (wlt) determine the relevance of long-term features to the current frame. Temporal fusion combines ConvLSTM states with current features, while an adaptive blending factor (α) balances global stability from LTM with local dynamics from STM based on frame quality. This dual-memory method ensures both temporal consistency and natural motion during speech. The temporal consistency is enforced through two key loss components:

First, we utilize an adversarial loss based on WGAN-GP:

where D(·) is the discriminator, Pg and Pr are distributions of generated and real images, and Lgp is the gradient penalty

Algorithm 1 Memory Bank Update in OrthoNet

Require: Current frame Ft, Long-term buffer BL, Shortterm states Sst, Quality threshold θq

Ensure: Temporally consistent features Fout

• 1: Qstruct ←QualityAssess(Ft)
• 2: if Qstruct > θq then
• 3: fteeth ←ExtractTeethFeatures(Ft)
• 4: Bnew ←Top-K(BL ∪ {fteeth},K = 30)
• 5: BL ←Bnew
• 6: end if
• 7: Snew ←Sst[2 :] ∪ {Ft}
• 8: ht ←ConvLSTM(Ft, Snew)
• 9: Sst ←Snew
• 10: wlt ← SoftmaxAttn(Ft,BL)
• 11: flt ← ∑30 i=1 wi lt · BiL
• 12: fst ←TemporalFusion(ht, Sst)
• 13: α← σ(Qstruct)
• 14: Fout ← α · flt + (1 - α) · fst
• 15: return Fout

term. Second, we introduce a temporal consistency loss:

where w(·) represents optical flow warping, ht is the ConvLSTM hidden state, and flstm(·) is the state transition function. The overall loss function is formulated as:

where {λ1, λ2, λ3, λ4} are empirically set to balance different loss terms.

4 Experiments

In this section, we present experimental results evaluating our OrthoNet framework. We introduce evaluation metrics for teeth quality and temporal consistency, then compare against state-of-the-art (SOTA) methods and conduct ablation studies to validate our design choices.

4.1 Experimental Settings

Dataset and Data Processing. We train the proposed method on High-Definition Talking Face (HDTF) [Zhang et al., 2021] dataset and our self-built high-resolution news anchor dataset. HDTF contains videos of 362 different identities with a total duration of 15.8 hours. The reason for choosing HDTF is that the teeth regions in this dataset are relatively clear and well-exposed during speech. Additionally, to enrich the diversity of teeth appearances under different speaking conditions and enhance the model's ability to learn stable and accurate teeth representations, we constructed a supplementary dataset collected from various news channels. This selfbuilt dataset features news anchors with appropriate speaking speeds and high probability of teeth exposure, making it ideal for capturing teeth features. The videos in this dataset have resolutions of 720P or 1080P with a total duration of 24.7 hours. We will publicly release our dataset to facilitate future research. The details of data processing are shown in Fig 4.

Evaluation Indicators. To evaluate teeth generation and temporal consistency in talking face videos, we propose a framework combining traditional and specialized metrics:

• Fr´echet Video Distance (FVD):

FVD = ∥µr-µg∥22 +Tr(Σr+Σg-2(ΣrΣg)1/2 ), (11) where µr, µg are mean feature embeddings, and Σr, Σg are covariance matrices of real and generated video sequences. Extracted using the I3D network, FVD evaluates temporal coherence of the face region.

• Structural Similarity Index (SSIM):

where µx, µy are local means, σ2x , σ2 y are variances, and σxy is covariance. Constants c1, c2 stabilize division.

• Teeth Stability Metric (TSM):

TSM quantifies temporal consistency of teeth shapes across frames, using perimeter-area ratio (rpa), aspect ratio (rar), and normalized area (an).

• Edge Clarity Index (ECI):

ECI evaluates sharpness and definition of teeth edges, where Et are edge pixels (Canny operator), Gx, Gy are Sobel gradients, and Gmax is the maximum gradient magnitude.

Implementation Details. We implement our framework using PyTorch and train it on four A6000 GPUs. The input frames are processed to 96 × 96 resolution with detected facial landmarks for teeth and lip mask generation. For network architecture, we set scale = 2 in DRM-Conv. The memory modules maintain a 30-frame long-term buffer and 4-frame short-term window based on ablation studies. During training, we employ the Adam optimizer with a learning rate of 1 × 10-5 and train for 30,000 epochs with batch size 4. The model is trained on the combined HDTF dataset and our selfbuilt dataset.

4.2 Experimental Results

Quantitative Analysis. We evaluate OrthoNet against five SOTA TFS methods: AniPortrait [Wei et al., 2024], Audio2Head [Wang et al., 2021], EchoMimic [Chen et al., 2024b], JoyHallo [Shi et al., 2024], and Hallo2 [Cui et al., 2024]. For each baseline, we compare its original version with a OrthoNet-enhanced version (denoted as "+OrthoNet"). Table 1 presents quantitative results across all evaluation metrics.

Our framework demonstrates significant improvements in shape stability, with TSM scores increasing by 9.8% to 15.7% across different baselines. Notably, EchoMimic+OrthoNet achieves the largest improvement from 0.728 to 0.843, while Hallo2+OrthoNet reaches the highest absolute score of 0.813 among enhanced methods. In terms of structural quality, the integration of OrthoNet consistently improves teeth edge definition and overall structural similarity, with ECI improvements of 0.2-0.3 points observed across all methods. AniPortrait+OrthoNet achieves particularly strong gains with a 6.2% increase in ECI. The temporal coherence, as measured by FVD scores, also shows consistent enhancement, with JoyHallo+OrthoNet achieving the most substantial improvement through a 2.8% reduction in FVD. As shown in Table 1, our method achieves significant improvements in teeth-specific metrics (TSM, ECI, SSIM) for the mouth region evaluation, while also demonstrating modest gains in global video quality as measured by FVD.

Qualitative Analysis. Fig. 5 presents visual comparisons highlighting the qualitative improvements brought by OrthoNet. The enhanced geometric consistency is particularly evident in the preservation of teeth shapes and proportions across frames during rapid mouth movements. Our method maintains consistent teeth width and height ratios under varying degrees of teeth occlusion, whereas baseline methods often exhibit noticeable shape fluctuations.

The preservation of fine-grained teeth features represents another significant improvement in our results. OrthoNet successfully maintains consistent teeth edges and surface textures, effectively preventing the common issue of detail loss during mouth motion. Furthermore, our framework exhibits robust performance in handling dynamic teeth occlusion, where OrthoNet successfully reconstructs partially visible teeth while maintaining structural coherence, addressing a major limitation of existing methods.

The memory-guided method also enables more natural transitions in teeth visibility. As shown in Fig. 5, our method eliminates abrupt changes and artifacts during mouth opening/closing sequences, resulting in more realistic mouth animations. This improvement is particularly noticeable in sequences with rapid speech patterns, where baseline methods often struggle to maintain consistency.

Cross-Method Performance Analysis. OrthoNet demonstrates its effectiveness as a general enhancement module through consistent performance improvements across diverse baseline architectures, with enhancement ratios varying by less than 3%. This highlights its adaptability and generalization capability. The framework complements baseline methods, preserving their strengths while improving teethrelated quality. For example, JoyHallo retains its strong lip synchronization performance with a 15.7% gain in teeth stability. Moreover, methods with lower initial teeth generation performance (e.g., EchoMimic) show larger absolute improvements, while high-performing baselines (e.g., Hallo2) achieve notable gains. These results validate OrthoNet's ability to enhance teeth consistency in TFS while maintaining compatibility with diverse baselines, proving the effectiveness of our memory-guided method.

4.3 Ablation Studies

We conduct ablation studies on OrthoNet's three key components-memory modules, feature extraction, and loss functions-to validate their effectiveness (Table 2).

Memory Module Analysis. We evaluate different memory configurations to validate our dual-memory design. Removing both memory modules causes significant degradation (TSM: -14.5% to 0.695), with LTM showing greater impact than STM when ablated individually (TSM: -8.9% vs -4.7%). For optimization of memory capacity, as presented in Table 3, experiments demonstrate that a 30-frame LTM buffer and a 4-frame STM window achieve optimal performance (TSM: 0.813). Smaller LTM sizes (10/20 frames, TSM: 0.767/0.789) or STM windows (2/3 frames, TSM: 0.785/0.798) lead to reduced stability, while larger sizes yield minimal improvements. These results validate both the effectiveness of our dual-memory architecture and its optimal configuration settings. As shown in Fig. 6, the model produces irregular teeth variations and temporal instability without memory modules, particularly evident in the sudden changes of teeth shape and size between consecutive frames, as well as the degradation of fine-grained details during mouth movements.

Feature Extraction Analysis. Our dual-branch design significantly outperforms standard convolution (TSM: -13.7% to 0.702). Between the two specialized branches, DRM-Conv shows greater importance in preserving teeth details, as its removal causes more substantial degradation (TSM: -8.0% to 0.748) compared to PKConv ablation (TSM: -4.4% to 0.777).

This indicates the crucial role of spatially-aware deformable convolution in capturing teeth morphological features. As shown in Fig. 6, standard convolution causes detail loss and unrealistic artifacts, manifesting as blurred teeth edges and inconsistent gaps between teeth that fail to capture the natural structural patterns present in real teeth during speech.

Loss Function Analysis. Using only basic reconstruction and adversarial losses limits the model's performance (TSM: 0.708). The Lteeth proves important, as its removal leads to notable degradation (TSM: -4.2% to 0.779) compared to temporal loss ablation (TSM: -7.4% to 0.753). The complete loss function achieves the best results across all metrics (TSM: 0.813, FVD: 153.6), validating our loss design.

5 Conclusion

We introduce OrthoNet, a plug-and-play framework for improving teeth realism in audio-driven TFS. It consists of two components: the Teeth Aligner, which preserves detail during mouth movement, and the Teeth Stabilizer, which ensures consistency through a memory system combining structural patterns with motion features. OrthoNet integrates easily into existing synthesis systems and addresses teeth hallucination and inconsistency issues during complex movements. Experimental results demonstrate significant improvements in temporal consistency and structural stability, enhancing visual realism in speech synthesis.

Ethical Statement

This research on talking face synthesis technology focuses solely on enhancing the realism of teeth representation for legitimate applications such as digital entertainment, education, and accessibility. We acknowledge the potential dualuse concerns of synthetic media technologies and emphasize that our work aims to improve visual quality rather than enable deception. All datasets used in this study were ethically sourced from public domain materials or with appropriate permissions. We support responsible development practices, including transparency about synthetic content and the advancement of detection technologies alongside generation capabilities.

Acknowledgments

This work is supported in part by the National Key Research and Development Program of China (No. 2023YFC3305600), the Joint Fund of Ministry of Education of China (8091B02072404), the National Natural Science Foundation of China (62132016, 62171343, and 62406238), the Natural Science Basic Research Program of Shaanxi (2020JC-23), the Fundamental Research Funds for the Central Universities (ZYTS25149), and the National Key Laboratory Foundation of China (Grant No. HTKJ2024KL504011).