Proceedings

GLDiTalker: Speech-Driven 3D Facial Animation with Graph Latent Diffusion Transformer

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 1548-1556.

https://doi.org/10.24963/ijcai.2025/173

Abstract

Speech-driven talking head generation is a critical yet challenging task with applications in augmented reality and virtual human modeling. While recent approaches using autoregressive and diffusion-based models have achieved notable progress, they often suffer from modality inconsistencies, particularly misalignment between audio and mesh, leading to reduced motion diversity and lip-sync accuracy. To address this, we propose GLDiTalker, a novel speech-driven 3D facial animation model based on a Graph Latent Diffusion Transformer. GLDiTalker resolves modality misalignment by diffusing signals within a quantized spatiotemporal latent space. It employs a two-stage training pipeline: the Graph-Enhanced Quantized Space Learning Stage ensures lip-sync accuracy, while the Space-Time Powered Latent Diffusion Stage enhances motion diversity. Together, these stages enable GLDiTalker to generate realistic, temporally stable 3D facial animations. Extensive evaluations on standard benchmarks demonstrate that GLDiTalker outperforms existing methods, achieving superior results in both lip-sync accuracy and motion diversity.

  1. [Cudeiro et al., 2019] Daniel Cudeiro, Timo Bolkart, Cas- sidy Laidlaw, Anurag Ranjan, and Michael J Black. Cap- ture, learning, and synthesis of 3d speaking styles. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10101-10111, 2019.
  2. [Edwards et al., 2016] Pif Edwards, Chris Landreth, Eugene Fiume, and Karan Singh. Jali: an animator-centric viseme model for expressive lip synchronization. ACM Transac- tions on graphics (TOG), 35(4):1-11, 2016.
  3. [Fan et al., 2022] Yingruo Fan, Zhaojiang Lin, Jun Saito, Wenping Wang, and Taku Komura. Faceformer: Speech- driven 3d facial animation with transformers. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18770-18780, 2022.
  4. [Fanelli et al., 2010] Gabriele Fanelli, Juergen Gall, Harald Romsdorfer, Thibaut Weise, and Luc Van Gool. A 3- d audio-visual corpus of affective communication. IEEE Transactions on Multimedia, 12(6):591-598, 2010.
  5. [Fu et al., 2024] Hui Fu, Zeqing Wang, Ke Gong, Keze Wang, Tianshui Chen, Haojie Li, Haifeng Zeng, and Wenxiong Kang. Mimic: Speaking style disentanglement for speech-driven 3d facial animation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1770-1777, 2024.
  6. [Gong et al., 2019] Shunwang Gong, Lei Chen, Michael Bronstein, and Stefanos Zafeiriou. Spiralnet++: A fast and highly efficient mesh convolution operator. In Proceedings of the IEEE/CVF international conference on computer vi- sion workshops, pages 0-0, 2019.
  7. [Goodfellow et al., 2020] Ian Goodfellow, Jean Pouget- Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Gen- erative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.
  8. [Karras et al., 2017] Tero Karras, Timo Aila, Samuli Laine, Antti Herva, and Jaakko Lehtinen. Audio-driven facial an- imation by joint end-to-end learning of pose and emotion. ACM Transactions on Graphics (TOG), 36(4):1-12, 2017.
  9. [Kingma and Dhariwal, 2018] Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 con- volutions. Advances in neural information processing sys- tems, 31, 2018.
  10. [Kingma and Welling, 2014] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. International Conference on Learning Representations (ICLR), 2014.
  11. [Li et al., 2017] Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. Learning a model of facial shape and expression from 4D scans. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 36(6), 2017.
  12. [Ng et al., 2022] Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, Trevor Darrell, Angjoo Kanazawa, and Shiry Ginosar. Learning to listen: Modeling non-deterministic dyadic fa- cial motion. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 20395-20405, 2022.
  13. [Park et al., 2022] Se Jin Park, Minsu Kim, Joanna Hong, Jeongsoo Choi, and Yong Man Ro. Synctalkface: Talk- ing face generation with precise lip-syncing via audio-lip memory. In Proceedings of the AAAI Conference on Arti- ficial Intelligence, volume 36, pages 2062-2070, 2022.
  14. [Peng et al., 2023a] Ziqiao Peng, Yihao Luo, Yue Shi, Hao Xu, Xiangyu Zhu, Hongyan Liu, Jun He, and Zhaoxin Fan. Selftalk: A self-supervised commutative training di- agram to comprehend 3d talking faces. In Proceedings of the 31st ACM International Conference on Multimedia, pages 5292-5301, 2023.
  15. [Peng et al., 2023b] Ziqiao Peng, Haoyu Wu, Zhenbo Song, Hao Xu, Xiangyu Zhu, Jun He, Hongyan Liu, and Zhaoxin Fan. Emotalk: Speech-driven emotional disentanglement for 3d face animation. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision (ICCV), pages 20687-20697, October 2023.
  16. [Ping et al., 2013] Heng Yu Ping, Lili Nurliyana Abdullah, Puteri Suhaiza Sulaiman, and Alfian Abdul Halin. Com- puter facial animation: A review. International Journal of Computer Theory and Engineering, 5(4):658, 2013.
  17. [Ren et al., 2023] Zhiyuan Ren, Zhihong Pan, Xin Zhou, and Le Kang. Diffusion motion: Generate text-guided 3d hu- man motion by diffusion model. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE, 2023.
  18. [Richard et al., 2021] Alexander Richard, Michael Zollhöfer, Yandong Wen, Fernando De la Torre, and Yaser Sheikh. Meshtalk: 3d face animation from speech using cross-modality disentanglement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1173-1182, 2021.
  19. [Shen and Tang, 2024] Fei Shen and Jinhui Tang. Imagpose: A unified conditional framework for pose-guided person generation. Advances in neural information processing systems, 37:6246-6266, 2024.
  20. [Shen et al., 2022] Shuai Shen, Wanhua Li, Zheng Zhu, Yueqi Duan, Jie Zhou, and Jiwen Lu. Learning dynamic facial radiance fields for few-shot talking head synthesis. In Computer Vision-ECCV 2022: 17th European Confer- ence, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XII, pages 666-682, 2022.
  21. [Shen et al., 2023] Shuai Shen, Wenliang Zhao, Zibin Meng, Wanhua Li, Zheng Zhu, Jie Zhou, and Jiwen Lu. Difftalk: Crafting diffusion models for generalized audio-driven portraits animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1982-1991, 2023.
  22. [Shen et al., 2025a] Fei Shen, Xin Jiang, Xin He, Hu Ye, Cong Wang, Xiaoyu Du, Zechao Li, and Jinhui Tang. Imagdressing-v1: Customizable virtual dressing. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 6795-6804, 2025.
  23. [Shen et al., 2025b] Fei Shen, Cong Wang, Junyao Gao, Qin Guo, Jisheng Dang, Jinhui Tang, and Tat-Seng Chua. Long-term talkingface generation via motion-prior condi- tional diffusion model. arXiv preprint arXiv:2502.09533, 2025.
  24. [Sohl-Dickstein et al., 2015] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermody- namics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
  25. [Song et al., 2024] Wenfeng Song, Xuan Wang, Shi Zheng, Shuai Li, Aimin Hao, and Xia Hou. Talkingstyle: Person- alized speech-driven 3d facial animation with style preser- vation. IEEE Transactions on Visualization and Computer Graphics, pages 1-12, 2024.
  26. [Stan et al., 2023] Stefan Stan, Kazi Injamamul Haque, and Zerrin Yumak. Facediffuser: Speech-driven 3d facial an- imation synthesis using diffusion. In Proceedings of the 16th ACM SIGGRAPH Conference on Motion, Interaction and Games, pages 1-11, 2023.
  27. [Taylor et al., 2012] Sarah L Taylor, Moshe Mahler, Barry- John Theobald, and Iain Matthews. Dynamic units of visual speech. In Proceedings of the 11th ACM SIG- GRAPH/Eurographics conference on Computer Anima- tion, pages 275-284, 2012.
  28. [Van Den Oord et al., 2017] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
  29. [Wohlgenannt et al., 2020] Isabell Wohlgenannt, Alexander Simons, and Stefan Stieglitz. Virtual reality. Business & Information Systems Engineering, 62:455-461, 2020.
  30. [Wu et al., 2023] Haozhe Wu, Songtao Zhou, Jia Jia, Jun- liang Xing, Qi Wen, and Xiang Wen. Speech-driven 3d face animation with composite and regional facial move- ments. In Proceedings of the 31st ACM International Con- ference on Multimedia, pages 6822-6830, 2023.
  31. [Xing et al., 2023] Jinbo Xing, Menghan Xia, Yuechen Zhang, Xiaodong Cun, Jue Wang, and Tien-Tsin Wong. Codetalker: Speech-driven 3d facial animation with dis- crete motion prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12780-12790, 2023.
  32. [Xu et al., 2013] Yuyu Xu, Andrew W Feng, Stacy Marsella, and Ari Shapiro. A practical and configurable lip sync method for games. In Proceedings of Motion on Games, pages 131-140. 2013.
  33. [Xu et al., 2024] Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. VASA-1: Lifelike audio- driven talking faces generated in real time. In The Thirty- eighth Annual Conference on Neural Information Process- ing Systems, 2024.
  34. [Zhang et al., 2023] Chenxu Zhang, Chao Wang, Jianfeng Zhang, Hongyi Xu, Guoxian Song, You Xie, Linjie Luo, Yapeng Tian, Xiaohu Guo, and Jiashi Feng. Dream-talk: diffusion-based realistic emotional audio-driven method for single image talking face generation. arXiv preprint arXiv:2312.13578, 2023.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards