Proceedings

SpeechHGT: A Multimodal Hypergraph Transformer for Speech-Based Early Alzheimer’s Disease Detection

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. AI4Tech: AI Enabling Technologies. Pages 9131-9139.

https://doi.org/10.24963/ijcai.2025/1015

Abstract

Early detection of Alzheimer's disease (AD) through spontaneous speech analysis represents a promising, non-invasive diagnostic approach. Existing methods predominantly rely on fusion-based multimodal deep learning, effectively integrating linguistic and acoustic features. However, these methods inadequately model higher-order interactions between modalities, reducing diagnostic accuracy. To address this, we introduce SpeechHGT, a multimodal hypergraph transformer designed to capture and learn higher-order interactions in spontaneous speech features. SpeechHGT encodes multimodal features as hypergraphs, where nodes represent individual features and hyperedges represent grouped interactions. A novel hypergraph attention mechanism enables robust modeling of both pairwise and higher-order interactions. Experimental evaluations on the DementiaBank datasets reveal that SpeechHGT achieves state-of-the-art performance, surpassing baseline models in accuracy and F1 score. These results highlight the potential of hypergraph-based models to improve AI-driven diagnostic tools for early AD detection.

  1. [Ahmed et al., 2019] Md Rishad Ahmed, Yuan Zhang, Zhi- quan Feng, Benny Lo, Omer T. Inan, and Hongen Liao. Neuroimaging and machine learning for dementia diag- nosis: Recent advancements and future prospects. IEEE Reviews in Biomedical Engineering, 12:19-33, 2019.
  2. [Alberdi et al., 2016] Ane Alberdi, Asier Aztiria, and Adrian Basarab. On the early diagnosis of alzheimer's disease from multimodal signals: A survey. Artificial intelligence in medicine, 71:1-29, 2016.
  3. [Bessadok et al., 2022] Alaa Bessadok, Mohamed Ali Mahjoub, and Islem Rekik. Graph neural networks in network neuroscience. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):5833-5848, 2022.
  4. [Chen et al., 2023] Xuchu Chen, Yu Pu, Jinpeng Li, and Wei-Qiang Zhang. Cross-lingual alzheimer's disease de- tection based on paralinguistic and pre-trained features. In ICASSP 2023 -2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), page 1-2. IEEE, June 2023.
  5. [Dell, 1986] Gary S. Dell. A spreading-activation theory of retrieval in sentence production. Psychological Review, 93(3):283-321, 1986.
  6. [Ding et al., 2024] Kewen Ding, Madhu Chetty, Azadeh Noori Hoshyar, Tanusri Bhattacharya, and Britt Klein. Speech based detection of alzheimer's disease: a survey of ai techniques, datasets and challenges. Artificial Intelli- gence Review, 57(12):1-43, 2024.
  7. [Ektefaie et al., 2023] Yasha Ektefaie, George Dasoulas, Ayush Noori, Maha Farhat, and Marinka Zitnik. Multi- modal learning with graphs. Nature Machine Intelligence, 5(4):340-350, April 2023.
  8. [Ilias and Askounis, 2022a] Loukas Ilias and Dimitris Ask- ounis. Multimodal deep learning models for detecting de- mentia from speech and transcripts. Frontiers in Aging Neuroscience, 14:830943, 2022.
  9. [Ilias and Askounis, 2022b] Loukas Ilias and Dimitris Ask- ounis. Multimodal deep learning models for detecting de- mentia from speech and transcripts. Frontiers in Aging Neuroscience, 14, March 2022.
  10. [Latif et al., 2021] Siddique Latif, Junaid Qadir, Adnan Qayyum, Muhammad Usama, and Shahzad Younis. Speech technology for healthcare: Opportunities, chal- lenges, and state of the art. IEEE Reviews in Biomedical Engineering, 14:342-356, 2021.
  11. [Li et al., 2024] Fan Li, Xiaoyang Wang, Dawei Cheng, Wenjie Zhang, Ying Zhang, and Xuemin Lin. Hypergraph self-supervised learning with sampling-efficient signals. In Proceedings of the Thirty-ThirdInternational Joint Confer- ence on Artificial Intelligence, IJCAI-2024. International Joint Conferences on Artificial Intelligence Organization, August 2024. [ IJCAI
  12. [Lin and Washington, 2024] Kaiying Lin and Peter Y. Wash- ington. Multimodal deep learning for dementia classifica- tion using text and audio. Scientific Reports, 14(1), June 2024.
  13. [Liu et al., 2021] Rui Liu, Berrak Sisman, and Haizhou Li. Graphspeech: Syntax-aware graph attention network for neural speech synthesis. In ICASSP 2021 -2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), page 6059-6063. IEEE, June 2021.
  14. [Liu et al., 2024] Xiaoqian Liu, Guoqiang Hu, Yangfan Du, Erfeng He, YingFeng Luo, Chen Xu, Tong Xiao, and Jingbo Zhu. Recent advances in end-to-end simultane- ous speech translation. In Proceedings of the Thirty- ThirdInternational Joint Conference on Artificial Intelli- gence, IJCAI-2024, page 8142-8150. International Joint Conferences on Artificial Intelligence Organization, Au- gust 2024. IJCAI
  15. [Luo et al., 2024] Xuexiong Luo, Jia Wu, Jian Yang, Shan Xue, Amin Beheshti, Quan Z. Sheng, David McAlpine, Paul Sowman, Alexis Giral, and Philip S. Yu. Graph neural networks for brain graph learning: A survey. In Proceed- ings of the Thirty-ThirdInternational Joint Conference on Artificial Intelligence, IJCAI-2024, page 8170-8178. In- ternational Joint Conferences on Artificial Intelligence Or- ganization, August 2024. IJCAI
  16. [Luz et al., 2020] Saturnino Luz, Fasih Haider, Sofia de la Fuente, Davida Fromm, and Brian MacWhinney. Alzheimer's dementia recognition through spontaneous speech: The adress challenge. In Interspeech 2020, interspeech 2020. ISCA, October 2020.
  17. [Luz et al., 2021] Saturnino Luz, Fasih Haider, Sofia de la Fuente, Davida Fromm, and Brian MacWhinney. Detect- ing cognitive decline using speech only: The adresso chal- lenge. In Interspeech 2021, interspeech 2021. ISCA, Au- gust 2021.
  18. [Luz et al., 2024] Saturnino Luz, Fasih Haider, Davida Fromm, Ioulietta Lazarou, Ioannis Kompatsiaris, and Brian MacWhinney. An overview of the adress-m signal processing grand challenge on multilingual alzheimer's dementia recognition through spontaneous speech. IEEE Open Journal of Signal Processing, 5:738-749, 2024.
  19. [Martinc and Pollak, 2020] Matej Martinc and Senja Pollak. Tackling the adress challenge: A multimodal approach to the automated recognition of alzheimer's dementia. In Interspeech 2020, interspeech 2020, page 2157-2161. ISCA, October 2020.
  20. [Marvi et al., 2024] Fahimeh Marvi, Yun-Hsuan Chen, and Mohamad Sawan. Alzheimer's disease diagnosis in the preclinical stage: Normal aging or dementia. IEEE Re- views in Biomedical Engineering, page 1-18, 2024.
  21. [Pacheco-Lorenzo et al., 2024] Moisés R. Pacheco-Lorenzo, Heidi Christensen, Luis E. Anido-Rifón, Manuel J. Fernández-Iglesias, and Sonia M. Valladares-Rodríguez. Analysis of voice biomarkers for the detection of cogni- tive impairment. IEEE Access, 12:122840-122851, 2024.
  22. [Peng et al., 2024] Ciyuan Peng, Jiayuan He, and Feng Xia. Learning on multimodal graphs: A survey. ArXiv Preprint ArXiv:2402.05322, 2024.
  23. [Petti et al., 2020] Ulla Petti, Simon Baker, and Anna Ko- rhonen. A systematic literature review of automatic alzheimer's disease detection from speech and language. Journal of the American Medical Informatics Association, 27(11):1784-1797, 2020.
  24. [Priyadarshinee et al., 2023] Prachee Priyadarshinee, Christopher Johann Clarke, Jan Melechovsky, Cindy Ming Ying Lin, Balamurali BT, and Jer-Ming Chen. Alzheimer's dementia speech (audio vs. text): Multi- modal machine learning at high vs. low resolution. Applied Sciences, 13(7):4244, 2023.
  25. [Pulido et al., 2020] María Luisa Barragán Pulido, Jesús Bernardino Alonso Hernández, Miguel Ángel Fer- rer Ballester, Carlos Manuel Travieso González, Jiří Mekyska, and Zdeněk Smékal. Alzheimer's disease and automatic speech analysis: a review. Expert systems with applications, 150:113213, 2020.
  26. [Pérez-Toro et al., 2021] P.A. Pérez-Toro, S.P. Bayerl, T. Arias-Vergara, J.C. Vásquez-Correa, P. Klumpp, M. Schuster, Elmar Nöth, J.R. Orozco-Arroyave, and K. Riedhammer. Influence of the interviewer on the automatic assessment of alzheimer's disease in the context of the adresso challenge. In Interspeech 2021, interspeech 2021, page 3785-3789. ISCA, August 2021.
  27. [Rohanian et al., 2021] Morteza Rohanian, Julian Hough, and Matthew Purver. Alzheimer's dementia recognition using acoustic, lexical, disfluency and speech pause fea- tures robust to noisy inputs. In Interspeech 2021, inter- speech 2021, page 3820-3824. ISCA, August 2021.
  28. [Rumelhart et al., 1986] David E Rumelhart, James L Mc- Clelland, PDP Research Group, et al. Parallel distributed processing, volume 1: Explorations in the microstructure of cognition: Foundations. The MIT press, 1986.
  29. [Sanh, 2019] V Sanh. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
  30. [Searle et al., 2020] Thomas Searle, Zina Ibrahim, and Richard Dobson. Comparing natural language processing techniques for alzheimer's dementia prediction in spon- taneous speech. In Interspeech 2020, interspeech 2020, page 2192-2196. ISCA, October 2020.
  31. [Shehzad et al., 2024] Ahsan Shehzad, Feng Xia, Shagufta Abid, Ciyuan Peng, Shuo Yu, Dongyu Zhang, and Karin Verspoor. Graph transformers: A survey, 2024.
  32. [Shehzad et al., 2025] Ahsan Shehzad, Dongyu Zhang, Shuo Yu, Shagufta Abid, and Feng Xia. Dynamic graph trans- former for brain disorder diagnosis. IEEE Journal of Biomedical and Health Informatics, page 1-14, 2025.
  33. [Tamm et al., 2023] Bastiaan Tamm, Rik Vandenberghe, and Hugo Van Hamme. Cross-lingual transfer learning for alzheimer's detection from spontaneous speech. In ICASSP 2023 -2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), page 1-2. IEEE, June 2023.
  34. [Turrisi et al., 2024] Rosanna Turrisi, Margherita Squillario, Giulia Abate, Daniela Uberti, and Annalisa Barla. An overview of data integration in neuroscience with focus on alzheimer's disease. IEEE Journal of Biomedical and Health Informatics, 28(4):1824-1835, April 2024.
  35. [Venugopalan et al., 2021] Janani Venugopalan, Li Tong, Hamid Reza Hassanzadeh, and May D Wang. Multimodal deep learning models for early detection of alzheimer's disease stage. Scientific reports, 11(1):3254, 2021.
  36. [Vrindha et al., 2023] M. K. Vrindha, V. Geethu, P. R. Anurenjan, S. Deepak, and K. G. Sreeni. A review of alzheimer's disease detection from spontaneous speech and text. In 2023 International Conference on Control, Communication and Computing (ICCC), page 1-5. IEEE, May 2023.
  37. [Yang et al., 2022] Qin Yang, Xin Li, Xinyun Ding, Feiyang Xu, and Zhenhua Ling. Deep learning-based speech anal- ysis for alzheimer's disease detection: a literature review. Alzheimer's Research & Therapy, 14(1):186, 2022.
  38. [Ying et al., 2021] Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? In M. Ranzato, A. Beygelz- imer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Sys- tems, volume 34, pages 28877-28888.
  39. Curran Associates, Inc., 2021.
  40. [Ying et al., 2023] Yangwei Ying, Tao Yang, and Hong Zhou. Multimodal fusion for alzheimer's disease recog- nition. Applied Intelligence, 53(12):16029-16040, 2023.
  41. [Yu et al., 2024] Shuo Yu, Shan Jin, Ming Li, Tabinda Sar- war, and Feng Xia. Long-range brain graph transformer. In The Thirty-eighth Annual Conference on Neural Infor- mation Processing Systems, 2024.
  42. [Zhang et al., 2021] Dongyu Zhang, Minghao Zhang, Het- ing Zhang, Liang Yang, and Hongfei Lin. Multimet: A multimodal dataset for metaphor understanding. In Pro- ceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), page 3214-3225. Association for Computational Linguistics, 2021.
  43. [Zhang et al., 2024] Lu Zhang, Junqi Qu, Haotian Ma, Tong Chen, Tianming Liu, and Dajiang Zhu. Exploring alzheimer's disease: a comprehensive brain connectome- based survey. Psychoradiology, 4:kkad033, 2024.
  44. [Zhu et al., 2021] Youxiang Zhu, Abdelrahman Obyat, Xi- aohui Liang, John A. Batsis, and Robert M. Roth. Wavbert: Exploiting semantic and non-semantic speech using wav2vec and bert for dementia detection. In Inter- speech 2021, interspeech 2021, page 3790-3794. ISCA, August 2021.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards