Proceedings

EyeCue: Driver Cognitive Distraction Detection via Gaze-Empowered Egocentric Video Understanding

Download BibTeX Cite

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. Main Track. Pages 3737-3745.

https://doi.org/10.24963/ijcai.2026/416

Abstract

Driver cognitive distraction is a major cause of road collisions and remains difficult to detect. Unlike manual or visual distraction, cognitive distraction is diverted by thoughts unrelated to driving, even when the driver appears visually attentive and exhibits no explicit physical movements. In this work, we propose EyeCue, a gaze-empowered egocentric video understanding framework, to detect driver cognitive distraction. A key insight is that cognitive distraction manifests in the interaction between eye gaze and visual context. To capture this interaction, EyeCue integrates eye gaze with egocentric video to enable context-aware modeling of the driver's attention over time. Furthermore, to tackle the limited scale and diversity of existing datasets, we introduce CogDrive, a comprehensive multi-scenario dataset that augments four existing driving datasets with cognitive distraction annotations. Through extensive evaluations on CogDrive, we show that EyeCue achieves the highest accuracy of 74.38%, outperforming 11 baselines from 6 model families by over 7%. Notably, EyeCue can achieve an accuracy of over 70% across various driving scenarios (different road types, times of day, and weather conditions) with strong generalizability. These results highlight the importance of modeling gaze-context interactions and the effectiveness of cross-modal interaction modeling for multimodal cognitive distraction detection. Our codes and CogDrive dataset resources are available at https://github.com/langzhang2000/EyeCue.

  1. [Al-Mekhlafi et al., 2024] Al-Baraa Abdulrahman Al- Mekhlafi, Ahmad Shahrul Nizam Isha, Nicholas Chileshe, Ahmed Farouk Kineber, Muhammad Ajmal, Abdul- lah O Baarimah, and Al-Hussein MH Al-Aidrous. Risk assessment of driver performance in the oil and gas trans- portation industry: Analyzing the relationship between driver vigilance, attention, reaction time, and safe driving practices. Heliyon, 10(6), 2024.
  2. [Bertasius et al., 2021] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, volume 2, page 4, 2021.
  3. [Bhagat et al., 2023] Hirva Bhagat, Sandesh Jain, Lynn Ab- bott, Akash Sonth, and Abhijit Sarkar. Driver gaze fix- ation and pattern analysis in safety critical events. In 2023 IEEE Intelligent Vehicles Symposium (IV), pages 1- 8. IEEE, 2023.
  4. [Cooper et al., 2014] Simon Cooper, Joanne Porter, and Linda Peach. Measuring situation awareness in emergency setting: a systematic review of tools and outcomes. Open Access Emergency Medicine, pages 1-7, 2014.
  5. [Deng et al., 2019] Tao Deng, Hongmei Yan, Long Qin, Thuyen Ngo, and BS Manjunath. How do drivers allocate their potential attention? driving fixation prediction via convolutional neural networks. IEEE Transactions on In- telligent Transportation Systems, 21(5):2146-2154, 2019.
  6. [Du et al., 2020] Na Du, X Jessie Yang, and Feng Zhou. Psy- chophysiological responses to takeover requests in condi- tionally automated driving. Accident Analysis & Preven- tion, 148:105804, 2020.
  7. [Engel et al., 2023] Jakob Engel, Kiran Somasundaram, Michael Goesele, Albert Sun, Alexander Gamino, An- drew Turner, Arjang Talattof, Arnie Yuan, Bilal Souti, Brighid Meredith, et al. Project aria: A new tool for egocentric multi-modal ai research. arXiv preprint arXiv:2308.13561, 2023.
  8. [Fang et al., 2021] Jianwu Fang, Dingxin Yan, Jiahuan Qiao, Jianru Xue, and Hongkai Yu. Dada: Driver attention pre- diction in driving accident scenarios. IEEE transactions on intelligent transportation systems, 23(6):4959-4971, 2021.
  9. [Figalová et al., 2023] Nikol Figalová, Jürgen Pichen, Vanchha Chandrayan, Olga Pollatos, Lewis L Chuang, and Martin Baumann. Manipulating drivers' mental workload: Neuroergonomic evaluation of the speed regulation n-back task using nasa-tlx and auditory p3a. In Adjunct Proceedings of the 15th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pages 145-149, 2023.
  10. [Guo et al., 2024] Zizheng Guo, Qing Liu, Lin Zhang, Zhen- ning Li, and Guofa Li. L-tla: A lightweight driver distrac- tion detection method based on three-level attention mech- anisms. IEEE Transactions on Reliability, 73(4):1731- 1742, 2024.
  11. [Huang et al., 2025] Yexin Huang, Yongbin Lin, Lishengsa Yue, Zhihong Yao, and Jie Wang. From gaze to move- ment: Predicting visual attention for autonomous driving human-machine interaction based on programmatic imi- tation learning. In Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision, pages 26146- 26155, 2025.
  12. [Ilaslan et al., 2023] Muhammet Ilaslan, Chenan Song, Joya Chen, Difei Gao, Weixian Lei, Qianli Xu, Joo Lim, and Mike Shou. Gazevqa: A video question answering dataset for multiview eye-gaze task-oriented collaborations. In Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pages 10462-10479, 2023.
  13. [Jammot et al., 2025] Matthias Jammot, Björn Braun, Paul Streli, Rafael Wampfler, and Christian Holz. egoemo- tion: Egocentric vision and physiological signals for emo- tion and personality recognition in real-world tasks. arXiv preprint arXiv:2510.22129, 2025.
  14. [Kashevnik et al., 2021] Alexey Kashevnik, Roman Shchedrin, Christian Kaiser, and Alexander Stocker. Driver distraction detection methods: A literature review and framework. IEEE Access, 9:60063-60076, 2021.
  15. [Konrad et al., 2024] Robert Konrad, Nitish Padmanaban, J Gabriel Buckmaster, Kevin C Boyle, and Gordon Wet- zstein. Gazegpt: Augmenting human capabilities using gaze-contingent contextual ai for smart eyewear. arXiv preprint arXiv:2401.17217, 2024.
  16. [Li et al., 2023] Guofa Li, Yufei Yuan, Delin Ouyang, Long Zhang, Bangwei Yuan, Xiaoyu Chang, Zizheng Guo, and Gang Guo. Driver distraction from the eeg perspective: A review. IEEE Sensors Journal, 24(3):2329-2349, 2023.
  17. [Lin et al., 2023] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Mu- nan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projec- tion. arXiv preprint arXiv:2311.10122, 2023.
  18. [Maralappanavar et al., 2016] Shweta Maralappanavar, ReenaKumari Behera, and Uma Mudenagudi. Driver's distraction detection based on gaze estimation. In 2016 international conference on advances in computing, com- munications and informatics (icacci), pages 2489-2494. IEEE, 2016.
  19. [Min et al., 2024] Chen Min, Dawei Zhao, Liang Xiao, Jian Zhao, Xinli Xu, Zheng Zhu, Lei Jin, Jianshu Li, Yulan Guo, Junliang Xing, et al. Driveworld: 4d pre-trained scene understanding via world models for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15522- 15533, 2024.
  20. [Musabini and Chetitah, 2020] Antonyo Musabini and Mounsif Chetitah. Heatmap-based method for estimating drivers' cognitive distraction. In 2020 IEEE 19th Interna- tional Conference on Cognitive Informatics & Cognitive Computing (ICCI* CC), pages 179-186. IEEE, 2020.
  21. [NHTSA, 2023] NHTSA. Budget estimates: Fiscal year 2024. Technical report, U.S. Department of Transporta- tion, National Highway Traffic Safety Administration, Washington, D.C., March 2023.
  22. [NHTSA, 2025] NHTSA. Research note: Distracted driving in 2023. Technical Report DOT HS 813 703, U.S. Depart- ment of Transportation, National Highway Traffic Safety Administration, April 2025.
  23. [Ojstersek and Topolsek, 2019] Tina Cvahte Ojstersek and Darja Topolsek. Eye tracking use in researching driver dis- traction: A scientometric and qualitative literature review approach. Journal of eye movement research, 12(3):10- 16910, 2019.
  24. [Oyama et al., 2019] Akane Oyama, Shuko Takeda, Yuki Ito, Tsuneo Nakajima, Yoichi Takami, Yasushi Takeya, Koichi Yamamoto, Ken Sugimoto, Hideo Shimizu, Mune- hisa Shimamura, et al. Novel method for rapid as- sessment of cognitive impairment using high-performance eye-tracking technology. Scientific reports, 9(1):12932, 2019.
  25. [Palazzi et al., 2018] Andrea Palazzi, Davide Abati, Francesco Solera, Rita Cucchiara, et al. Predicting the driver's focus of attention: the dr (eye) ve project. IEEE transactions on pattern analysis and machine intelligence, 41(7):1720-1733, 2018.
  26. [Pei et al., 2024] Baoqi Pei, Guo Chen, Jilan Xu, Yuping He, Yicheng Liu, Kanghua Pan, Yifei Huang, Yali Wang, Tong Lu, Limin Wang, et al. Egovideo: Exploring egocen- tric foundation model and downstream adaptation. arXiv preprint arXiv:2406.18070, 2024.
  27. [Qiao et al., 2025] Yu Qiao, Xiaohui Yang, Jing Wang, Tongzhen Si, and Qingbei Guo. Driver cognitive distrac- tion detection based on eye movement behavior and inte- gration of multi-view space-channel feature. Expert Sys- tems with Applications, 266:125975, 2025.
  28. [Rekimoto, 2025] Jun Rekimoto. Gazellm: Multimodal llms incorporating human visual attention. arXiv preprint arXiv:2504.00221, 2025.
  29. [Sarkar, 2022] Abhijit Sarkar. A comprehensive safety anal- ysis for gaze fixation of drivers to outside scene. Human Factors in Transportation, 60(60), 2022.
  30. [Sonth et al., 2023] Akash Sonth, Abhijit Sarkar, Hirva Bha- gat, and Lynn Abbott. Explainable driver activity recogni- tion using video transformer in highly automated vehicle. In 2023 IEEE Intelligent Vehicles Symposium (IV), pages 1-8. IEEE, 2023.
  31. [Tong et al., 2022] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre- training. Advances in neural information processing sys- tems, 35:10078-10093, 2022.
  32. [Vaswani et al., 2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  33. [Wang et al., 2024] Yi Wang, Kunchang Li, Xinhao Li, Ji- ashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, et al. Internvideo2: Scal- ing foundation models for multimodal video understand- ing. In European Conference on Computer Vision, pages 396-416. Springer, 2024.
  34. [Xia et al., 2018] Ye Xia, Danqing Zhang, Jinkyu Kim, Ken Nakayama, Karl Zipser, and David Whitney. Predicting driver attention in critical situations. In Asian conference on computer vision, pages 658-674. Springer, 2018.
  35. [Yan et al., 2024] Kun Yan, Zeyu Wang, Lei Ji, Yuntao Wang, Nan Duan, and Shuai Ma. Voila-a: Aligning vision- language models with user's gaze attention. Advances in Neural Information Processing Systems, 37:1890-1918, 2024.
  36. [Zhang et al., 2025] Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106, 2025.
  37. [Zhang et al., 2026] Lang Zhang, JinYi Yoon, Matthew Cor- bett, Abhijit Sarkar, and Bo Ji. Eyecue: Driver cogni- tive distraction detection via gaze-empowered egocentric video understanding. arXiv preprint arXiv:2605.07859, 2026.
  38. [Zhou et al., 2024] Yunsong Zhou, Linyan Huang, Qingwen Bu, Jia Zeng, Tianyu Li, Hang Qiu, Hongzi Zhu, Minyi Guo, Yu Qiao, and Hongyang Li. Embodied understand- ing of driving scenarios. In European Conference on Com- puter Vision, pages 129-148. Springer, 2024.
  39. [Zhou et al., 2025] Yuchen Zhou, Jiayu Tang, Xiaoyan Xiao, Yueyao Lin, Linkai Liu, Zipeng Guo, Hao Fei, Xiaobo Xia, and Chao Gou. Where, what, why: Towards explainable driver attention prediction. arXiv preprint arXiv:2506.23088, 2025.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards