Proceedings

Directional Hallucinations: Ideological Drift in News-Grounded LLM Question Answering

Download BibTeX Cite

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. AI and Social Good. Pages 7474-7482.

https://doi.org/10.24963/ijcai.2026/831

Abstract

Large language models (LLMs) are increasingly used to answer questions about political information, including in election-adjacent information settings where factual errors and ideological distortions are high-stakes. We present a reproducible measurement framework that treats hallucinations, unsupported statements in document-grounded QA, as diagnostic signals of ideological drift. Using 21,727 expert-labeled U.S. political news articles from QBias spanning left, center, and right sources, we (i) generate an article-specific question, (ii) elicit document-grounded answers from three open-weight LLMs and one proprietary model, (iii) detect sentence-level hallucinations via reference-based comparison, (iv) classify the ideological valence of hallucinated sentences with a fine-tuned stance classifier, and (v) probe output logits to relate token-level uncertainty to hallucination and drift. Hallucination rates vary substantially across models and concentrate in contentious topics, while source-ideology differences in hallucination frequency are modest. In contrast, hallucination content exhibits robust leftward drift: a majority of hallucinated sentences are classified as left-leaning, including among hallucinations generated from right-leaning sources. Logit-level analysis shows hallucinations arise in high-entropy generation contexts, and in some models uncertainty also predicts leftward drift, consistent with an "uncertainty → guessing" mechanism. We discuss implications for auditing AI-mediated political information and for designing safeguards in election-relevant deployments.

  1. [Allcott and Gentzkow, 2017] Hunt Allcott and Matthew Gentzkow. Social media and fake news in the 2016 elec- tion. Journal of economic perspectives, 31(2):211-236, 2017.
  2. [Bai et al., 2025] Hui Bai, Jan G Voelkel, Shane Muldowney, Johannes C Eichstaedt, and Robb Willer. Llm-generated messages can persuade humans on policy issues. Nature Communications, 16(1):6037, 2025.
  3. [Bang et al., 2024] Yejin Bang, Delong Chen, Nayeon Lee, and Pascale Fung. Measuring political bias in large lan- guage models: What is said and how it is said. arXiv preprint arXiv:2403.18932, 2024.
  4. [Bender et al., 2021] Emily M Bender, Timnit Gebru, An- gelina McMillan-Major, and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610- 623, 2021.
  5. [Chen et al., 2026] Jieying Chen, Karen de Jong, Andreas Poole, Jan Burakowski, Elena Elderson Nosti, Joep Windt, and Chendi Wang. Uncovering political bias in large lan- guage models using parliamentary voting records. arXiv preprint arXiv:2601.08785, 2026.
  6. [Cinelli et al., 2025] Cinelli, Gianmarco De Francisci Morales, Alessandro Galeazzi, Walter Quattrociocchi, and Michele Starnini. The echo chamber effect on social media. ResearchGate, February 2025.
  7. [de Swart et al., 2025] Milena de Swart, Floris den Hengst, and Jieying Chen. Detecting linguistic bias in govern- ment documents using large language models. In Guodong Long, Michale Blumestein, Yi Chang, Liane Lewin-Eytan, Zi Helen Huang, and Elad Yom-Tov, editors, Proceedings of the ACM on Web Conference 2025, WWW 2025, Syd- ney, NSW, Australia, 28 April 2025-2 May 2025, pages 5034-5044. ACM, 2025.
  8. [Gallegos et al., 2024] Isabel O Gallegos, Ryan A Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Der- noncourt, Tong Yu, Ruiyi Zhang, and Nesreen K Ahmed. Bias and fairness in large language models: A survey. Computational Linguistics, 50(3):1097-1179, 2024.
  9. [Geifman and El-Yaniv, 2017] Yonatan Geifman and Ran El-Yaniv. Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017.
  10. [Gu et al., 2024] Yuzhe Gu, Ziwei Ji, Wenwei Zhang, Chengqi Lyu, Dahua Lin, and Kai Chen. Anah-v2: Scaling analytical hallucination annotation of large language mod- els. Advances in Neural Information Processing Systems, 37:60012-60039, 2024.
  11. [Guo et al., 2017] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural net- works. In International conference on machine learning, pages 1321-1330. PMLR, 2017.
  12. [Haak and Schaer, 2023] Fabian Haak and Philipp Schaer. Qbias -A Dataset on Media Bias in Search Queries and Query Suggestions. In Proceedings of the 15th ACM Web Science Conference 2023, pages 239-244, April 2023.
  13. [Hackenburg and Margetts, 2024] Kobi Hackenburg and He- len Margetts. Evaluating the persuasive influence of political microtargeting with large language mod- els. Proceedings of the National Academy of Sciences, 121(24):e2403116121, 2024.
  14. [Hackenburg et al., 2025] Kobi Hackenburg, Ben M Tap- pin, Luke Hewitt, Ed Saunders, Sid Black, Hause Lin, Catherine Fist, Helen Margetts, David G Rand, and Christopher Summerfield. The levers of political persua- sion with conversational artificial intelligence. Science, 390(6777):eaea3884, 2025.
  15. [He et al., 2021] Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding shar- ing, 2021.
  16. [Huang et al., 2025] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qiang- long Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Prin- ciples, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1-55, 2025.
  17. [Ji et al., 2024] Ziwei Ji, Yuzhe Gu, Wenwei Zhang, Chengqi Lyu, Dahua Lin, and Kai Chen. Anah: Analyti- cal annotation of hallucinations in large language models. arXiv preprint arXiv:2405.20315, 2024.
  18. [Jiang et al., 2021] Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. How can we know when language models know? on the calibration of language models for question answering. Transactions of the Association for Computational Linguistics, 9:962-977, 2021.
  19. [Kalai et al., 2026] Adam Tauman Kalai, Ofir Nachum, San- tosh S Vempala, and Edwin Zhang. Evaluating large language models for accuracy incentivizes hallucinations. Nature, pages 1-3, 2026.
  20. [Lewis et al., 2020] Patrick Lewis, Ethan Perez, Aleksan- dra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural infor- mation processing systems, 33:9459-9474, 2020.
  21. [Lin et al., 2024] Zichao Lin, Shuyan Guan, Wending Zhang, Huiyan Zhang, Yugang Li, and Huaping Zhang. Towards trustworthy LLMs: A review on debiasing and dehallucinating in large language models. Artificial Intel- ligence Review, 57(9):243, August 2024.
  22. [Lin et al., 2025] Hause Lin, Gabriela Czarnek, Benjamin Lewis, Joshua P White, Adam J Berinsky, Thomas Costello, Gordon Pennycook, and David G Rand. Persuad- ing voters using human-artificial intelligence dialogues. Nature, pages 1-8, 2025.
  23. [Motoki et al., 2025] Fabio Y. S. Motoki, Valdemar Pinho Neto, and Victor Rangel. Assessing political bias and value misalignment in generative artificial intelli- gence. Journal of Economic Behavior & Organization, page 106904, February 2025.
  24. [Navigli et al., 2023] Roberto Navigli, Simone Conia, and Björn Ross. Biases in large language models: origins, in- ventory, and discussion. ACM Journal of Data and Infor- mation Quality, 15(2):1-21, 2023.
  25. [Niu et al., 2024] Cheng Niu, Yuanhao Wu, Juno Zhu, Sil- iang Xu, Kashun Shum, Randy Zhong, Juntong Song, and Tong Zhang. RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models, May 2024.
  26. [Noels et al., 2025] Sander Noels, Guillaume Bied, Maarten Buyl, Alexander Rogiers, Yousra Fettach, Jefrey Lijffijt, and Tijl De Bie. What large language models do not talk about: An empirical study of moderation and censorship practices, 2025.
  27. [Rettenberger et al., 2025] Luca Rettenberger, Markus Reis- chl, and Mark Schutera. Assessing political bias in large language models. Journal of Computational Social Sci- ence, 8(2):42, May 2025.
  28. [Rozado, 2025] David Rozado. Measuring Political Prefer- ences in AI Systems: An Integrative Approach, March 2025.
  29. [Santurkar et al., 2023] Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. Whose Opinions Do Language Models Re- flect?, March 2023.
  30. [Vosoughi et al., 2018] Soroush Vosoughi, Deb Roy, and Sinan Aral. The spread of true and false news online. sci- ence, 359(6380):1146-1151, 2018.
  31. [Weidinger et al., 2021] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021.
  32. [Zhang et al., 2023] Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models, September 2023.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards