Proceedings

BioDisco: Multi-Agent Hypothesis Generation with Dual-Mode Evidence, Iterative Feedback and Temporal Evaluation

Download BibTeX Cite

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. AI and Health. Pages 6761-6769.

https://doi.org/10.24963/ijcai.2026/752

Abstract

Identifying novel hypotheses is essential to scientific research, yet this process risks being overwhelmed by the sheer volume and complexity of available information. Existing automated methods often struggle to generate novel and evidence-grounded hypotheses, lack robust iterative refinement and rarely undergo rigorous temporal evaluation for future discovery potential. To address this, we propose BIODISCO, a multi-agent framework that draws upon language model-based reasoning and a dual-mode evidence system (biomedical knowledge graphs and automated literature retrieval) for grounded novelty, integrates an internal scoring and feedback loop for iterative refinement, and validates performance through pioneering temporal and human evaluations and a Bradley-Terry paired comparison model for statistical assessment. Evaluations suggest improved novelty and significance relative to ablated configurations and a generalist biomedical agent. Designed for flexibility and modularity, BIODISCO allows seamless integration of custom language models or knowledge graphs, and can be run with just a few lines of code.

  1. [Aamer et al., 2025] Naafey Aamer, Muhammad Nabeel Asim, Shan Munir, and Andreas Dengel. Automating AI discovery for biomedicine through knowledge graphs and LLM agents. bioRxiv, 2025.
  2. [Alkan et al., 2025] Atilla Kaan Alkan, Shashwat Sourav, Maja Jablonska, Simone Astarita, Rishabh Chakrabarty, Nikhil Garuda, Pranav Khetarpal, Maciej Pióro, Dimitrios Tanoglidis, Kartheik G Iyer, et al. A survey on hypoth- esis generation for scientific discovery in the era of large language models, 2025. arXiv:2504.05496.
  3. [Baek et al., 2025] Jinheon Baek, Sujay Kumar Jauhar, Sil- viu Cucerzan, and Sung Ju Hwang. ResearchAgent: It- erative research idea generation over scientific literature with large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, pages 6709-6738, April 2025.
  4. [Bélisle-Pipon, 2024] Jean-Christophe Bélisle-Pipon. Why we need to be careful with LLMs in medicine. Frontiers in Medicine, 11:1495582, 2024.
  5. [Bianchi et al., 2025] Owen Bianchi, Maya Willey, Chelsea X Alvarado, Benjamin Danek, Marzieh Khani, Nicole Kuznetsov, Anant Dadu, Syed Shah, Mathew J Koretsky, Mary B Makarious, et al. CARDBiomed- Bench: A benchmark for evaluating large language model performance in biomedical research: A novel question-and-answer benchmark designed to assess large language models' comprehension of biomedical research, piloted on neurodegenerative diseases, 2025. bioRxiv.
  6. [Bradley and Terry, 1952] Ralph Allan Bradley and Mil- ton E. Terry. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3- 4):324-345, 12 1952.
  7. [Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Sys- tems, 33:1877-1901, 2020.
  8. [Bürkner, 2021] Paul-Christian Bürkner. Bayesian item re- sponse modeling in R with brms and Stan. Journal of Sta- tistical Software, 100(5):1-54, 2021.
  9. [Chandak et al., 2023] Payal Chandak, Kexin Huang, and Marinka Zitnik. Building a knowledge graph to enable precision medicine. Scientific Data, 10(1):67, 2023.
  10. [Deka et al., 2022] Pritam Deka, Anna Jurek-Loughrey, et al. Evidence extraction to validate medical claims in fake news detection. In International Conference on Health Information Science, pages 3-15. Springer, 2022.
  11. [Dorner et al., 2024] Florian E Dorner, Vivian Y Nastl, and Moritz Hardt. Limits to scalable evaluation at the fron- tier: LLM as judge won't beat twice the data, 2024. arXiv:2410.13341.
  12. [Elo, 1978] Arpad E. Elo. The Rating of Chess Players, Past and Present. Arco Pub., New York, 1978.
  13. [Firth, 2004] D. Firth. Quasi-variances. Biometrika, 91(1):65-80, March 2004.
  14. [Ghafarollahi and Buehler, 2024] Alireza Ghafarollahi and Markus J. Buehler. SciAgents: Automating scientific dis- covery through bioinspired multi-agent intelligent graph reasoning. Advanced Materials, page 2413523, 2024.
  15. [Gottweis et al., 2025] Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, Khaled Saab, Dan Popovici, Jacob Blum, Fan Zhang, Katherine Chou, Avinatan Hassidim, Burak Gokturk, Amin Vahdat, Pushmeet Kohli, Yossi Matias, Andrew Carroll, Kavita Kulkarni, Nenad Tomasev, Yuan Guan, Vikram Dhillon, Eeshit Dhaval Vaishnav, Byron Lee, Tiago R D Costa, José R Penadés, Gary Peltz, Yun- han Xu, Annalisa Pawlosky, Alan Karthikesalingam, and Vivek Natarajan. Towards an AI co-scientist, 2025. [Himmelstein et al., 2017]
  16. Daniel Scott Himmelstein, An- toine Lizee, Christine Hessler, Leo Brueggeman, Sab- rina L Chen, Dexter Hadley, Ari Green, Pouya Khankha- nian, and Sergio E Baranzini. Systematic integration of biomedical knowledge prioritizes drugs for repurposing. elife, 6:e26726, 2017.
  17. [Hristovski et al., 2015] Dimitar Hristovski, Andrej Kastrin, Dejan Dinevski, and Thomas C Rindflesch. Constructing a graph database for semantic literature-based discovery. In MEDINFO 2015: eHealth-enabled Health, pages 1094- 1094. IOS Press, 2015.
  18. [Huang et al., 2025a] Kexin Huang, Ying Jin, Ryan Li, Michael Y. Li, Emmanuel Candes, and Jure Leskovec. Au- tomated hypothesis validation with agentic sequential fal- sifications. In Forty-second International Conference on Machine Learning, 2025.
  19. [Huang et al., 2025b] Kexin Huang, Serena Zhang, Hanchen Wang, Yuanhao Qu, Yingzhou Lu, Yusuf Roohani, Ryan Li, Lin Qiu, Gavin Li, Junze Zhang, Di Yin, Shruti Mar- waha, Jennefer N. Carter, Xin Zhou, Matthew Wheeler, Jonathan A. Bernstein, Mengdi Wang, Peng He, Jingtian Zhou, Michael Snyder, Le Cong, Aviv Regev, and Jure Leskovec. Biomni: A general-purpose biomedical AI agent, 2025.
  20. [Jimeno-Yepes et al., 2009] Antonio Jimeno-Yepes, Rafael Berlanga-Llavori, and Dietrich Rebholz-Schuhmann. Ex- ploitation of ontological resources for scientific literature analysis: searching genes and related diseases. In 2009 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, pages 7073-7078, 2009.
  21. [Jin et al., 2019] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering, 2019. arXiv:1909.06146.
  22. [Kanakarajan et al., 2022] Kamal raj Kanakarajan, Bhu- vana Kundumani, Abhijith Abraham, and Malaikannan Sankarasubbu. BioSimCSE: BioMedical sentence embed- dings using contrastive learning. In Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI), pages 81-86, December 2022.
  23. [Laurent et al., 2024] Jon M Laurent, Joseph D Janizek, Michael Ruzo, Michaela M Hinks, Michael J Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D White, and Samuel G Rodriques. Lab-bench: Measur- ing capabilities of language models for biology research, 2024. arXiv:2407.10362.
  24. [Lewis et al., 2020] Patrick Lewis, Ethan Perez, Aleksan- dra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings of the 34th International Con- ference on Neural Information Processing Systems, 2020.
  25. [Li et al., 2024] Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. LLMs-as-judges: A comprehensive survey on LLM-based evaluation methods, 2024. arXiv:2412.05579.
  26. [Liu et al., 2025] Haokun Liu, Sicong Huang, Jingyu Hu, Yangqiaoyu Zhou, and Chenhao Tan. HypoBench: To- wards systematic and principled benchmarking for hy- pothesis generation, 2025. arXiv:2504.11524.
  27. [Liusie et al., 2023] Adian Liusie, Potsawee Manakul, and Mark JF Gales. LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models, 2023. arXiv:2307.07889.
  28. [Liusie et al., 2024] Adian Liusie, Potsawee Manakul, and Mark Gales. LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computa- tional Linguistics, pages 139-151, March 2024.
  29. [Lu et al., 2024] Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The AI scientist: Towards fully automated open-ended scientific discovery, 2024. arXiv:2408.06292.
  30. [Majumder et al., 2024] Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. Discovery- bench: Towards data-driven discovery with large language models, 2024. arXiv:2407.01725.
  31. [Perdomo-Quinteiro and Belmonte-Hernández, 2024] Pablo Perdomo-Quinteiro and Alberto Belmonte-Hernández. Knowledge graphs for drug repurposing: a review of databases and methods. Briefings in Bioinformatics, 25(6):bbae461, 2024.
  32. [Qi et al., 2024] Biqing Qi, Kaiyan Zhang, Kai Tian, Haoxi- ang Li, Zhang-Ren Chen, Sihang Zeng, Ermo Hua, Hu Jin- fang, and Bowen Zhou. Large language models as biomed- ical hypothesis generators: A comprehensive evaluation. In First Conference on Language Modeling, 2024.
  33. [Rein et al., 2024] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level Google-proof Q&A benchmark. In First Conference on Language Modeling, 2024.
  34. [Ren et al., 2025] Shuo Ren, Pu Jian, Zhenjiang Ren, Chun- lin Leng, Can Xie, and Jiajun Zhang. Towards scientific in- telligence: A survey of LLM-based scientific agents, 2025. arXiv:2503.24047.
  35. [Singhal et al., 2023] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clini- cal knowledge. Nature, 620(7972):172-180, 2023.
  36. [Spangler et al., 2014] Scott Spangler, Angela D. Wilkins, Benjamin J. Bachman, Meena Nagarajan, Tajhal Dayaram, Peter J. Haas, Sam Regenbogen, Curtis R. Pickering, Austin Comer, Jeffrey N. Myers, Ioana Stanoi, Linda Kato, Ana Lelescu, Jacques J. Labrie, Neha Parikh, An- dreas Martin Lisewski, Lawrence A. Donehower, Ying Chen, and Olivier Lichtarge. Automated hypothesis gen- eration based on mining scientific literature. Proceedings of the 20th ACM SIGKDD International Conference on Knowledge discovery and data mining, 2014.
  37. [Sybrandt et al., 2018] Justin Sybrandt, M. Shtutman, and Ilya Safro. Large-scale validation of hypothesis genera- tion systems via candidate ranking. 2018 IEEE Interna- tional Conference on Big Data (Big Data), pages 1494- 1503, 2018.
  38. [Sybrandt et al., 2020] Justin Sybrandt, Ilya Tyagin, Michael Shtutman, and Ilya Safro. Agatha: automatic graph min- ing and transformer based hypothesis generation approach. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 2757- 2764, 2020.
  39. [Xiong et al., 2025] Guangzhi Xiong, Eric Xie, Corey Williams, Myles Kim, Amir Hassan Shariatmadari, Sikun Guo, Stefan Bekiranov, and Aidong Zhang. Toward re- liable scientific hypothesis generation: Evaluating truth- fulness and hallucination in large language models, 2025. arXiv:2505.14599. IJCAI
  40. [Zheng et al., 2020] Shuangjia Zheng, Jiahua Rao, Ying Song, Jixian Zhang, Xianglu Xiao, Evandro Fei Fang, Yuedong Yang, and Zhangming Niu. PharmKG: a ded- icated knowledge graph benchmark for biomedical data mining. Briefings in Bioinformatics, 22(4):bbaa344, 12 2020.
  41. [Zhou et al., 2023] Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. Don't make your LLM an evaluation benchmark cheater, 2023. arXiv:2311.01964.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards