Proceedings

PENTESTLLMAGENT: A Task Dependency Graph Planning-Based Multi-Agent Framework for Automated Penetration Testing

Download BibTeX Cite

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. AI4Tech: AI Enabling Technologies. Pages 6518-6527.

https://doi.org/10.24963/ijcai.2026/725

Abstract

Fully autonomous IP-to-Root penetration testing remains challenging for LLM agents. We conduct an exploratory study on 10 LLMs and introduce AutoPentest-Bench, an end-to-end benchmark with 13 VulnHub targets and 93 sub-tasks. From 130 interaction logs, we identify three challenges: Rigid Strategy, Contextual Forgetting, and Command Generation Hallucination. To address them, we propose PentestLLMAgent, which integrates a Task Dependency Graph (TDG) for dynamic planning and backtracking; a Hierarchical Multi-Agent Architecture (HMA) with function-calling-based tool invocation, output filtering, and semantic compression, and Executable Knowledge-Guided Command Generation (EKG-CG) for retrieving and executing pre-validated, environment-compatible commands. Evaluations demonstrate strong effectiveness: on end-to-end AutoPentest-Bench, PentestLLMAgent achieves a 77% success rate; on AutoPT’s web exploitation benchmark, it attains a 95% overall pass rate; and on the privilege escalation benchmark, it achieves a 100% success rate. In realistic end-to-end runs, it averages 10.9 minutes, 36 interaction rounds, and 69.6K tokens per target. The code, benchmark, and executable knowledge base are publicly available at https://github.com/sanbai123/PentestLLMAgent_code-and-videos.

  1. [Cheng et al., 2025] Fengxiang Cheng, Haoxuan Li, Fen- rong Liu, Robert van Rooij, Kun Zhang, and Zhouchen Lin. Empowering LLMs with logical reasoning: A com- prehensive survey. In James Kwok, editor, Proceedings of the Thirty-Fourth International Joint Conference on Artifi- cial Intelligence (IJCAI-25), pages 10400-10408. Interna- tional Joint Conferences on Artificial Intelligence Organi- zation, August 2025. Survey Track. IJCAI
  2. [Deng et al., 2024] Gelei Deng, Yi Liu, Víctor Mayoral- Vilches, Peng Liu, Yuekang Li, Yuan Xu, Tianwei Zhang, Yang Liu, Martin Pinzger, and Stefan Rass. PentestGPT: Evaluating and harnessing large language models for auto- mated penetration testing. In 33rd USENIX Security Sym- posium (USENIX Security 24), pages 847-864, Philadel- phia, PA, August 2024. USENIX Association.
  3. [Fang et al., 2024] Richard Fang, Rohan Bindu, Akul Gupta, and Daniel Kang. LLM agents can autonomously exploit one-day vulnerabilities, 2024.
  4. [Ferrag et al., 2025] Mohamed Amine Ferrag, Norbert Ti- hanyi, and Mérouane Debbah. From LLM reasoning to autonomous AI agents: A comprehensive review, 2025. Accessed: 2026-01-20.
  5. [Happe and Cito, 2023] Andreas Happe and Jürgen Cito. Getting pwn'd by AI: Penetration testing with large lan- guage models. In Proceedings of the 31st ACM Joint Eu- ropean Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE '23), pages 2082-2086. ACM, 2023.
  6. [Happe and Cito, 2025] Andreas Happe and Jürgen Cito. On the surprising efficacy of LLMs for penetration-testing, 2025. Accessed: 2026-01-20.
  7. [Kimminich, 2024] Björn Kimminich. OWASP Juice Shop: Probably the most modern and sophisticated insecure web application. OWASP project, 2024.
  8. [Lewis et al., 2020] Patrick Lewis, Ethan Perez, Aleksan- dra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Process- ing Systems, volume 33, pages 9459-9474, 2020.
  9. [National Institute of Standards and Technology, 2026] National Institute of Standards and Technology. NVD JSON 2.0 vulnerability data feed: CVE-2025. National Vulnerability Database data feed, 2026. Year feed; accessed 2026-01-17.
  10. [OpenAI, 2025] OpenAI. Function calling. OpenAI API Documentation, 2025. Accessed: 2026-01-20.
  11. [Plaat et al., 2025] Aske Plaat, Max van Duijn, Niki van Stein, Mike Preuss, Peter van der Putten, and Kees Joost Batenburg. Agentic large language models, a survey. Jour- nal of Artificial Intelligence Research, 84(29), 2025. Ac- cessed: 2026-01-20.
  12. [Scarfone et al., 2008] Karen Scarfone, Murugiah Soup- paya, Amanda Cody, and Angela Orebaugh. Techni- cal guide to information security testing and assessment. Technical Report NIST Special Publication 800-115, Na- tional Institute of Standards and Technology, 2008.
  13. [Shao et al., 2024] Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Mil- ner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, and Muhammad Shafique. NYU CTF Bench: A scalable open- source benchmark dataset for evaluating LLMs in offen- sive security. In Advances in Neural Information Process- ing Systems 37 (NeurIPS 2024), Datasets and Benchmarks Track, 2024.
  14. [Sharma, 2025] Chaitanya Sharma. Retrieval-augmented generation: A comprehensive survey of architectures, en- hancements, and robustness frontiers, 2025. Accessed: 2026-01-20.
  15. [Shen et al., 2025] Xiangmin Shen, Lingzhi Wang, Zhenyuan Li, Yan Chen, Wencheng Zhao, Dawei Sun, Jiashui Wang, and Wei Ruan. PentestAgent: Incor- porating LLM agents to automated penetration testing. In Proceedings of the 20th ACM Asia Conference on Computer and Communications Security (ASIA CCS '25), pages 375-391, Hanoi, Vietnam, 2025. ACM. Accessed: 2026-01-20.
  16. [Tran et al., 2025] Khanh-Tung Tran, Dung Dao, Minh- Duong Nguyen, Quoc-Viet Pham, Barry O'Sullivan, and Hoang D. Nguyen. Multi-agent collaboration mecha- nisms: A survey of LLMs, 2025. Accessed: 2026-01-20.
  17. [World Economic Forum, 2025] World Economic Forum. Global cybersecurity outlook 2025. Technical report, World Economic Forum, 2025. Accessed: 2025-12-26.
  18. [Wu et al., 2025] Benlong Wu, Guoqiang Chen, Kejiang Chen, Xiuwei Shang, Jiapeng Han, Yanru He, Weiming Zhang, and Nenghai Yu. AutoPT: How far are we from the fully automated web penetration testing? IEEE Trans- actions on Information Forensics and Security, 20:9657- 9672, 2025.
  19. [Xu et al., 2024] Jiacen Xu, Jack W. Stokes, Geoff McDon- ald, Xuesong Bai, David Marshall, Siyue Wang, Adith Swaminathan, and Zhou Li. AutoAttacker: A large language model guided system to implement automatic cyber-attacks, 2024.
  20. [Zhu et al., 2026] Yuxuan Zhu, Antony Kellermann, Akul Gupta, Philip Li, Richard Fang, Rohan Bindu, and Daniel Kang. Teams of LLM agents can exploit zero-day vul- nerabilities. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 23-35, Ra- bat, Morocco, March 2026. Association for Computational Linguistics.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards