Proceedings

L2M2: A Hierarchical Framework Integrating Large Language Model and Multi-agent Reinforcement Learning

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 99-107.

https://doi.org/10.24963/ijcai.2025/12

Abstract

Multi-agent reinforcement learning (MARL) has demonstrated remarkable success in collaborative tasks, yet faces significant challenges in scaling to complex scenarios requiring sustained planning and coordination across long horizons. While hierarchical approaches help decompose these tasks, they typically rely on hand-crafted subtasks and domain-specific knowledge, limiting their generalizability. We present L2M2, a novel hierarchical framework that leverages large language models (LLMs) for high-level strategic planning and MARL for low-level execution. L2M2 enables zero-shot planning that supports both end-to-end training and direct integration with pre-trained MARL models. Experiments in the VMAS environment demonstrate that L2M2's LLM-guided MARL achieves superior performance while requiring less than 20% of the training samples compared to baseline methods. In the MOSMAC environment, L2M2 demonstrates strong performance with pre-defined subgoals and maintains substantial effectiveness without subgoals - scenarios where baseline methods consistently fail. Analysis through kernel density estimation reveals L2M2's ability to automatically generate appropriate navigation plans, demonstrating its potential for addressing complex multi-agent coordination tasks.

  1. [Agarwal et al., 2023] Ananye Agarwal, Ashish Kumar, Ji- tendra Malik, and Deepak Pathak. Legged Locomotion in Challenging Terrains using Egocentric Vision. In CoRL 2023, volume 205 of Proc. Mach. Learn. Res., pages 403- 415. PMLR, 2023.
  2. [Ahilan and Dayan, 2019] Sanjeevan Ahilan and Peter Dayan. Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning. arXiv:1901.08492v1 [cs.MA], January 2019.
  3. [Bettini et al., 2022] Matteo Bettini, Ryan Kortvelesy, Jan Blumenkamp, and Amanda Prorok. VMAS: A Vector- ized Multi-agent Simulator for Collective Robot Learning. In DARS 2022, pages 42-56. Springer, November 2022.
  4. [Blumenkamp et al., 2024] Jan Blumenkamp, Ajay Shankar, Matteo Bettini, Joshua Bird, and Amanda Prorok. The Cambridge RoboMaster: An Agile Multi-Robot Research Platform. In DARS 2024. Springer, 2024.
  5. [Cao et al., 2024] Yuji Cao, Huan Zhao, Yuheng Cheng, Ting Shu, Yue Chen, Guolong Liu, Gaoqi Liang, Junhua Zhao, Jinyue Yan, and Yun Li. Survey on Large Language Model- Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods. IEEE Trans. Neural Netw. Learn. Syst., pages 1-21, November 2024.
  6. [Carta et al., 2023] Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre-Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. In ICML 2023, volume 202, pages 3676-3713. JMLR.org, July 2023.
  7. [Dalal et al., 2023] Murtaza Dalal, Tarun Chiruvolu, Deven- dra Singh Chaplot, and Ruslan Salakhutdinov. Plan-Seq- Learn: Language Model Guided RL for Solving Long Hori- zon Robotics Tasks. In ICLR 2024, October 2023.
  8. [Ellis et al., 2023] Benjamin Ellis, Jonathan Cook, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Maha- jan, Jakob Foerster, and Shimon Whiteson. SMACv2: An improved benchmark for cooperative multi-agent reinforce- ment learning. In Adv. Neural Inf. Process. Syst., volume 36, pages 37567-37593.
  9. Curran Associates, Inc., 2023.
  10. [Elsayed and Mahmood, 2023] Mohamed Elsayed and A. Rupam Mahmood. Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learning. In ICLR 2023, October 2023.
  11. [Foerster et al., 2018] Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon White- son. Counterfactual multi-agent policy gradients. In AAAI 2018, pages 2974-2982. AAAI Press, February 2018.
  12. [Geng et al., 2024a] Minghong Geng, Shubham Pateria, Bud- hitama Subagdja, and Ah-Hwee Tan. Benchmarking MARL on Long Horizon Sequential Multi-Objective Tasks. In AA- MAS 2024, pages 2279-2281. IFAAMAS, May 2024.
  13. [Geng et al., 2024b] Minghong Geng, Shubham Pateria, Bud- hitama Subagdja, and Ah-Hwee Tan. HiSOMA: A hierar- chical multi-agent model integrating self-organizing neural networks with multi-agent deep reinforcement learning. Expert Syst. Appl., 252:124117, October 2024.
  14. [Geng et al., 2025] Minghong Geng, Shubham Pateria, Bud- hitama Subagdja, and Ah-Hwee Tan. MOSMAC: A Multi- agent Reinforcement Learning Benchmark on Sequential Multi-objective Tasks. In AAMAS 2025, pages 867-876. IFAAMAS, May 2025.
  15. [Gupta et al., 2023] Nikunj Gupta, G. Srinivasaraghavan, Swarup Mohalik, Nishant Kumar, and Matthew E. Tay- lor. HAMMER: Multi-level coordination of reinforcement learning agents via learned messaging. Neural Comput. Appl., October 2023.
  16. [Hernandez-Leal et al., 2019] Pablo Hernandez-Leal, Michael Kaisers, Tim Baarslag, and Enrique Munoz de Cote. A Survey of Learning in Multiagent Environments: Dealing with Non-Stationarity. arXiv:1707.09183v2 [cs.MA], March 2019.
  17. [Huang et al., 2025] Jen-tse Huang, Eric John Li, Man Ho Lam, Tian Liang, Wenxuan Wang, Youliang Yuan, Wenx- iang Jiao, Xing Wang, Zhaopeng Tu, and Michael Lyu. Competing Large Language Models in Multi-Agent Gam- ing Environments. In ICLR 2025, April 2025.
  18. [Jiang and Agarwal, 2018] Nan Jiang and Alekh Agarwal. Open Problem: The Dependence of Sample Complexity Lower Bounds on Planning Horizon. In COLT 2018, vol- ume 75 of Proc. Mach. Learn. Res., pages 3395-3398.
  19. PMLR, July 2018.
  20. [Li et al., 2023a] Huao Li, Yu Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Charles Lewis, and Katia Sycara. Theory of Mind for Multi-Agent Collaboration via Large Language Models. In EMNLP 2023, pages 180-192. ACL, December 2023.
  21. [Li et al., 2023b] Wenhao Li, Dan Qiao, Baoxiang Wang, Xi- angfeng Wang, Bo Jin, and Hongyuan Zha. Semantically Aligned Task Decomposition in Multi-Agent Reinforce- ment Learning. arXiv:2305.10865v2 [cs.LG], May 2023.
  22. [Liu et al., 2024a] Jijia Liu, Chao Yu, Jiaxuan Gao, Yuqing Xie, Qingmin Liao, Yi Wu, and Yu Wang. LLM-Powered Hierarchical Language Agent for Real-time Human-AI Co- ordination. In AAMAS 2024, pages 1219-1228. IFAAMAS, May 2024.
  23. [Liu et al., 2024b] Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. A dynamic LLM-powered agent net- work for task-oriented agent collaboration. In First Confer- ence on Language Modeling, 2024.
  24. [Llama Team, 2024] Llama Team. The Llama 3 Herd of Mod- els. arXiv:2407.21783v3 [cs.AI], 2024.
  25. [Lowe et al., 2017] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environ- ments. In Adv. Neural Inf. Process. Syst., volume 30, pages 6379-6390.
  26. [Mahajan et al., 2019] Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. MAVEN: Multi-Agent Variational Exploration. In Advances in Neural Information Processing Systems, volume 32, Red Hook, NY, USA, 2019.
  27. Curran Associates, Inc. [Mandi et al., 2024]
  28. Zhao Mandi, Shreeya Jain, and Shuran Song. RoCo: Dialectic Multi-Robot Collaboration with Large Language Models. In ICRA 2024, pages 286-299, May 2024.
  29. [Oroojlooy and Hajinezhad, 2023] Afshin Oroojlooy and Davood Hajinezhad. A review of cooperative multi-agent deep reinforcement learning. Appl. Intell., 53(11):13677- 13722, June 2023.
  30. [Papoudakis et al., 2021] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks. In Proc. Neural Inf. Pro- cess. Syst. Track on Datasets and Benchmarks, volume 1.
  31. [Rashid et al., 2018] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX: Monotonic Value Function Fac- torisation for Deep Multi-Agent Reinforcement Learning. In ICML 2018, volume 80 of Proc. Mach. Learn. Res., pages 4295-4304. PMLR, July 2018.
  32. [Samvelyan et al., 2019] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. In AAMAS 2019, pages 2186-2188. IFAAMAS, May 2019.
  33. [Son et al., 2019] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Hostallero, and Yung Yi. QTRAN: Learning to Factorize with Transformation for Cooperative Multi- Agent Reinforcement Learning. In ICML 2019, volume 97 of Proc. Mach. Learn. Res., pages 5887-5896. PMLR, 2019.
  34. [Sunehag et al., 2018] Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zam- baldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value- Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward. In AAMAS 2018, pages 2085-2087. IFAAMAS, July 2018.
  35. [Szot et al., 2024] Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Rin Metcalf, Walter Talbott, Natalie Mackraz, R. Devon Hjelm, and Alexander T. To- shev. Large Language Models as Generalizable Policies for Embodied Tasks. In ICLR 2024, May 2024.
  36. [Tan, 2007] Ah-Hwee Tan. Direct code access in self- organizing neural networks for reinforcement learning. In IJCAI 2007, pages 1071-1076.
  37. [Vezhnevets et al., 2017] Alexander Sasha Vezhnevets, Si- mon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. FeUdal Networks for Hierarchical Reinforcement Learning. In ICML 2017, pages 3540-3549. PMLR, July 2017.
  38. [Vinyals et al., 2017] Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, John Quan, Stephen Gaffney, Stig Petersen, Karen Simonyan, Tom Schaul, Hado van Has- selt, David Silver, Timothy Lillicrap, Kevin Calderone, Paul Keet, Anthony Brunasso, David Lawrence, Anders Ekermo, Jacob Repp, and Rodney Tsing. StarCraft II: A New Chal- lenge for Reinforcement Learning. arXiv:1708.04782v1 [cs.LG], August 2017.
  39. [Yu et al., 2022] Chao Yu, Akash Velu, Eugene Vinitsky, Jiax- uan Gao, Yu Wang, Alexandre Bayen, and YI WU. The Sur- prising Effectiveness of PPO in Cooperative Multi-Agent Games. In Adv. Neural Inf. Process. Syst., volume 35, pages 24611-24624.
  40. Curran Associates, Inc., 2022.
  41. [Yu et al., 2023] Bangguo Yu, Hamidreza Kasaei, and Ming Cao. Co-NavGPT: Multi-Robot Cooperative Visual Semantic Navigation using Large Language Models. arXiv:2310.07937v2 [cs.RO], December 2023.
  42. [Zhang et al., 2019] Huichu Zhang, Siyuan Feng, Chang Liu, Yaoyao Ding, Yichen Zhu, Zihan Zhou, Weinan Zhang, Yong Yu, Haiming Jin, and Zhenhui Li. CityFlow: A Multi- Agent Reinforcement Learning Environment for Large Scale City Traffic Scenario. In WWW 2019, pages 3620- 3624. ACM, May 2019.
  43. [Zhang et al., 2023] Hongxin Zhang, Weihua Du, Jiaming Shan, Qinhong Zhou, Yilun Du, Joshua B. Tenenbaum, Tianmin Shu, and Chuang Gan. Building Cooperative Embodied Agents Modularly with Large Language Models. In ICLR 2024, October 2023.
  44. [Zhou et al., 2024] Zihao Zhou, Bin Hu, Chenyang Zhao, Pu Zhang, and Bin Liu. Large Language Model as a Policy Teacher for Training Reinforcement Learning Agents. In IJCAI 2024, pages 5671-5679. IJCAI Organization, August 2024. IJCAI

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards