Proceedings

Integrating Intent Understanding and Optimal Behavior Planning for Behavior Tree Generation from Human Instructions

Download BibTeX Cite

Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence. Main Track. Pages 6832-6840.

https://doi.org/10.24963/ijcai.2024/755

Abstract

Robots executing tasks following human instructions in domestic or industrial environments essentially require both adaptability and reliability. Behavior Tree (BT) emerges as an appropriate control architecture for these scenarios due to its modularity and reactivity. Existing BT generation methods, however, either do not involve interpreting natural language or cannot theoretically guarantee the BTs' success. This paper proposes a two-stage framework for BT generation, which first employs large language models (LLMs) to interpret goals from high-level instructions, then constructs an efficient goal-specific BT through the Optimal Behavior Tree Expansion Algorithm (OBTEA). We represent goals as well-formed formulas in first-order logic, effectively bridging intent understanding and optimal behavior planning. Experiments in the service robot validate the proficiency of LLMs in producing grammatically correct and accurately interpreted goals, demonstrate OBTEA's superiority over the baseline BT Expansion algorithm in various metrics, and finally confirm the practical deployability of our framework. The project website is https://dids-ei.github.io/Project/LLM-OBTEA.

  1. [Ahn et al., 2022] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, et al. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
  2. [Banerjee, 2018] Bikramjit Banerjee. Autonomous Acquisi- tion of Behavior Trees for Robot Control. pages 3460- 3467, 2018.
  3. [Brown et al., 2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Win- ter, Christopher Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few- shot learners. 2020.
  4. [Cai et al., 2021] Zhongxuan Cai, Minglong Li, Wanrong Huang, and Wenjing Yang. BT Expansion: A Sound and Complete Algorithm for Behavior Planning of Intelligent Robots with Behavior Trees. pages 6058-6065. AAAI Press, 2021.
  5. [Chaudhuri et al., 2021] Swarat Chaudhuri, Kevin Ellis, Oleksandr Polozov, Rishabh Singh, Armando Solar- Lezama, Yisong Yue, et al. Neurosymbolic programming. Foundations and Trends® in Programming Languages, 7(3):158-243, 2021.
  6. [Chen et al., 2023] Yaran Chen, Wenbo Cui, Yuanwen Chen, Mining Tan, Xinyao Zhang, Dongbin Zhao, and He Wang. RoboGPT: An intelligent agent of making embodied long- term decisions for daily instruction tasks. arXiv preprint arXiv:2311.15649, 2023.
  7. [Colledanchise and Ögren, 2018] Michele Colledanchise and Petter Ögren. Behavior Trees in Robotics and AI: An Introduction. CRC Press, 2018.
  8. [Colledanchise et al., 2019] Michele Colledanchise, Ramviyas Parasuraman, and Petter Ögren. Learning of Behavior Trees for Autonomous Agents. IEEE Transactions on Games, 11(2):183-189, 2019.
  9. [Driess et al., 2023] Danny Driess, Fei Xia, Mehdi SM Saj- jadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023.
  10. [Fikes and Nilsson, 1971] Richard E. Fikes and Nils J. Nils- son. Strips: A new approach to the application of theorem proving to problem solving. 2(3-4):189-208, December 1971. IJCAI
  11. [French et al., 2019] Kevin French, Shiyu Wu, Tianyang Pan, Zheming Zhou, and Odest Chadwicke Jenkins. Learning behavior trees from demonstration. In 2019 International Conference on Robotics and Automation (ICRA), pages 7791-7797. IEEE, 2019.
  12. [Hodel, 2013] Richard E Hodel. An Introduction to Mathe- matical Logic. Courier Corporation, 2013.
  13. [Li et al., 2021] Shen Li, Daehyung Park, Yoonchang Sung, Julie A Shah, and Nicholas Roy. Reactive task and mo- tion planning under temporal logic specifications. In 2021 IEEE International Conference on Robotics and Automa- tion (ICRA), pages 12618-12624. IEEE, 2021.
  14. [Liang et al., 2022] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as Policies: Language Model Programs for Embodied Control. In arXiv Preprint arXiv:2209.07753, 2022.
  15. [Liang et al., 2023] Xiwen Liang, Liang Ma, Shanshan Guo, Jianhua Han, Hang Xu, Shikui Ma, and Xiaodan Liang. MO-VLN: A Multi-Task Benchmark for Open-set Zero- Shot Vision-and-Language Navigation. arXiv, 2023.
  16. [Lim et al., 2010] Chong-U Lim, Robin Baumgarten, and Simon Colton. Evolving behaviour trees for the com- mercial game DEFCON. In Applications of Evolution- ary Computation: EvoApplicatons 2010: EvoCOMPLEX, EvoGAMES, EvoIASP, EvoINTELLIGENCE, EvoNUM, and EvoSTOC, Istanbul, Turkey, April 7-9, 2010, Proceed- ings, Part I, pages 100-110. Springer, 2010.
  17. [Liu et al., 2023a] Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. LLM+P: Empowering Large Language Models with Opti- mal Planning Proficiency. arXiv, 2023.
  18. [Liu et al., 2023b] Chengwu Liu, Jianhao Shen, Huajian Xin, Zhengying Liu, Ye Yuan, Haiming Wang, Wei Ju, Chuanyang Zheng, Yichun Yin, Lin Li, et al. Fimo: A challenge formal dataset for automated theorem proving. arXiv preprint arXiv:2309.04295, 2023.
  19. [Lykov and Tsetserukou, 2023] Artem Lykov and Dzmitry Tsetserukou. LLM-BRAIn: AI-driven Fast Generation of Robot Behaviour Tree based on Large Language Model. arXiv preprint arXiv:2305.19352, 2023.
  20. [Lykov et al., 2023] Artem Lykov, Maria Dronova, Niko- lay Naglov, Mikhail Litvinov, Sergei Satsevich, Artem Bazhenov, Vladimir Berman, Aleksei Shcherbak, and Dzmitry Tsetserukou. LLM-MARS: Large Language Model for Behavior Tree Generation and NLP-enhanced Dialogue in Multi-Agent Robot Systems. arXiv preprint arXiv:2312.09348, 2023.
  21. [Neupane and Goodrich, 2019] Aadesh Neupane and Michael Goodrich. Learning Swarm Behaviors using Grammatical Evolution and Behavior Trees. pages 513-520. IJCAI Organization, 2019. IJCAI
  22. [Neupane and Goodrich, 2023] Aadesh Neupane and Michael A Goodrich. Designing Behavior Trees from Goal-Oriented LTLf Formulas. arXiv preprint arXiv:2307.06399, 2023.
  23. [Pereira and Engel, 2015] Renato de Pontes Pereira and Paulo Martins Engel. A framework for constrained and adaptive behavior-based agents. arXiv preprint arXiv:1506.02312, 2015.
  24. [Puig et al., 2018] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Tor- ralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8494- 8502, 2018.
  25. [Scheide et al., 2021] Emily Scheide, Graeme Best, and Ge- offrey A. Hollinger. Behavior Tree Learning for Robotic Task Planning through Monte Carlo DAG Search over a Formal Grammar. pages 4837-4843, Xi'an, China, May 2021. IEEE.
  26. [Shridhar et al., 2020] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for ev- eryday tasks. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 10740-10749, 2020.
  27. [Singh et al., 2022] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Trem- blay, Dieter Fox, Jesse Thomason, and Animesh Garg. ProgPrompt: Generating Situated Robot Task Plans using Large Language Models. September 2022.
  28. [Song et al., 2023] Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M. Sadler, Wei-Lun Chao, and Yu Su. LLM-Planner: Few-Shot Grounded Planning for Embod- ied Agents with Large Language Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2998-3009, 2023.
  29. [Tadewos et al., 2022] Tadewos G Tadewos, Abdullah Al Redwan Newaz, and Ali Karimoddini. Specification- guided behavior tree synthesis and execution for coordination of autonomous systems. Expert Systems with Applications, 201:117022, 2022.
  30. [Zeng et al., 2022] Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al. Socratic models: Composing zero- shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022.
  31. [Zhu et al., 2023] Zhaocheng Zhu, Yuan Xue, Xinyun Chen, Denny Zhou, Jian Tang, Dale Schuurmans, and Hanjun Dai. Large Language Models can Learn Rules. arXiv, October 2023.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards