Proceedings

LLM-based Collaborative Agents with Pedagogy-guided Interaction Modeling for Timely Instructive Feedback Generation in Task-oriented Group Discussions

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. AI and Social Good. Pages 9972-9980.

https://doi.org/10.24963/ijcai.2025/1108

Abstract

Large language models (LLMs) fundamentally reshape learning and teaching models, shifting tutoring systems from supporting individual learning to facilitating collaborative learning (CL) like task-oriented group discussions. However, existing AI tutors struggle to guide CL, as they seldom model the interactions between AI tutors and students. Therefore, they cannot scaffold students to complete tasks collaboratively, which impairs learning outcomes and pedagogy adaptability. Additionally, existing AI tutors fail to make use of CL theories to generate instructive feedback, which leads to undesirable interactions such as over-instruction and limits students' autonomy. In this paper, we propose an LLM-based collaborative agent that innovatively leverages pedagogical strategies to sense discussion stages, detect learning issues, identify the timing of intervention, and generate instructive feedback. To develop the agent, we first design a prompting strategy based on a CL theory, that is, the Community of Inquiry, to cultivate the agent to understand the discussion status. Second, a multi-agent interaction framework is proposed to simulate the collaborative learning behavior between AI tutors and students. Meanwhile, a synthetic task-oriented group discussion dataset, namely CLTeach, is generated, which consists of 27k manually-verified multi-party dialogues with fine-grained annotations of instructive feedback and explanations. Lastly, we use CLTeach to fine-tune the LLM agent, ultimately enabling it to generate instructive feedback at the right time to support students in CL. Extensive experiments demonstrate that our agent achieves state-of-the-art performance in feedback generation and has the potential to mimic human teachers effectively.

  1. [Achiam et al., 2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Alt- man, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. [Akyol and Garrison, 2011] Zehra Akyol and D Randy Gar- rison. Assessing metacognition in an online community of inquiry. The Internet and higher education, 14(3):183- 190, 2011.
  3. [Bai et al., 2023] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023.
  4. [Cai et al., 2024] Zhenyao Cai, Seehee Park, Nia Nixon, and Shayan Doroudi. Advancing knowledge together: inte- grating large language model-based conversational ai in small group collaborative learning. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1-9, 2024.
  5. [Carletta et al., 2005] Jean Carletta, Simone Ashby, Se- bastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, et al. The ami meeting corpus: A pre-announcement. In International workshop on ma- chine learning for multimodal interaction, pages 28-39. Springer, 2005.
  6. [Chen et al., 2024] Jiahao Chen, Zitao Liu, Mingliang Hou, Xiangyu Zhao, and Weiqi Luo. Multi-turn classroom di- alogue dataset: Assessing student performance from one- on-one conversations. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 5333-5337, 2024.
  7. [Dan et al., 2023] Yuhao Dan, Zhikai Lei, Yiyang Gu, Yong Li, Jianghao Yin, Jiaju Lin, Linhao Ye, Zhiyan Tie, Yougen Zhou, Yilei Wang, et al. Educhat: A large-scale language model-based chatbot system for intelligent edu- cation. arXiv preprint arXiv:2308.02773, 2023.
  8. [Ding et al., 2024] Yuyang Ding, Hanglei Hu, Jie Zhou, Qin Chen, Bo Jiang, and Liang He. Boosting large language models with socratic method for conversational mathemat- ics teaching. In Proceedings of the 33rd ACM Interna- tional Conference on Information and Knowledge Man- agement, pages 3730-3735, 2024.
  9. [Gan et al., 2023] Wensheng Gan, Zhenlian Qi, Jiayang Wu, and Jerry Chun-Wei Lin. Large language models in ed- ucation: Vision and opportunities. In 2023 IEEE inter- national conference on big data (BigData), pages 4776- 4785. IEEE, 2023.
  10. [Garrison and Arbaugh, 2007] D Randy Garrison and J Ben Arbaugh. Researching the community of inquiry frame- work: Review, issues, and future directions. The Internet and higher education, 10(3):157-172, 2007.
  11. [Garrison, 2022] DR Garrison. Shared metacognition in a community of inquiry. Online learning, 26(1):6-18, 2022.
  12. [Gottipati et al., 2013] Swapna Gottipati, Minghui Qiu, Yanchuan Sim, Jing Jiang, and Noah A. Smith. Learn- ing topics and positions from Debatepedia. In David Yarowsky, Timothy Baldwin, Anna Korhonen, Karen Livescu, and Steven Bethard, editors, Proceedings of the 2013 Conference on Empirical Methods in Natural Lan- guage Processing, pages 1858-1868, Seattle, Washington, USA, October 2013. Association for Computational Lin- guistics.
  13. [Han et al., 2023] Jieun Han, Haneul Yoo, Yoonsu Kim, Junho Myung, Minsun Kim, Hyunseung Lim, Juho Kim, Tak Yeon Lee, Hwajung Hong, So-Yeon Ahn, et al. Recipe: How to integrate chatgpt into efl writing educa- tion. In Proceedings of the Tenth ACM Conference on Learning@ Scale, pages 416-420, 2023.
  14. [Holstein et al., 2019] Kenneth Holstein, Bruce M McLaren, and Vincent Aleven. Designing for complementarity: Teacher and student needs for orchestration support in ai- enhanced classrooms. In Artificial Intelligence in Educa- tion: 20th International Conference, AIED 2019, Chicago, IL, USA, June 25-29, 2019, Proceedings, Part I 20, pages 157-171. Springer, 2019.
  15. [Hu et al., 2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  16. [Janin et al., 2003] Adam Janin, Don Baron, Jane Edwards, Dan Ellis, David Gelbart, Nelson Morgan, Barbara Peskin, Thilo Pfau, Elizabeth Shriberg, Andreas Stolcke, et al. The icsi meeting corpus. In 2003 IEEE International Confer- ence on Acoustics, Speech, and Signal Processing, 2003. Proceedings.(ICASSP'03)., volume 1, pages I-I. IEEE, 2003.
  17. [Jiang et al., 2023] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
  18. [Jin et al., 2024] Hyoungwook Jin, Seonghee Lee, Hyungyu Shin, and Juho Kim. Teach ai how to code: Using large llnguage models as teachable agents for programming ed- ucation. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pages 1-28, 2024.
  19. [Joshi et al., 2017] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale dis- tantly supervised challenge dataset for reading compre- hension. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Pa- pers), pages 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics.
  20. [Karadzhov et al., 2023] Georgi Karadzhov, Tom Stafford, and Andreas Vlachos. Delidata: A dataset for delibera- tion in multi-party problem solving. Proceedings of the ACM on Human-Computer Interaction, 7(CSCW2):1-25, 2023.
  21. [Laal and Laal, 2012] Marjan Laal and Mozhgan Laal. Col- laborative learning: What is it? Procedia-Social and Be- havioral Sciences, 31:491-495, 2012.
  22. [Lin, 2004] Chin-Yew Lin. ROUGE: A package for auto- matic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
  23. [Liu et al., 2024a] Jiayu Liu, Zhenya Huang, Tong Xiao, Jing Sha, Jinze Wu, Qi Liu, Shijin Wang, and Enhong Chen. Socraticlm: Exploring socratic personalized teach- ing with large language models. In The Thirty-eighth An- nual Conference on Neural Information Processing Sys- tems, 2024.
  24. [Liu et al., 2024b] Zhengyuan Liu, Stella Xin Yin, Geyu Lin, and Nancy F. Chen. Personality-aware student simulation for conversational intelligent tutoring systems. In Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 626-642, Miami, Florida, USA, November 2024. Association for Computa- tional Linguistics.
  25. [Macina et al., 2023] Jakub Macina, Nico Daheim, Sankalan Pal Chowdhury, Tanmay Sinha, Manu Ka- pur, Iryna Gurevych, and Mrinmaya Sachan. Mathdial: A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems. arXiv preprint arXiv:2305.14536, 2023.
  26. [Nguyen et al., 2024] Ha Nguyen, Victoria Nguyen, Saríah López-Fierro, Sara Ludovise, and Rossella Santagata. Simulating climate change discussion with large language models: Considerations for science communication at scale. In Proceedings of the Eleventh ACM Conference on Learning @ Scale, L@S '24, page 28-38, New York, NY, USA, 2024. Association for Computing Machinery.
  27. [Pal Chowdhury et al., 2024] Sankalan Pal Chowdhury, Vilém Zouhar, and Mrinmaya Sachan. Autotutor meets large language models: A language model tutor with rich pedagogy and guardrails. In Proceedings of the Eleventh ACM Conference on Learning@ Scale, pages 5-15, 2024.
  28. [Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: A method for au- tomatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Compu- tational Linguistics, pages 311-318, 2002.
  29. [Rafailov et al., 2024] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your lan- guage model is secretly a reward model. Advances in Neu- ral Information Processing Systems, 36, 2024.
  30. [Rajpurkar et al., 2016] Pranav Rajpurkar, Jian Zhang, Kon- stantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas, November 2016. Association for Computational Linguis- tics. [Shani et al., 2024]
  31. Lior Shani, Aviv Rosenberg, Asaf Cas- sel, Oran Lang, Daniele Calandriello, Avital Zipori, Hila Noga, Orgad Keller, Bilal Piot, Idan Szpektor, et al. Multi- turn reinforcement learning from preference human feed- back. arXiv preprint arXiv:2405.14655, 2024.
  32. [Shea and Bidjerano, 2009] Peter Shea and Temi Bidjerano. Cognitive presence and online learner engagement: A cluster analysis of the community of inquiry framework. Journal of Computing in Higher Education, 21:199-217, 2009.
  33. [Shea and Bidjerano, 2010] Peter Shea and Temi Bidjerano. Learning presence: Towards a theory of self-efficacy, self- regulation, and the development of a communities of in- quiry in online and blended learning environments. Com- puters & education, 55(4):1721-1731, 2010.
  34. [Shea et al., 2012] Peter Shea, Suzanne Hayes, Sedef Uzuner Smith, Jason Vickers, Temi Bidjerano, Alexandra Pickett, Mary Gozza-Cohen, Jane Wilde, and Shoubang Jian. Learning presence: Additional research on a new conceptual element within the community of inquiry (coi) framework. The internet and higher education, 15(2):89-95, 2012.
  35. [Sonkar et al., 2023] Shashank Sonkar, Naiming Liu, Deb- shila Mallick, and Richard Baraniuk. CLASS: A design framework for building intelligent tutoring systems based on learning science principles. In Findings of the Associ- ation for Computational Linguistics: EMNLP 2023, pages 1941-1961, Singapore, December 2023. Association for Computational Linguistics.
  36. [Sonkar et al., 2024] Shashank Sonkar, Kangqi Ni, Sapana Chaudhary, and Richard Baraniuk. Pedagogical alignment of large language models. In Findings of the Associa- tion for Computational Linguistics: EMNLP 2024, pages 13641-13650, Miami, Florida, USA, November 2024. As- sociation for Computational Linguistics.
  37. [Wang et al., 2024] Shen Wang, Tianlong Xu, Hang Li, Chaoli Zhang, Joleen Liang, Jiliang Tang, Philip S Yu, and Qingsong Wen. Large language models for education: A survey and outlook. arXiv preprint arXiv:2403.18105, 2024.
  38. [Wei et al., 2021] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
  39. [Xu and Zhang, 2023] Songlin Xu and Xinyu Zhang. Lever- aging generative artificial intelligence to simulate student learning behavior. arXiv preprint arXiv:2310.19206, 2023.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards