Proceedings

SRJudge: Empowering Large Language Models with Selective Reasoning for Fine-Grained Knowledge Concept Tagging

Download BibTeX Cite

Proceedings of the Thirty-Fifth International Joint Conference on Artificial Intelligence. Main Track. Pages 6036-6044.

https://doi.org/10.24963/ijcai.2026/672

Abstract

Knowledge concept tagging aims to assign specific concept or topic labels to educational content, which is essential for both educators and learners in traditional and online teaching practices. Recent work has explored large language models (LLMs) for this task, achieving promising performance. However, LLMs still struggle to select the correct concept from a large-scale candidate set due to the high dimensionality of the decision space. In this paper, we propose a novel three-stage Select-Reason-Judge (SRJudge) framework, which empowers LLMs with selective reasoning capability for fine-grained knowledge concept tagging. Specifically, the Selector in Stage 1 first narrows the candidate concepts to a top-K shortlist by fine-tuning a small language model (SLM), e.g., BERT, since the top-K predictions hit the correct concept in most cases, thereby reducing the decision space of correct candidates. Next, the Stage 2 Reasoner employs a lightweight LLM for refined reasoning over the shortlisted candidates. It further integrates an improved reinforcement learning strategy with a dynamic task-specific reward function and a pruning mechanism to better align with human reasoning preferences. Finally, a larger LLM acts as a judger that evaluates the overall rationality of the reasoning process and its explanations to determine the final output. In addition, we construct two high-quality datasets for further validation, i.e., the biology dataset S_Bio and the physics dataset S_Phy. Experimental results demonstrate that our method consistently outperforms state-of-the-art baselines across benchmark datasets, verifying its effectiveness and superiority. Resources are available at: https://github.com/Nicozwy/SRJudge.

  1. [Achiam et al., 2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Alt- man, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint:2303.08774, 2023.
  2. [Bai et al., 2023] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint:2309.16609, 2023.
  3. [Cui et al., 2024] Chaoran Cui, Yumo Yao, Chunyun Zhang, Hebo Ma, Yuling Ma, Zhaochun Ren, Chen Zhang, and James Ko. Dgekt: A dual graph ensemble learning method for knowledge tracing. ACM Transactions on Information Systems, 42(3):1-24, 2024.
  4. [Devlin et al., 2019] Jacob Devlin, Ming-Wei Chang, Ken- ton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In Proceedings of the 2019 conference of the North American chapter of the association for computational lin- guistics: human language technologies, pages 4171-4186, 2019.
  5. [Ding et al., 2025a] Ziqi Ding, Xiaolu Wang, Yuzhuo Wu, Guitao Cao, and Liangyu Chen. Tagging knowledge con- cepts for math problems based on multi-label text classi- fication. Expert Systems with Applications, 267:126232, 2025.
  6. [Ding et al., 2025b] Ziqi Ding, Xiaolu Wang, Yuzhuo Wu, Guitao Cao, and Liangyu Chen. Tagging knowledge con- cepts for math problems based on multi-label text classi- fication. Expert Systems with Applications, 267:126232, 2025.
  7. [Ghosh et al., 2024] Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Deepali Aneja, Zeyu Jin, Ramani Duraiswami, Dinesh Manocha, et al. A closer look at the limitations of instruction tuning. In Proceedings of the 41st International Conference on Machine Learning, 2024.
  8. [Gu, 2023] Qiuhan Gu. Llm-based code generation method for golang compiler testing. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineer- ing, pages 2201-2203, 2023.
  9. [Guo et al., 2025] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shi- rong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning. arXiv preprint:2501.12948, 2025.
  10. [Huang et al., 2023] Tao Huang, Shengze Hu, Huali Yang, Jing Geng, Sannyuya Liu, Hao Zhang, and Zongkai Yang. Pqsct: Pseudo-siamese bert for concept tagging with both questions and solutions. IEEE Transactions on Learning Technologies, 16(5):831-846, 2023.
  11. [Kullback and Leibler, 1951] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79-86, 1951.
  12. [Li et al., 2022] Qian Li, Hao Peng, Jianxin Li, Congying Xia, Renyu Yang, Lichao Sun, Philip S Yu, and Lifang He. A survey on text classification: From traditional to deep learning. ACM Transactions on Intelligent Systems and Technology, 13(2):1-41, 2022.
  13. [Li et al., 2024] Shiyu Li, Yang Tang, Shizhe Chen, and Xi Chen. Conan-embedding: General text embed- ding with more and better negative samples. arXiv preprint:2408.15710, 2024.
  14. [Li et al., 2025] Hang Li, Tianlong Xu, Ethan Chang, and Qingsong Wen. Knowledge tagging with large language model based multi-agent system. In Proceedings of the 39th AAAI Conference on Artificial Intelligence, pages 28775-28782, 2025.
  15. [Liu et al., 2019] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint:1907.11692, 2019.
  16. [Liu et al., 2024] Yonghao Liu, Lan Huang, Fausto Giunchiglia, Xiaoyue Feng, and Renchu Guan. Improved graph contrastive learning for short text classification. In Proceedings of the 38th AAAI conference on artificial intelligence, pages 18716-18724, 2024.
  17. [Moore et al., 2024] Steven Moore, Robin Schmucker, Tom Mitchell, and John Stamper. Automated generation and tagging of knowledge components from multiple-choice questions. In Proceedings of the 7th ACM conference on learning@ scale, pages 122-133, 2024.
  18. [Ozyurt et al., 2024] Yilmazcan Ozyurt, Stefan Feuerriegel, and Mrinmaya Sachan. Automated knowledge concept annotation and question representation learning for knowl- edge tracing. arXiv preprint:2410.01727, 2024.
  19. [Ozyurt et al., 2025] Yilmazcan Ozyurt, Tunaberk Almaci, Stefan Feuerriegel, and Mrinmaya Sachan. Personalized exercise recommendation with semantically-grounded knowledge tracing. In Advances in Neural Information Processing Systems, 2025.
  20. [Saha and Schneider, 2013] Avigit K Saha and Kevin A Schneider. A discriminative model approach for suggest- ing tags automatically for stack overflow questions. In 2013 10th Working Conference on Mining Software Repos- itories, pages 73-76. IEEE, 2013.
  21. [Shao et al., 2024] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open lan- guage models. arXiv preprint:2402.03300, 2024.
  22. [Sun et al., 2018] Bo Sun, Yunzong Zhu, Yongkang Xiao, Rong Xiao, and Yungang Wei. Automatic question tag- ging with deep neural networks. IEEE Transactions on Learning Technologies, 12(1):29-43, 2018.
  23. [Thirunavukarasu et al., 2023] Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 29(8):1930-1940, 2023.
  24. [Touvron et al., 2023] Hugo Touvron, Thibaut Lavril, Gau- tier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- othée Lacroix, Baptiste Rozière, Naman Goyal, Eric Ham- bro, Faisal Azhar, et al. Llama: Open and efficient founda- tion language models. arXiv preprint:2302.13971, 2023.
  25. [Wang et al., 2024] Shanshan Wang, Zhen Zeng, Xun Yang, Ke Xu, and Xingyi Zhang. Boosting neural cognitive diag- nosis with student's affective state modeling. In Proceed- ings of the 38th AAAI conference on artificial intelligence, pages 620-627, 2024.
  26. [Xia et al., 2025] Shijie Xia, Xuefeng Li, Yixin Liu, Tong- shuang Wu, and Pengfei Liu. Evaluating mathematical reasoning beyond accuracy. In Proceedings of the 39th AAAI Conference on Artificial Intelligence, pages 27723- 27730, 2025.
  27. [Yang et al., 2022] Jiuding Yang, Weidong Guo, Bang Liu, Yakun Yu, Chaoyue Wang, Jinwen Luo, Linglong Kong, Di Niu, and Zhen Wen. Tag: Toward accurate social me- dia content tagging with a concept graph. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Dis- covery and Data Mining, pages 4332-4341, 2022.
  28. [Yang et al., 2025] Zhiwei Yang, Jiahua Yang, Longtao Wang, Rongxin Huo, Huiru Lin, Yuxuan Zhou, and Weiqi Luo. Local-global cascaded ensemble learning on hybrid experts for knowledge concept tagging. In Proceedings of the 26th International Conference on Artificial Intelli- gence in Education, pages 408-421. Springer, 2025.
  29. [Zhang et al., 2021] Xiao Zhang, Meng Liu, Jianhua Yin, Zhaochun Ren, and Liqiang Nie. Question tagging via graph-guided ranking. ACM Transactions on Information Systems, 40(1):1-23, 2021.
  30. [Zhang et al., 2025] Yazhou Zhang, Mengyao Wang, Qiuchi Li, Prayag Tiwari, and Jing Qin. Pushing the limit of llm capacity for text classification. In Companion Proceedings of the ACM on Web Conference 2025, pages 1524-1528, 2025.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards