Proceedings

R2DQG: A Quality Meets Diversity Framework for Question Generation over Knowledge Bases

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 8231-8240.

https://doi.org/10.24963/ijcai.2025/915

Abstract

The task of Knowledge-Based Question Generation (KBQG) involves generating natural language questions from structured knowledge sources, posing unique challenges in balancing linguistic diversity and semantic relevance. Existing models often focus on maximizing surface-level similarity to ground-truth questions, neglecting the need for diverse syntactic forms and leading to semantic drift during generation. To overcome these challenges, we propose Refine-Reinforced Diverse Question Generation (R2DQG), a two-phase framework leveraging a generation-then-refinement paradigm. The Generator first constructs a diverse set of expressive templates using dependency parse tree similarity, capturing a wide range of syntactic patterns and styles. These templates guide the creation of question drafts, ensuring both diversity and semantic relevance. In the second phase, a Corrector module refines the drafts to mitigate semantic drift and enhance overall coherence and quality. Experiments on public datasets show that R2DQG outperforms state-of-the-art models in generating diverse, contextually accurate questions. Moreover, synthetic datasets generated by R2DQG enhance downstream QA performance, underscoring the practical utility of our approach.

  1. Method Top-3 Questions Top-5 Questions Top-10 Questions D@3 D-1 B-1 ME Avg D@5 D-1 B-1 ME Avg D@10 D-1 B-1 ME Avg Graph2Seqs methods G2S+AE[Chen et al., 2023] 16.08 38.25 41.10 28.25 30.92 18.08 28.29 41.21 28.26 28.96 19.92 18.14 41.01 28.21 26.82 G2S+AE+RL[Chen et al., 2023] 15.72 37.85 42.72 29.80 31.52 17.29 27.79 42.76 29.89 29.43 19.24 17.82 42.79 29.96 27.45 PLMs methods MHQG[Kumar et al., 2019] 10.49 35.25 30.23 25.52 25.37 12.31 26.08 30.36 25.63 23.60 14.24 16.72 30.13 25.40 21.62 MHQG+AE[Kumar et al., 2019] 11.82 35.94 31.84 27.38 26.74 13.55 26.59 31.86 27.53 24.88 15.31 17.05 31.92 27.38 22.92 JointGT[Ke et al., 2021] 17.65 38.83 55.05 32.68 36.05 19.52 28.51 55.08 32.69 33.95 21.59 18.28 54.98 32.49 31.84 DSM[Guo et al., 2022] 17.30 38.64 59.65 33.48 37.27 19.37 28.58 59.63 33.42 35.25 20.95 18.21 59.69 33.57 33.10 DiversifyQG[Guo et al., 2024c] 25.50 42.50 48.70 30.73 36.86 27.22 31.44 48.72 30.82 34.55 28.76 20.03 48.53 30.58 31.98 T5 16.61 38.22 49.35 29.17 33.34 18.22 28.27 49.47 29.35 31.33 20.01 18.01 49.26 28.99 29.07 Bart 16.63 38.47 49.87 29.62 33.65 18.37 28.25 49.87 29.42 31.48 20.05 18.04 49.83 29.6 29.38 LLMs methods GPT-4-Turbo 19.88 41.19 53.35 31.45 36.47 21.56 30.47 53.44 31.48 34.24 23.54 19.41 53.49 31.60 32.01 GPT-3.5-Turbo 19.92 40.91 51.52 30.61 35.74 21.53 30.04 51.42 30.42 33.35 23.13 19.14 51.60 30.62 31.12 KQG-CoT[Liang et al., 2023] 17.68 39.09 54.15 31.25 35.54 19.30 28.92 54.07 31.14 33.36 21.18 18.42 54.26 31.40 31.32 RoleAgentQG[Zhao et al., 2024] 18.55 39.62 54.97 31.91 36.26 20.18 29.09 54.91 31.85 34.01 22.26 18.53 54.86 31.83 31.87 SGSH(B+S)[Guo et al., 2024b] 19.12 41.05 63.84 35.53 39.89 21.01 30.14 63.91 35.61 37.67 22.95 19.20 63.82 35.46 35.36 SGSH[Guo et al., 2024b] 20.05 41.29 65.51 36.22 40.77 21.67 30.32 65.42 36.12 38.38 23.22 19.32 65.49 36.21 36.06 Ours R 2 DQG 36.93 47.32 58.18 33.29 43.93 38.72 36.95 58.41 33.40 41.87 40.64 22.69 58.40 33.38 38.78 Table 1: Overall performance comparison between the three categories KBQG methods and our R 2 DQG method on WQ datasets. Since few KBQG studies address the problem of how to generate both high-quality and diverse questions simultaneously, we have chosen many quality-centered models as baselines. [Key: Best Single Metric; Best Average Metric ]. Method Top-3 Questions Top-5 Questions Top-10 Questions D@3 D-1 B-1 ME Avg D@5 D-1 B-1 ME Avg D@10 D-1 B-1 ME Avg Graph2Seqs methods G2S+AE[Chen et al., 2023] 17.75 33.48 73.70 40.20 41.28 19.90 24.01 73.76 40.34 39.50 21.60 15.45 73.77 40.31 37.78 G2S+AE+RL[Chen et al., 2023] 16.31 33.39 73.94 40.23 40.97 18.24 24.12 73.81 40.12 39.07 19.98 15.62 73.8 40.11 37.38 PLMs methods MHQG[Kumar et al., 2019] 11.18 29.17 55.36 31.63 31.83 12.89 21.07 55.46 31.66 30.27 14.93 13.65 55.23 31.48 28.82 MHQG+AE[Kumar et al., 2019] 12.35 29.43 56.71 32.95 32.86 14.44 21.26 56.81 33.07 31.40 16.15 13.68 56.59 32.85 29.82 JointGT[Ke et al., 2021] 17.55 34.02 78.72 45.80 44.02 19.45 24.40 78.68 45.71 42.06 21.24 15.70 78.65 45.72 40.33 DSM[Guo et al., 2022] 17.66 33.98 81.29 46.90 44.96 19.66 24.37 81.22 46.81 43.02 21.43 15.78 81.41 46.92 41.38 DiversifyQG[Guo et al., 2024c] 28.11 40.60 59.38 33.65 40.44 29.97 29.12 59.45 33.80 38.08 31.71 18.86 59.49 33.71 35.94 T5 16.97 33.67 74.48 40.74 41.46 19.09 24.32 74.65 40.91 39.74 21.04 15.65 74.39 40.74 37.96 Bart 17.20 33.73 76.52 41.78 42.31 19.02 24.19 76.47 41.74 40.36 20.78 15.57 76.64 41.91 38.72 LLMs methods GPT-4-Turbo 20.45 34.2 75.89 40.97 42.88 22.51 24.53 75.86 40.89 40.95 24.31 15.78 75.85 40.84 39.20 GPT-3.5-Turbo 21.40 34.93 73.14 40.63 42.53 23.19 25.05 73.13 40.59 40.49 24.81 16.22 73.11 40.58 38.68 KQG-CoT[Liang et al., 2023] 18.25 33.20 77.79 43.47 43.18 20.10 23.98 77.59 43.46 41.28 21.93 15.53 77.70 43.28 39.61 RoleAgentQG[Zhao et al., 2024] 19.82 33.56 78.90 44.91 44.30 21.86 24.24 78.84 44.75 42.42 23.59 15.70 78.76 44.82 40.72 SGSH(B+S)[Guo et al., 2024b] 20.71 33.88 81.96 48.11 46.16 22.43 24.48 82.14 48.22 44.32 24.35 15.75 82.13 48.30 42.63 SGSH[Guo et al., 2024b] 21.41 34.21 84.74 49.23 47.40 23.33 24.71 84.67 49.22 45.48 25.13 15.90 84.73 49.14 43.72 Ours R 2 DQG 35.41 45.70 72.08 40.28 48.37 38.72 34.61 72.02 40.21 46.39 41.95 21.23 72.07 40.26 43.93
  2. [Agrawal et al., 2024] Garima Agrawal, Kuntal Pal, Yuli Deng, Huan Liu, and Ying-Chih Chen. Cyberq: Generat- ing questions and answers for cybersecurity education us- ing knowledge graph-augmented llms. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 23164-23172, 2024.
  3. [Banerjee and Lavie, 2005] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceed- ings of the acl workshop on intrinsic and extrinsic evalua- tion measures for machine translation and/or summariza- tion, pages 65-72, 2005.
  4. [Bi et al., 2020] Sheng Bi, Xiya Cheng, Yuan-Fang Li, Yongzhen Wang, and Guilin Qi. Knowledge-enriched, type-constrained and grammar-guided question generation over knowledge bases. In Proceedings of the 28th Inter- national Conference on Computational Linguistics, pages 2776-2786, 2020.
  5. [Cao and Wang, 2021] Shuyang Cao and Lu Wang. Control- lable open-ended question generation with a new question type ontology. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6424-6439, 2021.
  6. [Chen et al., 2023] Yu Chen, Lingfei Wu, and Mohammed J Zaki. Toward subgraph-guided knowledge graph question generation with graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2023.
  7. [Dang et al., 2024a] Jisheng Dang, Huicheng Zheng, Xiao- hao Xu, Longguang Wang, and Yulan Guo. Beyond ap- pearance: Multi-frame spatio-temporal context memory networks for efficient and robust video object segmenta- tion. IEEE Transactions on Image Processing, 2024.
  8. [Dang et al., 2024b] Jisheng Dang, Huicheng Zheng, Xiao- hao Xu, Longguang Wang, Qingyong Hu, and Yulan Guo. Adaptive sparse memory networks for efficient and robust video object segmentation. IEEE Transactions on Neural Networks and Learning Systems, 2024.
  9. [Deschamps et al., 2021] Arthur Deschamps, Sujatha Das Gollapalli, and See Kiong Ng. On generating fact-infused question variations. In Proceedings of the International Conference on Recent Advances in Natural Language Pro- cessing (RANLP 2021), pages 335-345, 2021.
  10. [Elsahar et al., 2018] Hady Elsahar, Christophe Gravier, and Frederique Laforest. Zero-shot question generation from knowledge graphs for unseen predicates and entity types. In Proceedings of NAACL-HLT, pages 218-228, 2018.
  11. [Fei et al., 2022] Zichu Fei, Xin Zhou, Tao Gui, Qi Zhang, and Xuan-Jing Huang. Lfkqg: A controlled generation framework with local fine-tuning for question generation over knowledge bases. In Proceedings of the 29th Inter- national Conference on Computational Linguistics, pages 6575-6585, 2022.
  12. [Guo et al., 2022] Shasha Guo, Jing Zhang, Yanling Wang, Qianyi Zhang, Cuiping Li, and Hong Chen. Dsm: Ques- tion generation over knowledge base via modeling diverse subgraphs with meta-learner. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4194-4207, 2022.
  13. [Guo et al., 2024a] Shasha Guo, Lizi Liao, Cuiping Li, and Tat-Seng Chua. A survey on neural question genera- tion: Methods, applications, and prospects. arXiv preprint arXiv:2402.18267, 2024. IJCAI
  14. [Guo et al., 2024b] Shasha Guo, Lizi Liao, Jing Zhang, Yan- ling Wang, Cuiping Li, and Hong Chen. Sgsh: Stimulate large language models with skeleton heuristics for knowl- edge base question generation. In Findings of the Associ- ation for Computational Linguistics: NAACL 2024, pages 4613-4625, 2024.
  15. [Guo et al., 2024c] Shasha Guo, Jing Zhang, Xirui Ke, Cuip- ing Li, and Hong Chen. Diversifying question gen- eration over knowledge base via external natural ques- tions. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Re- sources and Evaluation (LREC-COLING 2024), pages 5096-5108, 2024.
  16. [He et al., 2021] Gaole He, Yunshi Lan, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. Improving multi-hop knowledge base question answering by learning interme- diate supervision signals. In Proceedings of the 14th ACM international conference on web search and data mining, pages 553-561, 2021.
  17. [Holtzman et al., 2019] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neu- ral text degeneration. arXiv preprint arXiv:1904.09751, 2019.
  18. [Hu et al., 2021] Edward J Hu, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
  19. [Jia and Liang, 2016] Robin Jia and Percy Liang. Data re- combination for neural semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 12- 22, 2016.
  20. [Ke et al., 2021] Pei Ke, Haozhe Ji, Yu Ran, Xin Cui, Liwei Wang, Linfeng Song, Xiaoyan Zhu, and Minlie Huang. Jointgt: Graph-text joint representation learning for text generation from knowledge graphs. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2526-2538, 2021.
  21. [Kim et al., 2019] Yanghoon Kim, Hwanhee Lee, Joongbo Shin, and Kyomin Jung. Improving neural question gener- ation using answer separation. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 6602-6609, 2019.
  22. [Kumar et al., 2019] Vishwajeet Kumar, Yuncheng Hua, Ganesh Ramakrishnan, Guilin Qi, Lianli Gao, and Yuan- Fang Li. Difficulty-controllable multi-hop question gener- ation from knowledge graphs. In The Semantic Web-ISWC 2019: 18th International Semantic Web Conference, Auck- land, New Zealand, October 26-30, 2019, Proceedings, Part I 18, pages 382-398. Springer, 2019.
  23. [Li et al., 2024] Jinhong Li, Xuejie Zhang, Jin Wang, and Xiaobing Zhou. Deep question generation model based on dual attention guidance. International Journal of Machine Learning and Cybernetics, pages 1-11, 2024.
  24. [Liang et al., 2023] Yuanyuan Liang, Jianing Wang, Hanlun Zhu, Lei Wang, Weining Qian, and Yunshi Lan. Prompt- ing large language models with chain-of-thought for few- shot knowledge base question generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4329-4343, 2023.
  25. [Lu et al., 2024] Kangkang Lu, Yanhua Yu, Hao Fei, Xuan Li, Zixuan Yang, Zirui Guo, Meiyu Liang, Mengran Yin, and Tat-Seng Chua. Improving expressive power of spec- tral graph neural networks with eigenvalue correction. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 38, pages 14158-14166, 2024.
  26. [Lyu et al., 2021] Chenyang Lyu, Lifeng Shang, Yvette Gra- ham, Jennifer Foster, Xin Jiang, and Qun Liu. Improv- ing unsupervised question answering via summarization- informed question generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4134-4148, 2021.
  27. [Martins et al., 2020] Pedro Henrique Martins, Zita Mar- inho, and André FT Martins. Sparse text generation. In Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pages 4252-4273, 2020.
  28. [Narayan et al., 2022] Shashi Narayan, Gonc ¸alo Simões, Yao Zhao, Joshua Maynez, Dipanjan Das, Michael Collins, and Mirella Lapata. A well-composed text is half done! composition sampling for diverse conditional gen- eration. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1319-1339, 2022.
  29. [Papineni et al., 2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for au- tomatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Compu- tational Linguistics, pages 311-318, 2002.
  30. [Qi et al., 2020] Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. Stanza: A python natural language processing toolkit for many human lan- guages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 101-108, 2020.
  31. [Quan et al., 2019] Zhe Quan, Zhi-Jie Wang, Yuquan Le, Bin Yao, Kenli Li, and Jian Yin. An efficient framework for sentence similarity modeling. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(4):853- 865, 2019.
  32. [Seyler et al., 2017] Dominic Seyler, Mohamed Yahya, and Klaus Berberich. Knowledge questions from knowledge graphs. In Proceedings of the ACM SIGIR international conference on theory of information retrieval, pages 11- 18, 2017.
  33. [Song et al., 2019] Haoyu Song, Weinan Zhang, Yiming Cui, Dong Wang, and Ting Liu. Exploiting persona infor- mation for diverse generation of conversational responses. pages 5190-5196, 08 2019. IJCAI
  34. [Sultan et al., 2020] Md Arafat Sultan, Shubham Chandel, Ramón Fernandez Astudillo, and Vittorio Castelli. On the importance of diversity in question generation for qa. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5651-5656, 2020.
  35. [Sun et al., 2018] Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William Cohen. Open domain question answering using early fusion of knowledge bases and text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4231-4242, 2018.
  36. [Wang et al., 2020] Zhen Wang, Siwei Rao, Jie Zhang, Zhen Qin, Guangjian Tian, and Jun Wang. Diversify question generation with continuous content selectors and question type modeling. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 2134-2143, 2020.
  37. [Wang et al., 2024] Yuling Wang, Changxin Tian, Binbin Hu, Yanhua Yu, Ziqi Liu, Zhiqiang Zhang, Jun Zhou, Liang Pang, and Xiao Wang. Can small language models be good reasoners for sequential recommendation? In Pro- ceedings of the ACM Web Conference 2024, pages 3876- 3887, 2024.
  38. [Zhang and Zhu, 2021] Zhiling Zhang and Kenny Zhu. Di- verse and specific clarification question generation with keywords. In Proceedings of the web conference 2021, pages 3501-3511, 2021.
  39. [Zhao et al., 2024] Runhao Zhao, Jiuyang Tang, Weixin Zeng, Ziyang Chen, and Xiang Zhao. Zero-shot knowl- edge graph question generation via multi-agent llms and small models synthesis. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 3341-3351, 2024.
  40. [Zhou et al., 2018] Mantong Zhou, Minlie Huang, and Xi- aoyan Zhu. An interpretable reasoning network for multi- relation question answering. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2010-2022, 2018.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards