Proceedings

Integrating Answer Set Programming and Large Language Models for Enhanced Structured Representation of Complex Knowledge in Natural Language

Download BibTeX Cite

Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. Main Track. Pages 4330-4338.

https://doi.org/10.24963/ijcai.2025/482

Abstract

Answer Set Programming (ASP) and Large Language Models (LLMs) have emerged as powerful tools in Artificial Intelligence, each offering unique capabilities in knowledge representation and natural language processing, respectively.

In this paper, we combine the strengths of the two paradigms with the aim of improving the structured representation of complex knowledge encoded in natural language.

In a nutshell, the structured representation is obtained by combining syntactic structures extracted by LLMs and semantic aspects encoded in the knowledge base.

The interaction between ASP and LLMs is driven by a YAML file specifying prompt templates and domain-specific background knowledge.

The proposed approach is evaluated using a set of benchmarks based on a dataset obtained from problems of ASP Competitions.

The results of our experiment show that ASP can sensibly improve the F1-score, especially when relatively small models are used.

  1. [Alviano and Grillo, 2024] Mario Alviano and Lorenzo Grillo. Answer set programming and large language models interaction with YAML: preliminary report. In Emanuele De Angelis and Maurizio Proietti, editors, CILC 2024, Rome, Italy, June 26-28, 2024, volume 3733 of CEUR Workshop Proceedings. CEUR-WS.org, 2024.
  2. [Alviano et al., 2024] Mario Alviano, Fabrizio Lo Scudo, Lorenzo Grillo, and Luis Angel Rodriguez Reiners. An- swer set programming and large language models interac- tion with YAML: second report. In Lucía Gómez Álvarez et al., editor, KoDis+CAKR+SYNERGY@KR 2024, Hanoi, Vietnam, November 2-8, 2024, volume 3876 of CEUR Workshop Proceedings. CEUR-WS.org, 2024.
  3. [Basu et al., 2021] Kinjal Basu, Sarat Chandra Varanasi, Farhad Shakerin, Joaquín Arias, and Gopal Gupta. Knowledge-driven natural language understanding of en- glish text and its applications. In AAAI 2021, February 2-9, 2021, pages 12554-12563. AAAI Press, 2021.
  4. [Blloshmi et al., 2021] Rexhina Blloshmi, Simone Conia, Rocco Tripodi, Roberto Navigli, et al. Generating senses and roles: An end-to-end model for dependency-and span- based semantic role labeling. In IJCAI 2021, pages 3786- 3793, 2021.
  5. [Brown and et al., 2020] Tom B. Brown and et al. Lan- guage models are few-shot learners. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Bal- can, and Hsuan-Tien Lin, editors, NeurIPS 2020, Decem- ber 6-12, 2020, virtual, 2020.
  6. [Cabot and Navigli, 2021] Pere-Lluís Huguet Cabot and Roberto Navigli. Rebel: Relation extraction by end-to-end language generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2370- 2381, 2021.
  7. [Calimeri et al., 2020] Francesco Calimeri, Wolfgang Faber, Martin Gebser, Giovambattista Ianni, Roland Kaminski, Thomas Krennwallner, Nicola Leone, Marco Maratea, Francesco Ricca, and Torsten Schaub. ASP-Core-2 input language format. TPLP, 20(2):294-309, 2020.
  8. [Cao et al., 2021] Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. Autoregressive entity retrieval. In ICLR. OpenReview.net, 2021.
  9. [Cappanera et al., 2023] Paola Cappanera, Marco Gavanelli, Maddalena Nonato, and Marco Roma. Logic-based ben- ders decomposition in answer set programming for chronic outpatients scheduling. TPLP, 23(4):848-864, 2023.
  10. [Cardellini et al., 2024] Matteo Cardellini, Paolo De Nardi, Carmine Dodaro, Giuseppe Galatà, Anna Giardini, Marco Maratea, and Ivan Porro. Solving rehabilitation schedul- ing problems via a two-phase ASP approach. TPLP, 24(2):344-367, 2024.
  11. [Caruso et al., 2024] Simone Caruso, Carmine Dodaro, Marco Maratea, Marco Mochi, and Francesco Riccio. CNL2ASP: converting controlled natural language sen- tences into ASP. TPLP, 24(2):196-226, 2024.
  12. [Chowdhery and et al., 2023] Aakanksha Chowdhery and et al. Palm: Scaling language modeling with pathways. J. Mach. Learn. Res., 24:240:1-240:113, 2023.
  13. [Dabre et al., 2020] Raj Dabre, Chenhui Chu, and Anoop Kunchukuttan. A survey of multilingual neural machine translation. ACM Computing Surveys (CSUR), 53(5):1- 38, 2020.
  14. [Gebser et al., 2011] Martin Gebser, Roland Kaminski, and Torsten Schaub. Complex optimization in answer set pro- gramming. TPLP, 11(4-5):821-839, 2011.
  15. [Gebser et al., 2020] Martin Gebser, Marco Maratea, and Francesco Ricca. The seventh answer set programming competition: Design and results. TPLP, 20(2):176-204, 2020.
  16. [Gelfond and Lifschitz, 1990] Michael Gelfond and Vladimir Lifschitz. Logic programs with classical negation. In D. Warren and Peter Szeredi, editors, Logic Programming: Proc. of the Seventh International Conference, pages 579-597, 1990.
  17. [Hoffmann et al., 2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.
  18. [Ishay et al., 2023a] Adam Ishay, Zhun Yang, and Joohyung Lee. Leveraging large language models to generate an- swer set programs. In Proceedings of the 20th Interna- tional Conference on Principles of Knowledge Represen- tation and Reasoning, pages 374-383, 2023.
  19. [Ishay et al., 2023b] Adam Ishay, Zhun Yang, and Joohyung Lee. Leveraging Large Language Models to Generate An- swer Set Programs. In KR 2023, pages 374-383, 8 2023.
  20. [Jin et al., 2024] Hanlei Jin, Yang Zhang, Dan Meng, Jun Wang, and Jinghua Tan. A comprehensive survey on process-oriented automatic text summarization with ex- ploration of llm-based methods. CoRR, abs/2403.02901, 2024.
  21. [Kalyanpur et al., 2024] Aditya Kalyanpur, Kailash Sara- vanakumar, Victor Barres, Jennifer Chu-Carroll, David Melville, and David A. Ferrucci. LLM-ARC: enhanc- ing llms with an automated reasoning critic. CoRR, abs/2406.17663, 2024.
  22. [Kaur et al., 2025] Navdeep Kaur, Lachlan McPheat, Alessandra Russo, Anthony G Cohn, and Pranava Mad- hyastha. An empirical study of conformal prediction in llm with asp scaffolds for robust reasoning, 2025.
  23. [Li et al., 2024] Zhiming Li, Yushi Cao, Xiufeng Xu, Junzhe Jiang, Xu Liu, Yon Shin Teo, Shang-Wei Lin, and Yang Liu. Llms for relational reasoning: How far are we? In LLM4CODE@ICSE, pages 119-126, 2024.
  24. [Lin et al., 2024] Xinrui Lin, Yangfan Wu, Huanyu Yang, Yu Zhang, Yanyong Zhang, and Jianmin Ji. CLMASP: coupling large language models with answer set program- ming for robotic task planning. CoRR, abs/2406.03367, 2024.
  25. [Liu et al., 2022] Zihan Liu, Mostofa Patwary, Ryan Prenger, Shrimai Prabhumoye, Wei Ping, Mohammad Shoeybi, and Bryan Catanzaro. Multi-stage prompting for knowledgeable dialogue generation. In ACL 2022, pages 1317-1337, 2022.
  26. [Marek and Truszczynski, 1999] Victor W. Marek and Miroslaw Truszczynski. Stable models and an alternative logic programming paradigm. In The Logic Program- ming Paradigm, Artificial Intelligence, pages 375-398. Springer, 1999.
  27. [Niemelä, 1999] Ilkka Niemelä. Logic programs with stable model semantics as a constraint programming paradigm. Ann. Math. Artif. Intell., 25(3-4):241-273, 1999.
  28. [Nye et al., 2021] Maxwell Nye, Michael Tessler, Josh Tenenbaum, and Brenden M Lake. Improving coher- ence and consistency in neural sequence models with dual- system, neuro-symbolic reasoning. Advances in Neural Information Processing Systems, 34:25192-25204, 2021.
  29. [Petroni et al., 2019] Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. Language models as knowledge bases? arXiv preprint arXiv:1909.01066, 2019.
  30. [Rajasekharan et al., 2023a] Abhiramon Rajasekharan, Yankai Zeng, and Gopal Gupta. Argument analysis using answer set programming and semantics-guided large language models. In ICLP Workshops, 2023.
  31. [Rajasekharan et al., 2023b] Abhiramon Rajasekharan, Yankai Zeng, Parth Padalkar, and Gopal Gupta. Reliable natural language understanding with large language models and answer set programming. In ICLP 2023, Imperial College London, UK, 9th July 2023 -15th July 2023, volume 385 of EPTCS, pages 274-287, 2023.
  32. [Santana et al., 2024] Manuel A. Borroto Santana, Irfan Ka- reem, and Francesco Ricca. Towards automatic compo- sition of ASP programs from natural language specifica- tions. In IJCAI 2024, Jeju, South Korea, August 3-9, 2024, pages 6198-6206. ijcai.org, 2024. IJCAI
  33. [Taupe et al., 2021] Richard Taupe, Gerhard Friedrich, Kon- stantin Schekotihin, and Antonius Weinzierl. Solving configuration problems with ASP and declarative domain specific heuristics. In Michel Aldanondo, Andreas A. Falkner, Alexander Felfernig, and Martin Stettinger, edi- tors, Proceedings of the 23rd International Configuration Workshop (CWS/ConfWS 2021), Vienna, Austria, 16-17 September, 2021, volume 2945 of CEUR Workshop Pro- ceedings, pages 13-20. CEUR-WS.org, 2021.
  34. [Touvron and et al., 2023] Hugo Touvron and et al. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023.
  35. [Wang et al., 2024] Zhongsheng Wang, Jiamou Liu, Qiming Bao, Hongfei Rong, and Jingfeng Zhang. Chatlogic: Inte- grating logic programming with large language models for multi-step reasoning. In International Joint Conference on Neural Networks, IJCNN 2024, Yokohama, Japan, June 30 -July 5, 2024, pages 1-8. IEEE, 2024.
  36. [Wotawa, 2020] Franz Wotawa. On the use of answer set programming for model-based diagnosis. In Hamido Fu- jita, Philippe Fournier-Viger, Moonis Ali, and Jun Sasaki, editors, IEA/AIE 2020, Kitakyushu, Japan, September 22- 25, 2020, Proceedings, volume 12144 of LNCS, pages 518-529. Springer, 2020.
  37. [Yang et al., 2023] Zhun Yang, Adam Ishay, and Joohyung Lee. Coupling large language models with logic pro- gramming for robust and general reasoning from text. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5186-5219, 2023.
  38. [Yang et al., 2024] Xiaocheng Yang, Bingsen Chen, and Yik-Cheung Tam. Arithmetic reasoning with LLM: prolog generation & permutation. In Kevin Duh, Helena Gómez- Adorno, and Steven Bethard, editors, Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies: Short Papers, NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pages 699-710. Associa- tion for Computational Linguistics, 2024.
  39. [Zeng et al., 2024] Yankai Zeng, Abhiramon Rajasekharan, Parth Padalkar, Kinjal Basu, Joaquín Arias, and Gopal Gupta. Automated interactive domain-specific conversa- tional agents that understand human dialogs. In Mar- tin Gebser and Ilya Sergey, editors, PADL 2024, London, UK, January 15-16, 2024, Proceedings, volume 14512 of LNCS, pages 204-222. Springer, 2024.
  40. [Zhang et al., 2019] Haoyu Zhang, Jianjun Xu, and Ji Wang. Pretraining-based natural language generation for text summarization. arXiv preprint arXiv:1902.09243, 2019.
  41. [Zhang et al., 2023] Wenxuan Zhang, Xin Li, Yang Deng, Lidong Bing, and Wai Lam. A survey on aspect-based sentiment analysis: Tasks, methods, and challenges. IEEE Trans. Knowl. Data Eng., 35(11):11019-11038, 2023.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards