Proceedings

Reframing Spatial Reasoning Evaluation in Language Models: A Real-World Simulation Benchmark for Qualitative Reasoning

Download BibTeX Cite

Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence. Main Track. Pages 6342-6349.

https://doi.org/10.24963/ijcai.2024/701

Abstract

Spatial reasoning plays a vital role in both human cognition and machine intelligence, prompting new research into language models' (LMs) capabilities in this regard. However, existing benchmarks reveal shortcomings in evaluating qualitative spatial reasoning (QSR). These benchmarks typically present oversimplified scenarios or unclear natural language descriptions, hindering effective evaluation. We present a novel benchmark for assessing QSR in LMs, which is grounded in realistic 3D simulation data, offering a series of diverse room layouts with various objects and their spatial relationships. This approach provides a more detailed and context-rich narrative for spatial reasoning evaluation, diverging from traditional, toy-task-oriented scenarios. Our benchmark encompasses a broad spectrum of qualitative spatial relationships, including topological, directional, and distance relations. These are presented with different viewing points, varied granularities, and density of relation constraints to mimic real-world complexities. A key contribution is our logic-based consistency-checking tool, which enables the assessment of multiple plausible solutions, aligning with real-world scenarios where spatial relationships are often open to interpretation. Our benchmark evaluation of advanced LMs reveals their strengths and limitations in spatial reasoning. They face difficulties with multi-hop spatial reasoning and interpreting a mix of different view descriptions, pointing to areas for future improvement.

  1. [Alomari et al., 2022] Muhannad Alomari, Fangjun Li, David C Hogg, and Anthony G Cohn. Online perceptual learning and natural language acquisition for autonomous robots. Artificial Intelligence, 303:103637, 2022.
  2. [Bang et al., 2023] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity. arXiv preprint arXiv:2302.04023, 2023.
  3. [Bommasani et al., 2021] Rishi Bommasani, Drew A Hud- son, Ehsan Adeli, Russ Altman, Simran Arora, Syd- ney von Arx, Michael S Bernstein, Jeannette Bohg, An- toine Bosselut, Emma Brunskill, et al. On the oppor- tunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
  4. [Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing sys- tems, 33:1877-1901, 2020.
  5. [Cohn and Hernandez-Orallo, 2023] Anthony G Cohn and Jose Hernandez-Orallo. Dialectical language model evaluation: An initial appraisal of the commonsense spatial reasoning abilities of LLMs. arXiv preprint arXiv:2304.11164, 2023.
  6. [Cohn and Renz, 2008] Anthony G Cohn and Jochen Renz. Qualitative spatial representation and reasoning. In Frank Van Harmelen, Vladimir Lifschitz, and Bruce Porter, ed- itors, Handbook of knowledge representation, pages 551- 596. Elsevier, 2008.
  7. [Deitke et al., 2022] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, et al. Procthor: Large-scale embodied ai using procedural gen- eration. Advances in Neural Information Processing Sys- tems, 35:5982-5994, 2022.
  8. [Haviland, 1998] J B Haviland. Guugu Yimithirr cardinal directions. Ethos, 26(1):25-47, 1998.
  9. [Li et al., 2022] Fangjun Li, DC Hogg, and AG Cohn. Ontol- ogy knowledge-enhanced in-context learning for action- effect prediction. Advances in Cognitive Systems. ACS- 2022, 2022.
  10. [Li et al., 2024] Fangjun Li, David C Hogg, and Anthony G Cohn. Advancing spatial reasoning in large language mod- els: An in-depth evaluation and enhancement using the stepgame benchmark. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 38, pages 18500- 18507, 2024.
  11. [Ligozat, 1998] Gerard Ligozat. Reasoning about cardinal directions. Journal of Visual Languages & Computing, 9(1):23-44, 1998.
  12. [Mirzaee and Kordjamshidi, 2022] Roshanak Mirzaee and Parisa Kordjamshidi. Transfer learning with synthetic cor- pora for spatial role labeling and reasoning. In Proceed- ings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 6148-6165. Association for Computational Linguistics, December 2022.
  13. [Mirzaee et al., 2021] Roshanak Mirzaee, Hossein Rajaby Faghihi, Qiang Ning, and Parisa Kordjamshidi. SpartQA: a textual question answering benchmark for spatial reason- ing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4582- 4598, 2021.
  14. [OpenAI, 2023] OpenAI. GPT-4 technical report. ArXiv, abs/2303.08774, 2023.
  15. [Randell et al., 1992] David A Randell, Zhan Cui, and An- thony G Cohn. A spatial logic based on regions and con- nection. KR, 92:165-176, 1992.
  16. [Shi et al., 2022] Zhengxiang Shi, Qiang Zhang, and Aldo Lipani. Stepgame: A new benchmark for robust multi- hop spatial reasoning in texts. In Proceedings of the AAAI conference on Artificial Intelligence, volume 36, pages 11321-11329, 2022.
  17. [Touvron et al., 2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, et al. Llama 2: Open foundation and fine-tuned chat models, 2023.
  18. [Weston et al., 2016] Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart Van Merriënboer, Ar- mand Joulin, and Tomas Mikolov. Towards AI-complete question answering: A set of prerequisite toy tasks. In 4th International Conference on Learning Representations, ICLR, 2016.

References were extracted automatically from the PDF and may contain errors.

Welcome Future Conferences AI Journal Division IJCAI Proceedings About IJCAI IJCAI Awards