ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning

Zhao, Liang; Yu, En; Ge, Zheng; Yang, Jinrong; Wei, Haoran; Zhou, Hongyu; Sun, Jianjian; Peng, Yuang; Dong, Runpei; Han, Chunrui; Zhang, Xiangyu

doi:10.24963/ijcai.2024/193

ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning

Liang Zhao, En Yu, Zheng Ge, Jinrong Yang, Haoran Wei, Hongyu Zhou, Jianjian Sun, Yuang Peng, Runpei Dong, Chunrui Han, Xiangyu Zhang

Download BibTeX Cite

Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence. Main Track. Pages 1743-1752.

https://doi.org/10.24963/ijcai.2024/193

Keywords

Natural Language Processing: NLP: Question answering
Natural Language Processing: NLP: Dialogue and interactive systems
Computer Vision: CV: Embodied vision: Active agents, simulation
Computer Vision: CV: Multimodal learning

Abstract

Human-AI interactivity is a critical aspect that reflects the usability of Multimodal Large Language Models (MLLMs). However, existing end-to-end MLLMs only allow users to interact with them through language instructions, leading to the limitation of the interactive accuracy and efficiency. In this study, we present precise referring instructions that utilize diverse reference representations such as points and boxes as referring prompts to refer to the special region. This enables MLLMs to focus on the region of interest and achieve finer-grained interaction. Based on precise referring instruction, we propose ChatSpot, a unified end-to-end MLLM that supports diverse forms of interactivity including mouse clicks, drag-and-drop, and drawing boxes, which provides a more flexible and seamless interactive experience. We also construct a multi-grained vision-language instruction-following dataset based on existing datasets and GPT-4 generating. Furthermore, we design a series of evaluation tasks to assess the effectiveness of region recognition and interaction. Experimental results showcase ChatSpot's promising performance. Project page: https://github.com/Ahnsun/ChatSpot.

Liang Zhao, En Yu, Zheng Ge, Jinrong Yang, Haoran Wei, Hongyu Zhou, Jianjian Sun, Yuang Peng, Runpei Dong, Chunrui Han, and Xiangyu Zhang. 2024. ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24. IJCAI, 1743–1752. https://doi.org/10.24963/ijcai.2024/193

@inproceedings{ijcai2024p193,
  title     = {ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning},
  author    = {Zhao, Liang and Yu, En and Ge, Zheng and Yang, Jinrong and Wei, Haoran and Zhou, Hongyu and Sun, Jianjian and Peng, Yuang and Dong, Runpei and Han, Chunrui and Zhang, Xiangyu},
  booktitle = {Proceedings of the Thirty-Third International Joint Conference on
               Artificial Intelligence, {IJCAI-24}},
  publisher = {International Joint Conferences on Artificial Intelligence Organization},
  editor    = {Kate Larson},
  pages     = {1743--1752},
  year      = {2024},
  month     = {8},
  note      = {Main Track},
  doi       = {10.24963/ijcai.2024/193},
  url       = {https://doi.org/10.24963/ijcai.2024/193},
}

TY  - CPAPER
TI  - ChatSpot: Bootstrapping Multimodal LLMs via Precise Referring Instruction Tuning
AU  - Zhao, Liang
AU  - Yu, En
AU  - Ge, Zheng
AU  - Yang, Jinrong
AU  - Wei, Haoran
AU  - Zhou, Hongyu
AU  - Sun, Jianjian
AU  - Peng, Yuang
AU  - Dong, Runpei
AU  - Han, Chunrui
AU  - Zhang, Xiangyu
T2  - Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24
PB  - International Joint Conferences on Artificial Intelligence Organization
PY  - 2024
SP  - 1743
EP  - 1752
DO  - 10.24963/ijcai.2024/193
UR  - https://doi.org/10.24963/ijcai.2024/193
ER  -