A Multi-task Learning Approach for Image Captioning

A Multi-task Learning Approach for Image Captioning

Wei Zhao, Benyou Wang, Jianbo Ye, Min Yang, Zhou Zhao, Ruotian Luo, Yu Qiao

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence
Main track. Pages 1205-1211. https://doi.org/10.24963/ijcai.2018/168

In this paper, we propose a Multi-task Learning Approach for Image Captioning (MLAIC ), motivated by the fact that humans have no difficulty performing such task because they possess capabilities of multiple domains. Specifically, MLAIC consists of three key components: (i) A multi-object classification model that learns rich category-aware image representations using a CNN image encoder; (ii) A syntax generation model that learns better syntax-aware LSTM based decoder; (iii) An image captioning model that generates image descriptions in text, sharing its CNN encoder and LSTM decoder with the object classification task and the syntax generation task, respectively. In particular, the image captioning model can benefit from the additional object categorization and syntax knowledge. To verify the effectiveness of our approach, we conduct extensive experiments on MS-COCO dataset. The experimental results demonstrate that our model achieves impressive results compared to other strong competitors.
Keywords:
Machine Learning: Transfer, Adaptation, Multi-task Learning
Humans and AI: Cognitive Modeling
Computer Vision: Language and Vision