A Question Type Driven Framework to Diversify Visual Question Generation

A Question Type Driven Framework to Diversify Visual Question Generation

Zhihao Fan, Zhongyu Wei, Piji Li, Yanyan Lan, Xuanjing Huang

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence
Main track. Pages 4048-4054. https://doi.org/10.24963/ijcai.2018/563

Visual question generation aims at asking questions about an image automatically. Existing research works on this topic usually generate a single question for each given image without considering the issue of diversity. In this paper, we propose a question type driven framework to produce multiple questions for a given image with different focuses. In our framework, each question is constructed following the guidance of a sampled question type in a sequence-to-sequence fashion. To diversify the generated questions, a novel conditional variational auto-encoder is introduced to generate multiple questions with a specific question type. Moreover, we design a strategy to conduct the question type distribution learning for each image to select the final questions. Experimental results on three benchmark datasets show that our framework outperforms the state-of-the-art approaches in terms of both relevance and diversity.
Keywords:
Machine Learning: Neural Networks
Natural Language Processing: Natural Language Generation
Machine Learning: Deep Learning