AmazonQA: A Review-Based Question Answering Task

AmazonQA: A Review-Based Question Answering Task

Mansi Gupta, Nitish Kulkarni, Raghuveer Chanda, Anirudha Rayasam, Zachary C. Lipton

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence
Main track. Pages 4996-5002. https://doi.org/10.24963/ijcai.2019/694

Every day, thousands of customers post questions on Amazon product pages. After some time, if they are fortunate, a knowledgeable customer might answer their question. Observing that many questions can be answered based upon the available product reviews, we propose the task of review-based QA. Given a corpus of reviews and a question, the QA system synthesizes an answer. To this end, we introduce a new dataset and propose a method that combines informational retrieval techniques for selecting relevant reviews (given a question) and "reading comprehension" modelsĀ for synthesizing an answer (given a question and review). Our dataset consists of 923k questions, 3.6M answers and 14M reviews across 156k products. Building on the well-known Amazon dataset, we additionally collect annotations marking each question as either answerable or unanswerable based on the available reviews. A deployed system couldĀ first classify a question as answerable before attempting to generate a provisional answer. Notably, unlike many popular QA datasets, here the questions, passages, and answers are extracted from real human interactions. We evaluate a number of models for answer generation and propose strong baselines, demonstrating the challenging nature of this new task.
Keywords:
Natural Language Processing: Question Answering
Natural Language Processing: Resources and Evaluation
Natural Language Processing: Natural Language Processing
Machine Learning: Deep Learning