Get The Point of My Utterance! Learning Towards Effective Responses with Multi-Head Attention Mechanism

Get The Point of My Utterance! Learning Towards Effective Responses with Multi-Head Attention Mechanism

Chongyang Tao, Shen Gao, Mingyue Shang, Wei Wu, Dongyan Zhao, Rui Yan

Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence
Main track. Pages 4418-4424. https://doi.org/10.24963/ijcai.2018/614

Attention mechanism has become a popular and widely used component in sequence-to-sequence models. However, previous research on neural generative dialogue systems always generates universal responses, and the attention distribution learned by the model always attends to the same semantic aspect. To solve this problem, in this paper, we propose a novel Multi-Head Attention Mechanism (MHAM) for generative dialog systems, which aims at capturing multiple semantic aspects from the user utterance. Further, a regularizer is formulated to force different attention heads to concentrate on certain aspects. The proposed mechanism leads to more informative, diverse, and relevant response generated. Experimental results show that our proposed model outperforms several strong baselines.
Keywords:
Natural Language Processing: Dialogue
Natural Language Processing: Natural Language Generation