Towards Joint Intent Detection and Slot Filling via Higher-order Attention

Towards Joint Intent Detection and Slot Filling via Higher-order Attention

Dongsheng Chen, Zhiqi Huang, Xian Wu, Shen Ge, Yuexian Zou

Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence
Main Track. Pages 4072-4078. https://doi.org/10.24963/ijcai.2022/565

Recently, attention-based models for joint intent detection and slot filling have achieved state-of-the-art performance. However, we think the conventional attention can only capture the first-order feature interaction between two tasks and is insufficient. To address this issue, we propose a unified BiLinear attention block, which leverages bilinear pooling to synchronously explore both the contextual and channel-wise bilinear attention distributions to capture the second-order interactions between the input intent and slot features. Higher-order interactions are constructed by combining many such blocks and exploiting Exponential Linear activations. Furthermore, we present a Higher-order Attention Network (HAN) to jointly model them. The experimental results show that our approach outperforms the state-of-the-art results. We also conduct experiments on the new SLURP dataset, and give a discussion on HAN’s properties, i.e., robustness and generalization.
Keywords:
Natural Language Processing: Applications
Natural Language Processing: Dialogue and Interactive Systems
Natural Language Processing: Interpretability and Analysis of Models for NLP