Towards Joint Intent Detection and Slot Filling via Higher-order Attention

1 Introduction

Spoken language understanding (SLU) that aims to understand user oral instructions typically includes two modules: Intent detection (ID) and Slot filling (SF). Typically, ID is regarded as a semantic utterance classification problem to predict users intent, and SF is usually treated as a sequence labeling problem to extract semantic concepts of the spoken utterance. Take a flight-related utterance as an example, "Show flights from Seattle to San Diego", illustrated in Table 1. There are different slot labels for each word in the utterance, and a specific intent for the whole utterance.

Numerous works appealed to improve sentence and slot level semantics via mutual enhancement between ID and SF for their close correlation [Hakkani-T¨ur et al., 2016; Niu et al., 2019; Zhu et al., 2020; Qin et al., 2021]. In addition, the Multi-Head Attention (MHA) [Vaswani et al., 2017] was introduced and leveraged into the model [Li et al., 2018; Zhang et al., 2019; Qin et al., 2021] to provide the precise focus by linearly fusing the query and key via element-wise sum, and assign a weight to this value to arrive at a weighted sum representing the enhanced representations.

∗Contact Author

Despite their success, the attention based approaches suffer a limitation, i.e., they mainly exploit the first-order interactions between ID and SF features. We propose to distill second-order feature interactions via bilinear pooling [Kim et al., 2018], which calculates the outer product between two feature vectors, to trigger higher-order feature interactions. This technique triggers second-order feature interactions by establishing pairwise interactions between all queries and keys, and resulting in more expressive representations, which has been successfully applied in the field of Computer Vision [Lin et al., 2015; Fang et al., 2019; Yu et al., 2020; Pan et al., 2020; Kumari and Ekbal, 2021].

This paper introduces a BiLinear attention block illustrated in Figure 2 to trigger the second-order interactions between ID and SF tasks. Note that the significant difference from [Chen et al., 2022] is that we additionally leverage channel-wise bilinear attention simultaneously except for the conventional contextual attention computation. By combining lots of such blocks and equipping them with Exponential Linear Unit (ELU) [Barron, 2017], the model can build even higher-order feature interactions. Furthermore, we proposed the Dynamic Feature Fusion to explicitly fuse intent and slot features, instead of the straightforward concatenation of them. Finally, we present the Higher-order Attention Network (HAN) to leverage higher-order interactions for the SLU task, which is shown in Figure 1(b).

Extensive experiments on two benchmark datasets SNIPS [Coucke et al., 2018] and ATIS [Hemphill et al., 1990] proves the advancement of our approach. To further evaluate the proposed model, we conduct experiments on a recently released SLU dataset, i.e., SLURP [Bastianelli et al., 2020], which is substantially larger and more diverse than existing SLU datasets.

Our contributions in this work are three-fold:

• We propose a novel Higher-order Attention Network

(HAN) for SLU tasks, it achieves state-of-the-art results on two benchmark datasets in ID and SF tasks.

• HAN includes two essential modules: Higher-order Attention Layer (HAL) and Dynamic Feature Fusion Layer (DFFL). These two modules can be easily integrated into existing SLU models to boost their performance.
• We further give a deep exploration of the proposed methods from different views, including triggering higherorder interactions, attention visualization, robustness, and generalization analysis.

2 Conventional Approach

We first introduce the conventional approach illustrated in Figure 1(a), which consists of 4 modules: Self-attentive Embedder, Co-Interactive Attention Layer (CAL), Feature Fusion Layer (FFL), and SLU Decoder.

2.1 Self-attentive Embedder

Self-attentive Embedder targets to acquire utterance embeddings [Qin et al., 2019]. Given an input utterance, it uses a shared BiLSTM layer to acquire H ∈ Rn×d . Then the label attention [Cui and Zhang, 2019] is introduced to enrich the utterance embeddings with the intent and slot labeling information respectively:

whereWI ∈ RNI×d andWS ∈ RNS×d denote the label embedding matrix for ID and SF, NI and NS denotes the number of labels respectively. With Eq.(1), the acquired HI ∈ Rn×d and HS ∈ Rn×d could capture the intent and slot labeling information.

2.2 Co-Interactive Attention Layer (CAL)

The Co-Interactive Attention module is introduced to calculate the cross attention between intent feature vector and slot feature vector. Here we first provide a brief review of the attention module. Given the query q, we can obtain the attention distribution α over a set of keys K = {ki}ni=1 :

where ai represents the ith value in a, andW0,W1,W2 are weight matrices. The attention module outputs the enhanced feature ˆv via leveraging all values with contextual attention

HI and HS in Eq.(1) are fed into the CAL to be enriched with both the intent and slot features in a mutual way. As [Vaswani et al., 2017], we first map both HI and HS to two Query-Key-Value sets (QI,KI,VI) and (QS,KS,VS) through different weight matrices, respectively. Then QI, KS, and VS are used as queries, keys, and values, leading to the enhanced values:

where LN represents the layer normalization [Ba et al., 2016]. Similarly, we take QS, KI, and VI to obtain HS. We finally obtain the enhanced intent features HNI ∈ Rn×d and slot features HNS ∈ Rn×d after repeating N times.

2.3 Feature Fusion Layer (FFL)

In this subsection, we extend the Feed-Forward Network layer to explicitly fuse intent and slot information. we concatenate HNI and HNS to fuse the intent and slot features:

where [·, ·] indicates concatenation. Next, we adopt the shared FFN to acquire the updated ˆ HIN, ˆ HSN ∈ Rn×d :

2.4 SLUDecoder

SLU Decoder includes an Intent Predictor for ID and a CRF Layer for SF. Specially, we follow [Kim, 2014] to employ the maxpooling on ˆ HIN to obtain c, which is used to predict the intent label: oI ∼ ˆyI = softmax (WIc + bI ).

Besides, we apply a CRF layer [Niu et al., 2019] to model the labels relationships:

where OS =WS ˆ HSN +bS, ˆ y indicates the predicted label sequence, and f(yi-1, yi,OS) computes the transition score from yi-1 to yi.

3 The Proposed Approach

As shown in Figure 1(b), based on the conventional approach, we replace CAL and FFL with Higher-order Attention Layer (HAL) and Dynamic Feature Fusion Layer (DFFL), respectively, and thus obtain the proposed Higher-order Attention Network, i.e., HAN. Before introducing HAL and DFFL, we first describe the proposed BiLinear attention block illustrated in Figure 2.

3.1 BiLinear Attention Block

The BiLinear attention block uses Eq.(8) to replace Eq.(3). Given Q = {qi}ni=1 , K = {ki}ni=1 , and V = {vi}ni=1 (Their shapes are all Rn×d ), the BiLinear attention is formulated in Eq.(8):

Take a query q from Q, together with the ith ki from K and vi from V. Using the bilinear pooling calculation, the bilinear query-key and query-value joint representations Rki ,Rvi ∈ Rd are introduced to trigger the second-order feature interactions:

where Wk q , Wk , Wv q ,Wv ∈ Rd×d are all learnable weight matrices.

After that, we first obtain the contextual bilinear attention weights by transforming Rki into the corresponding attention weight through two fully-connected layers and a softmax function:

where WkR ∈ Rd×d , Wr ∈ R1×d , and Rk′ i is the converted bilinear query-key representation, ri is the ith element in r. Element γi in γ represents the ith key-value pair's contextual attention weight.

Secondly, we conduct a squeeze excitation calculation using all bilinear query-key representations {Rk′ i }ni=1 for channel-wise attention weight calculation, such operation aggregates {Rk′ i }ni=1 via average pooling, generating a channel descriptor: C = 1n n∑ Rk′ i . Then we leverage the sigmoid

i=1

activation on it to obtain channel-wise attention weight:

whereWc ∈ Rd×d . σ is the sigmoid activation.

In the final step, both contextual and channel-wise attention are simultaneously exploited to output the value ˆ vi:

We enumerate qi in Q, and get a set of values ˆV = { ˆ vi}ni=1 mentioned in Eq.(8).

Expansion via ELU. To further improve the second-order feature interactions in Eq.(9), we introduce higher-order feature interactions by equipping the block with Exponential Linear Unit [Barron, 2017], which is formulated as follows:

where the parameter α is set to 1 by default. We apply ELU to two dot product operations in Eq.(9) and obtain the updated Eq.(14):

The higher-order interactions brought by ELU can be proved via Taylor expansion. Technically, given two feature vectors A and B, their exponential bilinear pooling is formulated as:

where D is the feature vector's dimension, WjA/WjB is the j th row vector in weight matrices WA/WB.

3.2 Higher-order Attention Layer (HAL)

The HAL consists of N identical sub-layers (N = 2), each sub-layer includes a BiLinear attention block and a layer normalization module. Similar to CAL described previously, we first transformHI and HS acquired from Eq.(1) to two Query-Key-Value sets (QI,KI,VI) and (QS,KS,VS). Then the QI, KS, and VS are leveraged as queries, keys, and values, respectively, leading to the enhanced HI:

and we take QS, KI, and VI to obtain HS.

After repeating N times, we can acquire the enhanced intent and slot features, i.e., HNI , HNS ∈ Rn×d , which are endowed with the higher-order feature interactions in between.

3.3 Dynamic Feature Fusion Layer (DFFL)

Inspired by [Cornia et al., 2020] which measures the relevance between the output of the cross attention and the input query in the Meshed Decoder, we propose to fuse intent and slot features HNI and HNS dynamically. First, we compute two weight matrices αI and αS which refers to the relevance between the output and the input query of the last sub-layer in the HAL, and then obtain the fused features HIS, which can be defined as follows:

where WI,WS ∈ R2d×d , bI and bS are learnable bias parameters, QNI and QNS are acquired fromHNI and HNS using different linear projections. Following Eq.(6), we obtain the updated intent and slot features: ˆ HIN, ˆ HSN ∈ Rn×d .

3.4 Overall Loss Function

We adopt a joint learning scheme to optimize ID and SF tasks simultaneously as [Goo et al., 2018]. Specially, the total loss function is formulated as:

where ˆyI,j and ˆyS,j i is the gold intent label and gold slot label separately; M is the number of training data and Nj is the number of tokens in the j th data.

4 Experiments

4.1 Datasets and Setting

We conduct major experiments on SNIPS and ATIS. SNIPS has 13,084 utterances for training, 700 for validation, and 700 for testing. ATIS has 4,478 utterances for training, 500 for validation, and 893 for testing. Both datasets follow the partition as in [Goo et al., 2018]. We also conduct additional experiments on the newly released dataset SLURP 1 .

We use Adam as the optimizer. The size of hidden vector d is set to 128, batch size is 32, and learning rate is 1e-3. We evaluate the SLU performance about SF using F1 score, ID using accuracy, and sentence-level semantic frame parsing using overall accuracy. We map each word to a vector using the pretrained 300d GloVe embeddings [Pennington et al., 2014]. All experiments in this paper were conducted on a single NVIDIA GeForce GTX 2080 Ti GPU, and the model was implemented in PyTorch.

4.2 Comparison with BaselineWorks

RelatedWorks. We list threemostly related baselineworks for comparison, i.e., Stack-Propagation, Graph-LSTM, and Co-Interactive. Stack-Propagation first learns the ID task and uses the features acquired from hidden layers of ID to enhance the SF task; Graph-LSTM introduces a graph LSTM encoder to derive the token level embedding and the sentencelevel embedding which are used in SF and ID respectively; Co-Interactive introduces attention modules to mutually enrich the representations of utterances in SF and ID, and achieves the state-of-the-art result. They all have the conventional first-order attention involved.

As shown in Table 2, we can see that HAN outperforms the SOTA result. On both SNIPS and ATIS, HAN achieves an absolute 0.32% and 0.84% on accuracy increase on ID task, an absolute 1.31% and 0.56% on F1 score increase on SF task, an absolute 1.5% and 1.27% increase on the overall accuracy.

Furthermore, we also conduct experiments which uses BERT [Devlin et al., 2019] to update the BiLSTM in Selfattentive Embedder module. We can see that HAN can achieve improved accuracy on both datasets with BERT. Moreover, the performance of conventional network w/ BERT is still lower than HAN w/ BERT and HAN w/o BERT, which demonstrates the advantages of the proposed HAN.

4.3 Ablation Study

We also evaluate the contribution of each proposed modules: 1) BiLinear attention block; 2) Dynamic Feature Fusion module; 3) ELU module. We gradually integrate each module into the base setting (BiLSTM + SLU Decoder) and calculate the performance. As shown in Table 3, all these modules can boost the performance of HAN, and it reaches the best when equipping with ELU module (4.57% and 3.13% absolute overall accuracy gain on two datasets), which proves that exploiting the higher-order feature interactions between ID and SF can improve the performance of each other.

1 https://github.com/pswietojanski/slurp/tree/master/dataset/slurp

4.4 Optimal Number of Sub-layers in HAL/CAL

We evaluate the number of stacking BiLinear attention blocks in the HAL module, specific, the number N in HNI and HNS . We set the N between 1 to 5 while keeping other components fixed. As shown in Table 4, we can see that the performance of the HAN w/ ELU outperforms the one w/o ELU under all settings of N. In case N equals 2, the performance of HAN on both datasets reaches best.

Similarly, we evaluate the performance of conventional approach under different sub-layer number of CAL on ATIS dataset to find the optimal sub-layer number (the number N in Eq.(5)). As shown in Table 6, it reaches its best performance when N = 4, the overall accuracy is 87.23% which is still lower than 88.67% of HAN with N = 2. This is probably due to the ELU-equipped BiLinear attention blocks already mining the higher-order interactions between ID and SF tasks, HAN requires less stacking than that of the conventional approach.

4.5 Visualization of Attention

To show the learned higher-order attention, we visualize the first-order attention distribution in Eq.(2) and the higher-order attention distribution of the first sub-equation in Eq.(14) for comparison. In particular, we visualize the attention distribution of the intent feature to each token of slot features. As can be seen in Figure 3, where we use the utterance "What film is playing nearby" from the SNIPS dataset. The proposed higher-order attention focuses more on the keyword "film" and "nearby" that are most relevant to the intent "Search Screening Event", showing a better attention capture ability compared to the conventional first-order attention.

4.6 Robustness Analysis

To verify the robustness of higher-order attention towards hyper-parameters, we evaluate the performance of HAN and conventional approach under different learning rates. From Figure 4, we can see that higher-order model performs better than the first-order model under different learning rate. Besides, we find that under reasonable and task-specific range of the learning rate, i.e., 1e-4 to 1e-2, the higher-order based model performs consistently better than the first-order based model. When the learning rate is out of this range, i.e., bigger than 1e-2 or smaller than 1e-4, the performance of the firstorder based model reduces quickly or even drops to 0 when the learning rate is 0.1. While the performance of higherorder based model, also drops for out of range learning rate, can still maintain compared to the first-order based model, and thus shows its robustness to different learning rates.

4.7 Generalization Analysis

To explore the generalization of the proposed higher-order attention, we further incorporate HAL into two existing baselines (i.e., Stack-Propagation and Co-Interactive). For StackPropagation, HAL is inserted after its Self-Attentive Encoder. For Co-Interactive, we just replace its Co-Interactive Attention Layer with the HAL. Table 5 shows that baselines with HAL perform better than that with first-order attention in most cases. This verifies the generalization of the higherorder attention on other SLU models to some extent.

4.8 Evaluation on SLURP

We choose to use SNIPS and ATIS as the evaluation datasets, as they are widely used in previous works, therefore it is easy to compare with previous works. We also conduct an experiment on SLURP, which is a publicly available multi-domain dataset for SLU. As shown in Table 7, under the same setting, our HAN outperforms the conventional approach on three evaluation metrics.

5 Conclusion

This work proposes to trigger the second-order interactions between intent and slot features for the SLU task by introducing a novel BiLinear attention block, which simultaneously exploits both the contextual and channel-wise bilinear attention distributions. Likewise, higher-order feature interactions are further proposed via combining multiple BiLinear attention blocks and applying ELU on them. Furthermore, we introduce the HAN, which achieves new state-of-the-art results on both two benchmark SLU datasets. We also conduct a set of ablation studies to justify the importance of key components in our designed architecture. Lastly, the detailed discussion proves the effectiveness of the higher-order attention in terms of robustness and generalization.

Acknowledgements

Special acknowledgements are given to AOTO-PKUSZ Joint Research Center for its support. This paper is partially supported by Shenzhen Science & Technology Fundamental Research Programs (No: GXWD2020123116580700720200814115301001).