Diversity Features Enhanced Prototypical Network for Few-shot Intent Detection

1 Introduction

Intent Detection (ID), also called Intent Classification, is a crucial task in human-machine dialogue systems. ID is to identify user's real intents behind given utterances and then extracted intents are involved in downstream tasks. Recent advance in deep learning and availability of massive data have prompted a series of effective detection algorithms in ID domain.

However, the amount of conversation data is typically limited in brand-new dialogue systems. Therefore, the system should be able to learn the ID model from limited samples. In this Few-shot Intent Detection (FSID) task, datadriven methods could easily cause serious overfitting problems. Researchers propose few-shot learning (FSL) algorithms [Vinyals et al., 2016; Sung et al., 2018] to overcome the problem of lacking training samples. The driving force is to use the knowledge outside the target domain and few labeled data to construct a model with a good discrimination ability of recognizing novel classes. Generally, few-shot learning algorithms use the meta-learning procedure for training and testing [Finn et al., 2017]. To simulate the situation of insufficient data, meta-learning procedure constructs a larger number of different meta-tasks in the training stage. A metatask is constructed by sampling a small training set (support set) and test set (query set) from rich data domains. In each meta task, the objective is to correctly infer the class of query set samples by using support set samples. In the testing stage, the model can immediately identify the novel category according to limited samples instead of training a new model.

∗Corresponding Author

While the dialogue system keeps operating, new intents are continuously emerging, therefore the IDmodel should be able to distinguish the existing intents and newly identified intents appropriately. Existing few-shot learning approaches fall short in adapting to this Generalized Few-shot Intent Detection (GFSID) task [Xia et al., 2020] where both seen and unseen classes are present. A main challenge of both FSID and GFSID tasks is that limited training samples fail to cover the diversity of user expressions.

In this paper, motivated by the phenomenon of word analogies in word2vec [Mikolov et al., 2013], we propose a novel method named Diversity Features Enhanced Prototypical Network (DFEPN) to tackle the above challenge. In word2vec, a surprising property of word vectors is that word analogies can often be solved with vector arithmetic, e.g., "King" = "Queen" + "man" - "woman". Among them, "King" and "Queen" belong to same category while "man" and "woman" belong to another category. Similar to word vectors, the difference between two sentence vectors from the same category can be regarded as a diversity feature of user expression. We treat an intent category containing abundant labeled data as a source domain and a novel intent category as a target domain. Therefore, we propose to transfer diversity features from source domains to target domains, and Figure 1 shows a simple example. Firstly, we randomly select sample pairs from a source domain and calculate diversity features by the internal differences of sample pairs. After that, diversity features are added to a novel intent vector to obtain generated vectors. Fusing generated vectors with the original novel intent vector and getting a fused vector. In this way, we enhance features for novel intents in the high dimensional space. Besides, our method uses meta-learning procedure and construct abundant meta-tasks for training. To learn the interactive knowledge between support and query set vectors, we reuse the process of transferring diversity features to enhance query set vectors at the same time.

The primary contributions of this paper are as follows:

• We propose a novel method, DFEPN, to solve the FSID and GFSID problem. DFEPN uses diversity features of source domains to enhance the diversity of samples in target domains.
• We improve the performance of the method by enhancing support and query set vectors at the same time. It extracts the interactive knowledge between support and query set vectors.
• We conduct experiments on two benchmark intent detection datasets to test the performance of our method in both FSID and GFSID scenarios. Experimental results show that our method is superior to previous state-ofthe-art methods.

2 RelatedWork

2.1 Few-shot Learning

Few-shot learning aims to learn a model, which is trained on known category samples and can classify unknown category samples well. FSL methods can be divided into three types: optimization-based methods, metric-based methods and generation-based methods. Optimization-based methods aim to learn general initialization model parameters to ensure that parameters can be optimized well in few steps [Finn et al., 2017; Rusu et al., 2018]. Metric-based methods, including Matching Network (MN) [Vinyals et al., 2016], Prototypical Network (PN) [Snell et al., 2017] and Relation Network (RN) [Sung et al., 2018], aim to learn an appropriate classifier by modeling the distance distribution of samples. Generation-based methods aim to synthesize new samples for target classes based on few samples [Chen et al., 2020; Sun et al., 2021].

2.2 Generalized Few-shot Learning

Generalized few-shot learning is an extension of FSL, requiring the model to correctly classify samples in a joint label space that consists of existing and emerging categories. GFSL is a new investigation domain and current works mainly focus on the computer vision. Li et al. [Li et al., 2019] learn global class representations in the joint label space utilizing both existing and emerging category training samples. Shi et al. [Shi et al., 2019] use graph-convolution to model inter-class relationships and learn representative prototypes for joint classes. In the field of intent detection, Xia et al. [Xia et al., 2020] explore this problem first and simulate the distribution of unlabeled samples by utilizing labeled samples. Nguyen et al. [Nguyen et al., 2020] consider that coarser-grained semantics can provide additional knowledge and propose multiple semantic components via multi-head self-attention.

3 ProblemFormulation

Few-shot intent detection (FSID) and generalized few-shot intent detection (GFSID) are defined formally in this section.

The intent with a large number of labeled data is defined as seen intent and the intent with few labeled data as novel intent. We denote seen intent label space as Ys, novel intent label space as Yn, and Ys ∩ Yn = ∅. We define the seen intent set, Ds = {(x1, y1), (x2, y2), · · · , (xNs , yNs )}, yi ∈ Ys where x denotes a user utterance and Ns denotes the number of labeled samples. Similarly, the novel intent set Dn = {(x1, y1), (x2, y2), · · · , (xNn , yNn )}, yi ∈ Yn.

As summarized in Equation (1), the objective of FSID is to determine the most likely category of a new unlabeled sample x whose label belongs to Yn.

Unlike FSID, GFSID aims at classifying an unlabeled sample not only as the novel intents but also as the seen intents. We denote a joint intent set as Yj , Yj = Ys ∪Yn, a joint intent set as Dj = {(x1, y1), (x2, y2), · · · , (xNj , yNj )}, yi ∈ Yj .

As summarized in Equation (2), given an unlabeled sample x whose label belongs to Yj , the objective function of GFSID is to infer the most likely category of x.

4 Method

4.1 Overall Architecture

An overview of the Diversity Features Enhanced Prototypical Network (DFEPN) is shown in Figure 2. Our model framework includes four components: Semantic Encoder, Diversity Feature Generator, Feature Fusion and Similarity Classifier.

Our method uses a meta-learning framework and construct multiple training episode, simulating the situation of insufficient training samples. Generally, a training episode consists of support set and query set. Our model needs an additional auxiliary set to generate the diversity features of samples. We randomly select M intents (called auxiliary intents) from the training set and all examples belonging to these intents form the auxiliary training set Daux. All samples of the remaining intents (called base intents) form the base training set Dbase. For a C-way K-shot problem, we randomly select C intents from the base training set and then choose K samples within each selected intent. These samples constitute the support set, S = ⋃Cc=1 {xsc,k , ys c,k }Kk=1 , while a subset of the remaining samples act as the query set, Q = {xql , yq l }Ll=1 . An auxiliary set is formed by randomly selecting N sample pairs from the auxiliary training set, A = ⋃Nn=1 {xan,1 , xan,2 }. A set of sample pairs consists of two different samples pertaining to the same intents.

4.2 Semantic Encoder

Semantic Encoder (SE) aims to learn the semantic information of sentences and encode it into the high-dimensional space. Recently, the method of pre-training on a large corpus and then fine-tuning on target tasks has achieved great success. In FSL tasks, although it is unable to provide enough training samples for fine-tuning, the task-agnostic knowledge learned by pre-training is still helpful for semantic extraction [Deng et al., 2020]. Different from the previous work [Nguyen et al., 2020], we hope our model can benefit from the advantage of pre-trained language models and use BERT [Devlin et al., 2019] as the Semantic Encoder.

Instead of using the pre-trained model directly, we further pre-train BERT on the Masked Language Model (MLM) Task with the training dataset, which will greatly improve the quality of embedded representation [Dopierre et al., 2021]. We use the final representation of the [CLS] token as the semantic representation of the whole sentence x, e = SE(x), e ∈ Rd .

4.3 Diversity Feature Generator

Diversity Feature Generator (DFG) aims to transfer diversity features from the auxiliary set to target intent vectors. Samples from the auxiliary set A, the support set S and the query set Q are encoded as vectors ea , es and eq . The structure of the DFG is shown in Figure 3, where ϕ1 and ϕ2 are two linear layers.

First, sample pair vectors ean,1 , ean,2 and a support vector esc,k are mapped into a new hidden space by ϕ1, ˆ e = ϕ1(e), ˆ e ∈ Rm. The difference between ˆ ean,1 and ˆ ean,2 can be regarded as the diversity feature within the same category. Then, the enhanced sample vector is generated by adding this diversity feature to the target sample vector ˆ esc,k , ˆ ean,1 - ˆ ean,2 + ˆ esc,k . Finally, after mapping it into the original vector space by ϕ2, we can get the generated vector esg c,k . More specifically:

The auxiliary set A includes N sample pairs. Therefore, for each support vector, N diversity features are generated by the DFG.

4.4 Feature Fusion

The Feature Fusion (FF) module fuses generated vectors with the original support vector and then gets the prototype vector of each class. The algorithm of the Feature Fusion process is presented in Algorithm 1.

For each generated vectors esg c,k,n , n ∈ [1, 2, · · · ,N], we employ a residual connection [He et al., 2016] after the DFG,

Algorithm 1 Feature Fusion

Input: sample vector esc,k in support set S, generated features esg c,k,n

Parameter: linear layer weights W and bias b Output: prototype vector zc

• 1: for all samples k = 1, 2, · · · ,K in class c do
• 2: for n = 1, 2, · · · ,N do
• 3: ˆ esg c,k,n = LayerNorm(esg c,k,n + esc,k )
• 4: end for
• 5: ˆ esg c,k = concat( 1N ∑Nn=1 ˆ esg c,k,n , esc,k )
• 6: zc,k = tanh (Wˆ esg c,k + b)
• 7: end for
• 8: zc = 1K ∑Kk=1 zc,k
• 9: return zc

followed by layer normalization [Ba et al., 2016], and the output is

Next, the average of the N outputs and the original support vector are spliced together, the fused vector is

And then it is fed to a linear layer, whose weight is W and bias is b,

Finally, averaging the K enhanced sample vectors to get the prototype representation of the corresponding category,

4.5 Similarity Classifier

In order to learn the interactive knowledge between support and query set vectors, we also enhance query vectors through the above modules. A query vector eql is concerted to the enhanced query vector ˆ eql ,

The article [Snell et al., 2017] notes that Euclidean distance is more suitable for classifier of prototypical networks comparing with cosine distance. Therefore, we decide to use Euclidean distance to measure the similarity between ˆ eql and zc, getting the classification score,

For each training episode, given auxiliary set A, support set S and query set Q = {xql , yq l }Ll=1 , the training objective is to minimize the cross-entropy loss on the query set. The loss function is as follows:

where ˆ yq = softmax(-sql,c ) is the predicted probabilities of C classes.

5 Experiments

In this section, we conduct experiments on two benchmark intent detection datasets to test the performance of our method in both FSID and GFSID scenarios. we select various outstanding FSL algorithms as baselines, including MN [Vinyals et al., 2016], PN [Snell et al., 2017], RN [Sung et al., 2018], HATT [Gao et al., 2019]), MLMAN [Ye and Ling, 2019], HAPN [Sun et al., 2019], SMAN [Nguyen et al., 2020] and MLADA [Han et al., 2021]. MLADA is one of the stateof-the-art few-shot text classification models. SMAN is the state-of-the-art model for the GFSID task.

5.1 Datasets

In this section, we describe the details of two benchmark intent detection datasets used in experiments: SNIPS Natural Language Understanding benchmark (SNIPS) [Coucke et al., 2018] and NLU-Evaluation Dataset (NLUE) [Liu et al., 2019].

Following [Nguyen et al., 2020], we divide these two datasets to evaluate the performance of FSID and GFSID. As shown in Table 1, seen intents are divided into auxiliary intents and base intents randomly in advance. GFSID testing samples (joint samples) include all novel intent samples and 20 percent of seen intent samples. We use auxiliary and base samples to train our model and test it on novel samples (FSID) and joint samples (GFSID).

SNIPS: According to the data partition method in [Xia et al., 2018], the number of novel intents is set to 2. After selecting novel intents, we select one intent from the remaining intents as the auxiliary intent while the last four intents are base intents.

NLUE: Following [Liu et al., 2019], we choose the same 16 intents as novel intents, 10 intents as auxiliary intents and the last 38 intents as base intents.

5.2 Implementation Details

Following [Nguyen et al., 2020], we randomly construct 1000 training episodes with K= 1, 5 and C=2 (FSID) or 4 (GFSID) on SNIPS dataset. For NLUE dataset, we also structure 1000 training episodes with the same K and C=5 (FSID) or 10 (GFSID). In the training stage, we extract 20 samples from each class to form the query set on both datasets. Our model is trained with learning rate equals 2e-5 and the hidden dimension d is 768, m is 1024. The number of auxiliary sample pairs N is 25. We use BERT models from the Hugging Face [Wolf et al., 2020] team.

We evaluate models on both FSID and GFSID tasks. The S-J accuracy represents the GFSID testing results while the S-N accuracy denotes FSID results. H-acc is the harmonic mean of S-J accuracy and S-N accuracy, used to measure the comprehensive ability of models in both FSID and GFSID scenarios. Besides, these models are evaluated in two different testing procedures: Episodic Evaluation (eps) and Nonepisodic Evaluation (noneps).

Episodic Evaluation: Traditional FSL methods follow a principle: test and train conditions must match [Vinyals et al., 2016]. We construct 1000 episodes with 5 samples per class to form the query set in the testing stage.

Non-episodic Evaluation: [Nguyen et al., 2020] points out the above-mentioned episodic evaluation method lacks practicability in realistic applications. They recommend testing unlabeled samples only once and these samples are selected from the whole dataset in advance.

On SNIPS dataset, we use 5 different random seeds to carry out five groups of experiments, and the final average accuracy is shown in Table 2. On NLUE dataset, we use 10-fold crossvalidation to test our model and the final average accuracy is shown in Table 3.

5.3 Comparisons with State-of-the-arts

Table 2 and 3 indicate an overall improvement compare to previous baselines.

Our model performs worse than SMAN in only one situation in which testing 10-way 1-shot S-J on NLUE by episodic evaluation (87.35 < 89.54). However, when we changed the testing procedure to non-episodic evaluation, our model achieved the best S-J accuracy which is 6 percentage points higher than SMAN. This is mainly because the number of test samples per class in NLUE is much less than that in SNIPS (29.3 < 384), leading to a lot of repeated choices when constructing testing episodes. It magnifies the uncertainty of testing. We can see that all models have the same consistent trend of S-J accuracy under two different testing conditions on SNIPS when increasing the number of support set samples from 1 to 5. For example, the S-J accuracy of MN increases under both two testing standards (noneps: 73.50 to 77.31; eps: 82.67 to 84.60). However, only four models (PN, HAPN, MLADA and our model) maintain this consistency on NLUE and it proves the existence of the testing uncertainty mentioned above.

In most cases, the recognition accuracy of all models is decreased when evaluated on non-episodic evaluation instead of episodic evaluation because non-episodic evaluation is more challenging. However, S-N accuracies on SNIPS dataset which are evaluated by both two testing procedures are almost same. This is because the FSID task can be seemed as a binary classification problem in both testing conditions.

Our model gets more improvement in non-episodic testing than episodic testing. It shows that our model can be well adapted to the realistic environment. Meanwhile, the performance improvement of our model is more obvious in 1-shot situation. This is because the smaller number of support set samples, the more difficult it is to reflect the diversity of samples. In this case, enhancing the diversity of samples can effectively improve the performance.

5.4 Ablation Study

In this section, we conduct a large number of ablation studies to observe the effect of each individual component in the model. We test the performance of our full model and its ablations on SNIPS dataset. H-acc results are shown in Table 4.

"w/o FP" means that we use the pre-trained BERT model directly instead of further pre-training BERT with SNIPS dataset. The performances in the 1-shot classification and 5-shot classification decrease by 20.45% and 5.6% on nonepisodic procedure. It proves that our model needs Semantic Encoder to have a good initial semantic parsing ability, otherwise the difference between two auxiliary vectors is too huge, causing a lot of noise.

"w/o DFG" means that we directly use Prototypical Network with further pre-training BERT. On non-episodic procedure, the performances are reduced by 5.9% and 2.88% in 1-shot and 5-shot scenarios. To further analyze the impact of the DFG module, we feed 210 test samples (30 samples per intent) into models and get these high-dimensional representations. Next, we use t-SNE [Van der Maaten and Hinton, 2008] to reduce the dimension of original vectors and enhanced vectors. As shown in Figure 4, enhanced vectors are much more diverse and the boundary between two classes is also clearer, comparing with original vectors as in Figure 5.

Figure 5: t-SNE visualization of the original vectors.

"w/o FF" uses a simple strategy which the enhanced feature is the average of the original vector and diversity features to replace the Feature Fusion module. The performances are decreased under all conditions, especially in the 1-shot situation. It proves that the module has strong correction ability, playing an important role in extreme circumstances.

"w/o QE" means that query vectors are not enhanced by DFG and FF. We connect the query vector with its copy vector instead. The results illustrate that enhanced query vectors can better adapt to the realistic scene.

6 Conclusion

In this paper, we propose an effective Diversity Features Enhanced Prototypical Network for both few-shot and generalized few-shot intent detection. DFEPN enhances the diversity of samples in target domains by transferring diversity features from source domains. Experimental results show that DFEPN is superior to previous state-of-the-art methods on the SNIPS and NLUE datasets for FSID and GFSID tasks. In the future, we will explore the effectiveness of DFEPN in other tasks.

Acknowledgments

This research is supported by the Natural Science Foundation for Distinguished Young Scholars of Xinjiang Uygur Autonomous Region (2022D01E04), the Xinjiang Key Laboratory Fund under Grant No.2020D04050, the West Light Foundation of The Chinese Academy of Sciences (Grant No.2018-XBQNXZ-A-003) and the Xinjiang Science and Technology Major Project (No.2020A02001-1).