Decoupling Breaks Data Barriers: A Decoupled Pre-training Framework for Multi-intent Spoken Language Understanding

1 Introduction

Spoken Language Understanding (SLU) is an essential component of dialog systems, which can be used to extract the semantic parsing results of user query (e.g., intents and slots) [Tur and De Mori, 2011; Young et al., 2013; Qin et al., 2019]. Specifically, SLU consists of two typical sub-tasks: slot filling and intent detection. Take the query "add Despacito to POP playlist and play it" in the Figure 1 as an example, the former task is regarded as a sequence labeling task to output a sequence of slots (i.e., "O, B-music-name, O, B-playlist, O, O, O, O") and the latter task is considered as a classification task to predict intents (i.e., "AddToPlaylist, PlayMusic").

Recently, Gangadharaiah & Narayanaswamyet al. [2019] discover that over 50% of samples in the internal Amazon dataset exhibit multiple intents. Consequently, dominant SLU systems shift their eyes from single-intent settings to multi-intent scenarios [Qin et al., 2020; Wu et al., 2022; Song et al., 2022b]. To this end, Qin et al. [2020] introduce AGIF, a method that adaptively integrates information from multi-intent predictions into the slot filling process. Song et al. [2022b] incorporate statistical information regarding the co-occurrence frequencies of intents and slots to improve multi-intent SLU. Xing & Tsang et al. [2022] propose a heterogeneous semantics-label graphs framework to model the mutual guidance between the two tasks. Pham et al. [2023] introduce MISCA, a joint model with intent-slot and label attention mechanisms to capture intent-slot correlations, achieving promising performance.

However, despite its success, current dominant multi-intent SLU models still heavily rely on a large amount of annotated data for training, which poses a significant cost burden for the real-world scenario without sufficient data. In addition, annotating multi-intent data is extremely time-consuming and labor-intensive, because annotators are not only required to have professional knowledge in task-oriented dialogue systems but also to understand the complex multi-intent relationships within a user query. Therefore, in this work, we aim to investigate a research question "Can we leverage the readily accessible multi-intent-free data, such as NER and singleintent SLU data, to enhance multi-intent SLU?".

Motivated by this, in this paper, we introduce a novel decoupled pre-training framework (DPF) for multi-intent SLU, aiming to utilize abundant multi-intent-free data (i.e., NER and single-intent SLU data). Specifically, DPF decouples multi-intent SLU into two abilities: (1) the task-agnostic ability to identify the slot entity span and (2) the task-specific ability to predict domain-specific slot and intent labels simultaneously. Such decomposition allows us to design a two-stage pre-training framework to enhance the two abilities, respectively, which is shown in Figure 2. Concretely, to enhance the task-agnostic ability, DPF first proposes a task-agnostic ability pre-training stage. This stage contains a token-level entity span prediction pre-training task and sentence-level entity number prediction pre-training task, facilitating representation learning for token-level slot filling and sentence-level intent detection. Since the task-agnostic ability pre-training stage does not require any task-specific annotation data, this process can be achieved with abundant NER data (see Figure 2 (a)). To improve the taskspecific ability, DPF further proposes a task-specific ability pre-training stage, which mainly focuses on learning the mutual guidance between intent detection and slot filling by leveraging single-intent SLU data (see Figure 2 (b)). Finally, after completing the two-stage pre-training procedure, we achieve the multi-intent SLU by conducting the downstream task finetuning (see Figure 2(c)).

With the help of decoupled pre-training, it brings us at least two advantages: (1) By disentangling the multi-intent SLU into task-agnostic and task-specific ability learning, our model is able to sequentially solve multi-intent SLU step by step; (2) In contrast to previous approaches that heavily rely on multi-intent SLU training data, DPF can leverage a large amount of NER and single-intent SLU data, which are much easier to obtain.

Contributions of this work are summarized as:

1. To the best of our knowledge, this study is the first to investigate a decoupled pre-training framework (DPF) for multi-intent SLU;
2. DPF offers the advantage of utilizing a vast amount of multi-intent-free data to enhance multi-intent SLU, thus effectively alleviating the data scarcity problem;
3. Experimental results show that DPF achieves the superior performance and extensive analyses reveal its superiority in low-resources scenarios.

To facilitate the research, all codes used in this work are publicly available at https://github.com/LightChen233/DPF.

2 Methodology

This section provides the overall workflow of DPF, which consists of task-agnostic ability pretraining stage (TAAP) (§2.1), a task-specific ability pretraining stage (TSAP) (§2.2) and downstream task finetuning (DSTF) (§2.3). Specifically, Algorithm 1 shows the overall process of DPF. Lines 1-4 denote the two-stage pretraining procedures including taskagnostic and task-specific ability pretraining. Lines 5-7 denote the fine-tuning stage. We will describe each stage in the following sub-sections.

2.1 Stage 1: Task-Agnostic Ability Pretraining

Task-Agnostic Ability Pretraining (TAAP) is used to enhance the general task-agnostic ability of multi-intent SLU that does not require any task-oriented dialogue domain knowledge. Therefore, we can leverage abundant NER data for the taskagnostic ability pre-training. Since multi-intent SLU contains token-level slot filling and sentence-level intent detection sub-tasks, we introduce token-level entity span prediction pretraining task and sentence-level entity number prediction pretraining task to capture the token-level and sentence-level general knowledge, respectively.

Token-Level Entity Span Prediction

Token-level entity span prediction pretraining task can be used to capture token-level task-agnostic ability. Specifically, it is considered as a binary task of token-level entity span prediction to judge whether a token belongs to an entity span or not. As shown in Figure 2 (a), take the input sentence "John works at Google in California." as an example, the entities in this sentence are labeled as follows: "John [B-Person] works

Algorithm 1 Overall Workflow of DPF Input: Dataset D = {(task, x, y)i}|D| i=1 ; The number of trained epochs in each tasks etask max ; Initial model parameters Θ. Output: Trained Model Θ. // decoupled pretraining for TAAP and TSAP 1 for task in {TAAP, TSAP} do 2 for e in {1, ..., etask max } do // batch optimization 3 for B in {(task, x, y)j}|B| j=1 ∈ D do 4 Optimizing Θ using Ltask; // downstream task finetuning 5 for e in 1, ..., eDSTF max do 6 for B in {(DSTF, x, y)j}|B| j=1 ∈ D do 7 Optimizing Θ using LDSTF;

at Google [B-Company] in California [B-Location]", we use the sketch to replace the original entities annotation as "John [B-X] works at Google [B-X] in California [B-X]" to perform an entity binary prediction task.

Formally, given input sentence X = {[CLS], x1, x2, . . . , xn}, we first encode X to obtain hidden vector H = {h[CLS], h1, h2, . . . , hn}, where n is the sequence length:

where we use Debertav3 [He et al., 2023] as the Encoder in this work.

Furthermore, given the H, token-level entity span prediction pretraining task is used to output the entity sequence E = {e1 , e2 , . . . , en }, where e∗ ∈ {B-X, I-X,O}, which is denoted as:

where the WE and b are learnable parameters.

The objective of the token-level entity span prediction task is formulated as:

where ˆ ej refers to the gold entity label at j token.

Sentence-Level Entity Number Prediction

To enhance the sentence-level task-agnostic ability, we introduce a sentence-level entity number prediction pre-training task to predict the number of entities in an utterance, which achieves to improve the model's understanding ability of the whole sentence. The underlying intuition is that by accurately predicting the number of entities in a sentence, the model can successfully capture the entire sentence representation, which is beneficial for sentence-level intent detection.

Similarly, as shown in Figure 2 (b), given the input utterance "John works at Google in California." the number of entities is determined to be "3" (i.e., "John [B-X]", "Google [B-X]", and "California [B-X]"). Formally, the sentence representation h[CLS] is used to predict the number m of entities in an utterance, which is denoted as:

The training objective can be defined as follows:

where |m| is the maximum number of entities and ˆm denotes the golden entity number.

Joint Pretraining

We adopt a joint training method for token-level entity span prediction pretraining task and sentence-level entity number prediction pre-training task simultaneously, denoting as:

where α1 and α2 are hyper-parameters.

Pre-training Data Collection

The task-agnostic ability pretraining stage does not require the task-specific annotation about slot and intent, which allows us to leverage large amounts of NER data. Therefore, we collect various NER datasets, including BBN [Weischedel and Brunstein, 2005], CoNLL [Sang and De Meulder, 2003], FIGER [Ling and Weld, 2012], GUM [Zeldes, 2017], SLURP [Bastianelli et al., 2020], MITMovieCorpus [Liu et al., 2013], MITRestaurantCorpus [Liu et al., 2013], MultiCoNER [Fetahu et al., 2021], MultiNERD [Tedeschi and Navigli, 2022], OntoNotes 5.0 [Pradhan et al., 2013], Polyglot-NER [Al-Rfou et al., 2015], Ritter [Ritter et al., 2011], WikiANN [Pan et al., 2017], WikiNEuRal [Tedeschi et al., 2021], WNUT17 [Derczynski et al., 2017], XGLUE [Liang et al., 2020].

Overall, we collect 2.4 million training data for the taskagnostic ability pre-training procedure. The detailed data statistics are shown in Table 1.

2.2 Stage 2: Task-Specific Ability Pretraining

The core challenge of multi-intent SLU lies in effectively modeling interaction across intents and slots. Unfortunately, there is no sufficient annotated data to capture the interaction relationship in multi-intent SLU. Therefore, we introduce the task-specific ability pretraining (TSAP) stage to leverage relatively abundant single-intent SLU data to enhance the task interaction ability.

Specifically, we adopt an interaction module from the strong single-intent SLU model (DCA-Net) [Qin et al., 2021a] as our pretraining interaction module, which is used to learn task-specific interaction knowledge across slot filling and intent detection. Formally, given the hidden states H, TSAP considers the interaction between slot filling and intent detection to get the interaction representations I = {I[CLS], I1, I2, . . . , In}:

Furthermore, the interaction representations I are used to perform slot filling and single intent detection in TSAP, which are denoted as:

The training loss of the model is similar to Equation 6, which is the weighted sum of slot and intent loss. Finally, the pretrained interaction module can be strengthened by the taskspecific ability pretraining stage and is directly used in the downstream task fine-tuning process.

Pre-training Data Collection

For task-specific ability pre-training, we collect several single-intent SLU benchmarks that are easier to obtain compared to multi-intent SLU data, including ATIS [Hemphill et al., 1990], Facebook [Schuster et al., 2019], HWU64 [Liu et al., 2021], Leyzer [Sowa´nski and Janicki, 2020], MMNLU2022 [FitzGerald et al., 2023], SNIPS [Coucke et al., 2018] and xSID [van der Goot et al., 2021].

Finally, we collect 136K samples for TSAP stage. The detailed statistics are shown in Table 2.

2.3 Down-StreamTask Finetuning

After completing the task-agnostic and task-aware ability pretraining, we directly fine-tune the pretrained model for the multi-intent SLU task. During the downstream task finetuning (DSTF) stage, multiple intent detection and slot filling can be simultaneously optimized by multi-task learning.

3 Experiments

3.1 Implementation Settings

We evaluate DPF using two standard benchmarks: MixATIS and MixSNIPS [Qin et al., 2020]. The batch sizes for all experiments are selected from {8, 16, 32}, and the learning rates are set within the range [1 × 10-6 , 7 × 10-5 ]. We employ AdamW [Loshchilov and Hutter, 2019] for training our model with a weight decay parameter set to 1 × 10-8 . All experiments are conducted on V100 16G and V100 32G. All models are selected from the development set and evaluated on the test set.

3.2 Baselines

We compare our approach with the following non-pretrained models and pretrained models. Non-pretrained baselines include: AGIF [Qin et al., 2020] implements an adaptive graph network to facilitate detailed multi-intent information interactions; GL-GIN [Qin et al., 2021b] employs a non-autoregressive architecture to accelerate decoding in simultaneous multiple intent detection and slot filling tasks; SDJN [Chen et al., 2022a] presents three interconnected decoders and a self-distillation technique to establish coherence between multiple intents and slots; GIS-Co [Song et al., 2022b] utilizes statistical co-occurrence frequencies of intents and slots as prior knowledge, significantly improving the performance; Co-Guiding [Xing and Tsang, 2022] introduces a framework based on heterogeneous semanticslabel graphs, gradually establishing and modeling mutual guidance between intent detection and slot filling.

The pretrained models containing DCA-Net [Qin et al., 2021a] effectively utilize bidirectional, task-specific interaction for simultaneous slot filling and intent detection; TFMN [Chen et al., 2022b] utilizes an encoder for multigrain representations, enhancing the utterance understanding; DeBERTav3 [He et al., 2023] incorporates a disentangled attention mechanism in pre-training to improve representation quality; MISCA [Pham et al., 2023] introduces a dual mechanism of intent-slot co-attention and label attention, boosting multi-task interaction; DGIF [Zhu et al., 2023] uses label semantic information as enriched priors and constructs a multi-grain interactive graph to map intent-slot correlations; MTLN-GP [Wan et al., 2023] presents a multidimensional type-slot label interaction network coupled with a global pointer network for efficient handling of nested and non-nested slots and slot incoherence, leading to quicker inference. Besides, we replace the encoders of non-pretrained models with DeBERTav3 for fair comparison. Baseline results are token from previous works [Chen et al., 2022a; Chen et al., 2022b; Song et al., 2022b; Xing and Tsang, 2022; Zhu et al., 2023; Wan et al., 2023]. The reproduced results are based on OpenSLU [Qin et al., 2023] framework.

3.3 Main Results

Following the settings of Goo et al. [2018] and Qin et al. [2020], we evaluate the performance of multi-intent SLU with three metrics: F1 score for slot filling (Slot F1), accuracy score for intent detection (Intent Acc.), and the exact match accuracy (EMA). The results are presented in Table 3. Our observations are as follows:

(1) Pretrained models beat most of non-pretrained approaches. As illustrated in Table 3, we observe that after replacing the encoders of the models with DeBERTav3, the performance of most of the models improves substantially, demonstrating that the knowledge learned by the pre-trained models can be used to enhance the multi-intent SLU.

(2) DPF remarkably improves multi-intent SLU performance. As illustrated in Table 3, DPF significantly outperforms all baselines on two benchmarks, including both pre-trained and non-pre-trained models. Specifically, on MixATIS dataset, DPF outperforms the DGIF model by 4.7% on EMA, while on MixSNIPS dataset, it surpasses Debertav3+Co-guiding by 7.5% on EMA, which verifies the effectiveness of DPF.

3.4 Analysis

In this section, we conduct comprehensive analyses to answer the following questions to better understand our approach: (1) Does TAAP capture task-agnostic knowledge? (2) Does TSAP capture task-specific knowledge? (3) Can TAAP and TSAP help each other? (4) What are the impacts of the decoupled pre-training paradigm? (5) Can DPF generalize well on few-shot setting? (6) What is the performance of ChatGPT? (7) Why DPF works?

Answer1: TAAP can Capture both Sentence-level and Token-level Task-agnostic Knowledge

To verify the effectiveness of task-agnostic ability pretraining (TAAP), we remove the procedure of TAAP and only keep the task-specific pre-training unchanged, which we refer to as w/o TAAP.

As shown in Table 4, we observe that the removal of TAAP leads to a decrease in Slot F1 drops by 1.1% for MixATIS and 2.8% for MixSNIPS. Additionally, Intent Acc. declines by 3.0% on MixATIS and drops by 0.5% on MixSNIPS, which demonstrates the effectiveness of TAAP. We attribute it to the fact that the incorporated sentence-level entity num prediction and token-level entity span detection can inject more task-general knowledge into pre-training procedure, which the previous work can not achieve.

Answer2: TSAP can Boost Task-aware Interaction across Intent Detection and Slot Filling

To demonstrate the effectiveness of task-specific ability pretraining (TSAP), we only keep the task-agnostic ability pretraining and directly use it for fine-tuning on downstream tasks. We refer it to the w/o TSAP.

The results are shown in Table 4 (w/o TSAP). We find that EMA drops significantly: 6.6% on MixATIS and 9.7% on MixSNIPS. We suppose that the TSAP can improve the taskaware interaction ability across the two related tasks, which is crucial for the multi-intent SLU.

Answer3: Combination of TAAP and TSAP Brings Further Improvement

To verify the effectiveness of the combination usage of TAAP and TSAP, we remove the two-stage pretraining and directly fine-tune the model with multi-intent SLU data.

Table 4 shows that when both TAAP and TSAP are removed, the overall performance suffers a further decline.

(a) EMA on MixATIS on Low-Resource Setting.

Figure 4: Low-Resource Performance.

Specifically, the model only obtains 47.0% EMA on MixATIS and 83.1% EMA on MixSNIPS, which indicates that the two pretraining stages are complementary to each other and can better facilitate multi-intent SLU.

Answer4: Decoupled Pre-training Paradigm rather than Pre-training Data Matters in DPF

It is natural to wonder whether the final performance is primarily attributed to the collected pre-training data or the proposed two-stage decoupled pre-training paradigm. To explore the question, we conduct an experiment by collecting all pretraining data from two stages and only employing one stage pre-training approach (named as Hybrid Pre-training). The loss is the sentence-level and token-level joint loss similar to Equation 6.

The results are shown in Figure 3. We have the following observations: (1) Hybrid Pre-training outperforms the baselines on two benchmarks, which indicates the pre-training procedure can benefit the multi-intent SLU task. (2) DPF beats hybrid Pre-training on the MixATIS and MixSNIPS, with improvements of 2.7% and 3.4% respectively. This confirms that the improvement comes from the introduced twostage pre-training paradigm rather than the incorporated pretraining data.

(b) The intent and slot performance on MixSNIPS.

Figure 5: ChatGPT vs. DPF.

Answer5: DPF Works Better on Few-Shot Settings

In order to demonstrate the effectiveness of our model on lowresource settings, we only select 1% to 50% of the original data for fine-tuning.

The results are illustrated in Figure 4. From the results, we observe that DPF beats all baselines on few-shot settings, which indicates that DPF can work in low-resource scenarios. We attribute it to the fact that knowledge captured from the pre-training procedure can be transferred to the lowresource setting, which is consistent with the previous observation [Gururangan et al., 2020].

Answer6: Investigation of ChatGPT

Recently, large language models (e.g., ChatGPT 1 ) have dominated the performance in the NLP literature. A natural question arises: Can ChatGPT excellently address the multi-intent SLU task? To answer this question, in this experiment, we utilize the prompt from Pan et al. [2023] to investigate ChatGPT for multi-intent SLU.

The comparison results between ChatGPT and DPF are illustrated in Figure 5. We observe that DPF outperforms ChatGPT on MixATIS and MixSNIPS across all metrics, which demonstrates that simply relying on ChatGPT is not sufficient to fully solve the complex multi-intent SLU problem.

Answer7: Qualitative Analysis

To provide a more intuitive understanding of the model, we present a case study that includes the results of three different scenarios: (1) solely using task-agnostic ability pretraining, (2) solely using task-specific ability pretraining, and (3) simultaneously using both task-agnostic ability pretraining and task-specific ability pretraining (DPF).

The case study is shown in Figure 6. Take the example as shown in Figure 6 (a) and Figure 6 (b), it fails to yield accurate predictions when either the task-agnostic ability pretraining or task-specific ability pretraining stage is retained

1 https://platform.openai.com

(c) Case study for combined usage in TAAP and TSAP stage

Figure 6: Case study. Texts with blue boxes denote the correct prediction while texts with red boxes stand for the wrong prediction.

alone, and can only be predicted correctly with combined usage, as depicted in Figure 6 (c). We attribute it to the fact that only employing TAAP ignores the interaction across slot and intent, causing the intent prediction wrongly while solely using TSAP stage cannot capture the token-level knowledge for slot filling. In contrast, DPF can capture the interaction between intent and slot, as well as token-level knowledge for slot filling, which brings improvement.

4 RelatedWork

Multi-intent Spoken Language Understanding (Multi-intent SLU) has gained growing interest due to its ability to discern multiple intents from a given utterance [Qin et al., 2021c]. Motivated by this, researchers shift their focus from singleintent SLU to multi-intent SLU. To this end, Gangadharaiah & Narayanaswamy [2019] first investigate a joint modeling technique for multi-intent SLU. Qin et al. [2020] introduce an adaptive graph network that enhances the integration of intent data for detailed slot filling. More importantly, they release two benchmarks to facilitate the multi-intent SLU community. Qin et al. [2021b] subsequently introduce a non-autogressive framework to improve the decoding speed. Chen et al. [2022b] develop a transformer-based model called TFMN, which adds an auxiliary intent number detection task to improve the model performance. Song et al. [2022b] investigate the utilization of statistical co-occurrence frequencies between intents and slots for multi-intent SLU interaction. Similarly, Xing & Tsanget al. [2022] propose a novel framework based on heterogeneous semantics-label graphs for multi-intent SLU. Wu et al. [2022] and Song et al. [2022a] explore the method of prompt-based generative framework. Pham et al. [2023] present MISCA, a joint model using intent-slot and label attention mechanisms to capture correlations without any additional graphs. Cheng et al. [2023a] propose MRRL framework, which refines output based on references with reinforcement learning. Cheng et al. [2023b] introduce TKDF, improving student models via knowledge distillation with evaluator and curriculum learning, which achieves promising performance. Zhu et al. [2023] propose DGIF model to leverage label semantics as enriched priors, building a multi-layer interactive graph for intent-slot correlation analysis. Concurrently, Wang et al. [2023] present MTLN-GP framework to address both nested and non-nested slot challenges, which can significantly improve the inference speed and performance.

Though the above approaches achieve promising performance, their models still heavily rely on a large amount of annotated multi-intent SLU data. In contrast, our work introduces a decoupled pre-training framework for multi-intent SLU, which allows the model to leverage large amounts of multi-intent-free data, such as NER and single-intent SLU data. To the best of our knowledge, this study represents the first investigation into the utilization of additional multiintent-free data to enhance multi-intent SLU.

5 Conclusion

In this paper, we introduce a decoupled pretraining framework (DPF), achieving to leverage abundant multi-intent-free data to enhance multi-intent SLU. Specifically, DPF first decouples multi-intent SLU into task-agnostic and task-aware abilities. Furthermore, DPF introduces a two-stage pretraining paradigm to enhance the two abilities, respectively. Experimental results show that DPF achieves superior performance on the MixSNIPS and MixATIS datasets and can also successfully generalize to low-resource settings.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (NSFC) via grant 62306342, 62236004 and 61976072. Libo Qin and Qiguang Chen contributed equally. This work was also sponsored by CCF-Baidu Open Fund. We are grateful for resources from the High Performance Computing Center of Central South University. Libo Qin is the corresponding author.