Generating More Audios for End-to-End Spoken Language Understanding

1 Introduction

Spoken Language Understanding (SLU) focuses on comprehending the spoken utterances and generating relevant predictions, which is widely used in the personal assistants, spoken dialogue systems, and recent voice-controlled devices [Wang et al., 2005; Cheng et al., 2023a; Zhu et al., 2023]. As demonstrated in Figure 1, for the audio input, SLU typically contains two subtasks: Slot Filling (SF) and Intent Detection (ID). SF is a sequence labeling task [Tur and De Mori, 2011], aiming to assign the slot to each token in spoken utterance, and ID is a classification task [Cheng et al., 2023c], aiming to predict the intent label of the entire spoken utterance.

Traditional SLU approaches integrate an automatic speech recognition (ASR) model and a natural language understanding (NLU) model within the two-step pipelines [Hakkani-T¨ur et al., 2006; Morbini et al., 2012]. However, the prediction of ASR may contain errors and lose the prosodic information which is beneficial for NLU. Besides, cascaded systems usually have higher latency, which is not conducive to deploying models in real situations. For the reason, end-to-end SLU has attracted increasing attention in the recent years [Chen et al., 2018; Chung et al., 2021]. For end-to-end SLU, the predicted results are directly generated from input audio without generating the intermediate transcripts.

Due to the modality gap between audio representations and text embeddings [Mai et al., 2020], training end-to-end SLU models directly is much more challenging than training cascade SLU models. To address the issue, a mainstream method is to train end-to-end SLU models with the assistance of transcripts. [Seo et al., 2022] designs junctional representation to leverage the transcripts as the interface of the ASR model and the NLUmodel. [Ma et al., 2023b] adopts knowledge distillation and leverages the transcripts to train an NLU model as the teacher model. However, owing to the exorbitant expenses of collecting the transcripts, transcripts are not always available. In addition, there are still thousands of unwritten languages in the world, it is impractical to obtain their corresponding transcripts [Zhang et al., 2021]. Therefore, designing methods to train end-to-end SLU models without utilizing the transcripts is becoming an important research direction.

In the absence of transcripts, audio and their corresponding labels should be utilized more effectively to train the superior end-to-end SLU models. Motivated by the recent success of data augmentation in many other tasks [Fang and Feng, 2023; Cheng et al., 2023b; Wang et al., 2023b], we decide to generate more audios according to the labels. However, as a crossmodality task, directly utilizing labels to generate more audio will face two major difficulties:

(1) Modeling difficulty. Due to the modality gap between textual labels and audio inputs, training a single model to directly generate audios based on the labels is challenging. Motivated by the success of discrete tokens in various tasks [Hsu et al., 2021; Zhang et al., 2023a; Wang et al., 2023a], we propose to employ the discrete tokens as a bridge. We separately train two different language models, where the first language model is leveraged to transform the labels to semantic tokens, and the second language model uses the semantic tokens and acoustic tokens of audio inputs to obtain the generated acoustic tokens. By using the two-stage generation method, we can effectively address the challenges in modeling.

(2) Data scarcity. By contrast, SLU dataset is relatively small [Bastianelli et al., 2020; Gerz et al., 2021]. To achieve the high performance without relying on transcripts for training, it is helpful to generate more audio data. As a result, we design multiple prompts for the first model, enabling the generation of more semantic tokens that align with specified slots and intents. Acoustic tokens from different source audios are fed into the second language model to enhance the generation of audios with the diverse speaker characteristics and thereby further expand the training data.

In this paper, we propose GMA-SLU to generate more audios based on the labels. All experiments are conducted on a monolingual SLU benchmark dataset SLURP [Bastianelli et al., 2020] and the multilingual SLU dataset MINDS-14 [Gerz et al., 2021]. Experimental results show that our method surpasses the previous best Textless end-to-end SLU models and achieves a comparable performance with the models trained with the assistance of transcripts. Further analyses also verify the advantages and effectiveness of our proposed method.

To sum up, the contributions of our method are three-fold:

• We propose the framework GMA-SLU, which generates more audios based on labels to enhance end-to-end SLU.
• We separately train two language models to tackle modeling difficulty and data scarcity.
• Experiment results on two benchmark datasets show that our proposed model surpasses the previous best model.

2 RelatedWork

2.1 Spoken Language Understanding

SLU aims at comprehending and interpreting the spoken language. Cascaded SLU methods work on the ASR transcripts, where solving the error propagation poses a significant challenge [Lee et al., 2012; Chang and Chen, 2022]. Recent endto-end SLU approaches have gained attention, especially with the performance gap compared to cascaded SLU systems being mitigated in many cases due to the rich knowledge of pretrained models [Serdyuk et al., 2018; Haghani et al., 2018; Wang et al., 2021]. However, most of previous SLU methods use transcripts for training, which are not always available. In our study, we explore leveraging the audio input more effectively to maintain the high performance.

2.2 Discrete Tokens

Discrete tokens are widely used in various speech processing tasks [Borsos et al., 2022; Rubenstein et al., 2023; Dong et al., 2023b]. Current discrete tokens can be broadly classified into two types: semantic tokens [Baevski et al., 2020; Hsu et al., 2021] and acoustic tokens [Zeghidour et al., 2021; Borsos et al., 2022; Garcia et al., 2023]. Semantic tokens are derived from pre-trained models that use masked language modeling as their training objective [Vaessen and Van Leeuwen, 2022]. These tokens primarily capture the content information in the speech while neglecting the paralinguistic aspects [Polyak et al., 2021]. Acoustic tokens can be obtained from neural audio codecs with reconstruction as their training objective, aiming to capture all facets of information, including timbre, content, prosody, and recording conditions [D´efossez et al., 2022]. In our approach, we use semantic tokens as the bridge of the two language models to tackle the modality gap. Besides, we use acoustic tokens from different source audios to generate more audios with different speaker characteristics.

2.3 Language Models

Recently, natural language capabilities have obtained significant advancements via language modeling. Language modeling encompasses the approaches to predict the corresponding next tokens in the utterance or those masked spans [Devlin et al., 2019]. [Brown et al., 2020] introduces a novel large language model with 175 billion parameters, demonstrating the robust performance in many tasks in zero-shot, one-shot, and few-shot scenarios. [Ma et al., 2023a] finds that existing language models are not effective few-shot information extractors and proposes the novel adaptive paradigm to leverage the strengths of language models effectively. In our approach, we design two language models to generate more pseudo audios.

3 Method

In this section, we introduce the problem definition (§3.1) and the model architecture (§3.2). The audio generation contains three parts, including discrete tokens generation (§3.3), labelto-token language model (§3.4), and token-to-audio language model (§3.5). Finally, we present the data filtration (§3.6) and the training strategy (§3.7) of our method. Note that these two language models are trained from scratch, and we also report the results obtained by supervised fine-tuning the recent large language models in Sec. 5.7 for reference.

3.1 ProblemDefinition

Given the audio x = (x1, x2, . . . , xm), where m is the length of audio waveform x, cascade SLU methods first transform x to transcript y = (y1, y2, . . . , yn) and then predict the intent oI and the slot oS = (oS1 , oS2 , . . . , oSn ), where n is the length of transcript y. End-to-end SLU methods directly predict oI and oS based on x without generating y. It has been verified that utilizing transcript y for auxiliary training could improve the performance of End-to-end SLU methods. However, considering that transcripts are not always available for training, we decide to train the Textless end-to-end SLU model without using transcripts in this paper unless otherwise specified.

3.2 Model Architecture

Following previous work [Wang et al., 2021], our framework GMA-SLU consists of an acoustic encoder to generate representations for the audio input and two decoders for ID and SF, respectively. For a fair comparison, we use the HuBERT [Hsu et al., 2021] as the acoustic encoder. For ID, the decoder consists of two self-attention layers [Vaswani et al., 2017] and the following average pooling layer. For SF, the decoder consists of four self-attention layers. The overview of model architecture is illustrated in Figure 2.

The training objectives of ID and SF are as follows:

where ˆyi,I and ˆyi,S j denote the gold intent label and gold slot label, respectively, nI denotes the number of the intent labels, and nS denotes the number of the slot labels.

The final training objective L is as follows:

where α is the coefficient balancing the two subtasks.

3.3 Discrete Tokens Generation

We utilize two kinds of discrete tokens to train language models, including semantic tokens and acoustic tokens.

We use a pre-trained HuBERT 1 [Hsu et al., 2021] to generate semantic tokens. HuBERT produces continuous representations at a rate of 50Hz for the audio. We adopt the K-means clustering algorithm to continuous representations of the audio input and transform these continuous representations into their corresponding cluster indices subsequently. These cluster indices are regarded as semantic tokens. By this approach, the audio input x = (x1, x2, . . . , xm) is converted to semantic tokens z = (z1, z2, . . . , zT ), where zt ∈ {0, 1, ..., K-1}, zt denotes any token in z, K denotes the number of clusters, and T = ⌊ m 320 ⌋ denotes the number of frames. Through using semantic tokens, the modeling difficulty could be alleviated.

We utilize a pre-trained EnCodec 2 [D´efossez et al., 2022] to generate the acoustic tokens. The EnCodec model is a simple streaming and convolutional-based encoder-decoder architecture with a sequential modeling component utilized to the latent representation, both on the encoder side and decoder side, and the audio input is sampled at 24 kHz. Acoustic tokens can preserve the speech information more effectively than semantic tokens, so we decide to utilize them to assist in preserving the speech information of speakers.

1 https://github.com/facebookresearch/fairseq/tree/main/ examples/hubert

2 https://github.com/facebookresearch/encodec

3.4 Label-to-Token Language Model

As demonstrated in the left part of Figure 3, we train the labelto-token language model to generate semantic tokens according to the labels. Motivated by the recent success of the reduction strategy [Lee et al., 2022], the repeating semantic tokens are merged to obtain the reduced semantic tokens. In order to facilitate the generation of semantic tokens, we design several prompts to construct the training corpus for natural language based on the semantic tokens and labels. For example, a simple prompt can be like: "Generate the utterance whose intent is [I label] and slot is [S label]: [semantic token]", where [I label] will be replaced with the intent label, [S label] will be replaced with the slot label, and [semantic token] will be replaced with the reduced semantic tokens. More prompts are demonstrated in Table 1. Via using the prompts, the language model implicitly improves the alignment within the representation space between semantic tokens and labels. It should be noted that since the MINDS-14 dataset only includes the intent labels, the portion containing slot labels will be removed from the example prompts, and the processed prompt can be like: "Generate the utterance whose intent is [I label]: [semantic token]". During the training stage, semantic tokens of the source audios are utilized, and during the inference stage, the generated semantic tokens are fed into the token-to-audio language model to generate the pseudo audios. The prompt is randomly selected during both training and inference.

3.5 Token-to-Audio Language Model

As illustrated in the right part of Figure 3, we train the tokento-audio language model to generate pseudo audios. Its input contains three parts, where the first part is the acoustic tokens of another randomly selected audio from the training data, the second part is the semantic tokens obtained by expanding the semantic tokens from the label-to-token language model, and the third part is the corresponding acoustic tokens.

Considering that the semantic tokens obtained by the labelto-token language model are reduced, we introduce a duration predictor [Ren et al., 2020] to predict the duration of semantic tokens. The predictor consists of two 1D-convolutional layers with ReLU activation [Glorot et al., 2011], each succeeded by layer normalization and the dropout layer. Additionally, there is a linear layer to project the hidden states into the duration. Given a predicted duration vector d = (d1, d2, . . . , dT ∗) and the ground truth ˆd = ( ˆ d1, ˆ d2, . . . , ˆ dT ∗), the training objective LD of the duration predictor is as follows:

By using the duration vector, reduced semantic tokens can be expanded via repeating each semantic token. For instance, before being used as the input of the label-to-token language model, the semantic tokens (1, 2, 2, 3, 3, 4, 4) collapse to the reduced semantic tokens (1, 2, 3, 4) and the duration vector is (1, 2, 2, 2). Via predicting the duration, the reduced semantic tokens can be converted into original semantic tokens. During the training stage, ground truth duration is applied, and during the inference stage, the predicted duration is applied.

The acoustic tokens are generated by feeding the source audios into the encoder of the EnCodec model. Since the acoustic tokens can preserve useful speech information more effectively, we use them to assist in generating audios that matches the speech information of the speakers in the training set more accurately. To solve data scarcity, we randomly select another source audio from the training set to obtain acoustic tokens as the first part of the input. During the whole training stage, the third part is obtained from these source audios, and during inference, the generated acoustic tokens are fed into the decoder of the EnCodec model to synthesize the audios.

3.6 Data Filtration

In order to filter out the low-quality generated audios, we train an additional SLU model to predict the intent and slot of the generated audios, whose model architecture is the same as the architecture in Sec 3.2. We assess the intent accuracy and the slot SLU-F1 of the generated audios. Only the top ρ% audios are kept based on the sum of intent accuracy and slot SLU-F1. Similarly, since only intent labels are available in MINDS-14 dataset, only intent accuracy is considered for data filtration.

3.7 Training Strategy

Though we apply the data filtration, the generated audios still contain the noise. Therefore, to dominate the training process with the real audios, we first pre-train the SLU model utilizing generated audios, followed by fine-tuning on real audios.

4 Experiments

4.1 Datasets andMetrics

We conduct all the experiments on a monolingual SLU benchmark dataset SLURP 3 [Bastianelli et al., 2020] and the multilingual SLU dataset MINDS-14 4 [Gerz et al., 2021]. SLURP dataset constitutes the repository encompassing 72.2k real audio recordings and 69.3k synthetic audio for a broad range of speech commands, each encapsulating brief interactions with the home assistant, which is considered the most challenging SLU dataset due to its lexical complexity. SLURP is meticulously annotated across 3 semantic layers, including scenario, action, and entities, where a pair (scenario, action) defines an intent. Unlike SLURP dataset focuses on English utterances, MINDS-14 is a multilingual SLU dataset for the banking scenarios with 14 distinct intents in 14 languages. Each language incorporates approximately 600 utterances. We report the results of 4 languages, including en-US, fr-FR, pl-PL, and koKR with a 30-20-50% train-dev-test split following the previous work for a fair comparison [Conneau et al., 2022].

Following [Dong et al., 2023a; Peng et al., 2022; Conneau et al., 2022], for SLURP, we evaluate the performance of slot filling with SLU-F1 score, intent detection with accuracy, and for MINDS-14, we evaluate the performance of intent detection in different languages with accuracy.

4.2 Implementation Details

We utilize the pre-trained HuBERT 5 [Hsu et al., 2021] following the base configuration as the acoustic encoder. Following previous work [Wang et al., 2021], the batch size is set to 16. We apply the Adam optimizer [Kingma and Ba, 2015], and 4k warm-up updates to optimize parameters, where the learning rate is increased from 4e-4 to 2e-3. If the loss on dev set does not decrease for 5 epochs, the training process will early-stop to avoid overfitting. For all experiments, we choose the model that achieves the best performance on the dev set and evaluate it on the test set. The weight α is set to 0.9. For the semantic tokens, we utilize the pre-trained quantized model 6 where the number of clusters K is 100 to convert the audios to semantic tokens. During the data filtration stage, we set the threshold ρ to 80. All experiments are conducted at an Nvidia V100.

4.3 Baselines

SLURP We compare our GMA-SLU with four SLU baselines trained without transcripts, including MTL-SLT [Huang et al., 2022], Speech-Brain [Ravanelli et al., 2021], Branchformer [Peng et al., 2022], and HuBERT SLU [Wang et al.,

3 https://github.com/pswietojanski/slurp

4 https://huggingface.co/datasets/PolyAI/minds14

5 https://dl.fbaipublicfiles.com/hubert/hubert base ls960.pt

6 https://dl.fbaipublicfiles.com/textless nlp/gslm/hubert/km100/ km.bin

2021]. We also report the results of SLU models trained with the assistance of transcripts for reference, including CTI [Seo et al., 2022], CIF-PT [Dong et al., 2023a] with the Conformer [Gulati et al., 2020] as the acoustic encoder and CIF-PT with the pre-trained Data2vec [Baevski et al., 2022] as the acoustic encoder. In addition, we also report the results of recent large language models for reference, including ChatGPT [OpenAI, 2023] and SpeechGPT [Zhang et al., 2023a]. Since the pretrained models applied by different SLU models might be different, we list the pre-trained models in Table 2 for reference.

MINDS-14 We compare our GMA-SLUwithLaBSE [Gerz et al., 2021] and XLSR [Lozhkov, 2022]. In addition, we also report the results of ChatGPT for reference. LaBSE performs SLU in a cascade manner and the input is the ASR transcripts. BERT [Devlin et al., 2019] is applied as its pre-trained model. XLSR is trained without the assistance of transcripts and performs SLU in an end-to-end manner as GMA-SLU. Wav2vec 2.0 [Baevski et al., 2020] is the pre-trained model of XLSR. The performance of ChatGPT is from [He and Garner, 2023].

5 Results and Analysis

5.1 Results on SLURP Dataset

Experimental results on SLURP dataset are shown in Table 2, from which we have the following observations:

1. Our approach achieves the consistent improvements in slot filling and intent detection. Specifically, GMA-SLU surpasses the previous best model training without transcripts by 2.01% SLU-F1 and 1.59% accuracy. This improvement could be attributed to the proposed audio generation strategy, which can enlarge the training set and improve the robustness.
2. Another encouraging result is that our method achieves the comparable performance with the models trained with the assistance of transcripts, which outperforms CTI and CIF-PT with Conformer as the acoustic encoder. This result indicates that generating more audios through our approach can alleviate the reliance on transcripts, which is particularly meaningful in scenarios where transcripts are unavailable.

(3) Following the previous work [He and Garner, 2023], we use ASR transcripts to evaluate the performance of ChatGPT and SpeechGPT. In this evaluation, these models are provided with 20 examples and prompted to adapt to ASR errors. However, though these large language models have shown the superiority in few-shot learning and zero-shot learning, there is still a performance gap between them and our approach. This result implies that large language models might encounter difficulties in understanding spoken utterances. As a result, it is still an essential task to establish a robust SLU framework and it warrants further exploration and investigation.

5.2 Results onMINDS-14 Dataset

Table 3 shows the experimental results on MINDS-14 dataset. We can obviously observe that the proposed approach outperforms previous models in all languages, which further verifies the effectiveness of our approach.

5.3 Ablation Study

To verify the benefits of our approach from various angles, we undertake ablation studies on SLURP and MINDS-14, whose outcomes are shown in the lower part of Table 2 and Table 3.

Effect of Reduced Strategy We apply the reduced strategy to merge the repeated semantic tokens in the proposed labelto-token language model and utilize the duration predictor to expand the reduced semantic tokens in the token-to-audio language model. To evaluate the superiority of the reduced strategy, we conduct an ablation experiment where we remove the reduced strategy, and the original semantic tokens are used as the bridge between these two proposed language models. We refer this experiment to w/o Reduced Strategy in Table 2 and w/o RS in Table 3. We observe the significant decreases in all the metrics across the two datasets, affirming that the reduced strategy makes the positive contribution to our method, which aligns with past observations in [Lee et al., 2022]. We believe it is because reduced strategy could make the semantic tokens resemble text more closely and simplify the whole generation process due to the shortened length of semantic tokens.

Effect of Different Acoustic Tokens In the token-to-audio language model, acoustic tokens of another audio in the training set are used as the first part of the input. To verify that it is more effective to use different acoustic tokens, we remove it and retain only the second and third parts of the input. We refer the experiment to w/o Different Acoustic Tokens in Table 2 and w/o DAT in Table 3. We could clearly discern that the absence of the first part leads to the decreased performance. We believe the reason is that applying the first part can solve data scarcity issue more effectively. Besides, via incorporating the acoustic tokens as the first part of the inputs, the acoustic tokens of the third part could align more closely with the established rules of acoustic tokens during the generation process.

Effect of Data Filtration To filter out the low-quality generated audios, we first perform the data filtration before training the SLU model. To verify the superiority of data filtration, we remove it and refer it to w/o Data Filtration in Table 2 and w/o DF in Table 3. It is evident that the performance declines after removing the data filtration, which suggests that the data filtration indeed could improve the performance. It is because even the two language models are meticulously designed, the noise in the generated audios is still inevitable.

Effect of Two-Stage Training Strategy We train the SLU model using a two-stage training strategy to alleviate the negative effect of the noise in generated audios. In order to evaluate its effectiveness, we remove it and refer this ablation experiment to w/o Two-Stage Training Strategy and w/o TSTS in Table 2 and Table 3, respectively. We could observe a decline in performance on these two datasets, which substantiates that the two-stage training strategy is effective.

5.4 Impact of Data Filtration Ratio

When performing the data filtration, it is important to choose the appropriate threshold ρ. To investigate how ρ affects SLU performance, we constrain ρ in [0, 20, 40, 60, 80, 100] on the SLURP, whose results are shown in Figure 3. When ρ = 100, data filtration is equivalent to being removed, which indicates that all generated audios are retrained, and when ρ = 0, our method degrades to HuBERT SLU [Wang et al., 2021]. From the results, we observe that our method achieves the best performance at ρ = 80%. Besides, when ρ = 100%, the performance is still relatively high, which verifies that the generated audios maintain a relatively high level of quality.

5.5 Case Study

To further showcase the effectiveness of our approach in handling SLU, we provide a case study on SLURP dataset applying HuBERT SLU and our GMA-SLU. As shown in Table 4, HuBERT SLU predicts the scenario of intent and the slots of some tokens incorrectly, whereas our model can predict them accurately. This result substantiates that the generated audios indeed has the potential to enhance the performance.

5.6 Effect of Different Audio Codecs

Recently, there has been an increasing amount of research on audio codecs, and more and more speech tokenizers with high performance are proposed. To verify that our approach is also adapted to the advanced tokenizers, we replace the EnCodec with SpeechTokenizer [Zhang et al., 2023b], TiCodec [Ren et al., 2023], and HiFi-Codec [Yang et al., 2023], respectively.

The corresponding results on SLURP are demonstrated in Table 5. We can observe that through employing these advanced speech tokenizers, the corresponding performance can be further enhanced, underscoring the generality of our method.

5.7 Effect of Large Language Model

In this study, we train both two language models from scratch. In order to explore the effect of recent large language models, we perform supervised fine-tuning to the large language models to obtain the label-to-token language model. As shown in Table 6, we observe that the recent large language models can indeed enhance the performance of SLU. We attribute this improvement to abundant semantic knowledge in large language models. Besides, SpeechGPT [Zhang et al., 2023a] and LTUAS [Gong et al., 2023] can perform better than Llama 2 [Touvron et al., 2023]. We believe this is because SpeechGPT and LTU-AS are meticulously designed for audios, which enables the generation of more accurate semantic tokens.

6 Conclusion

In this paper, we propose the framework GMA-SLU for SLU, which utilizes two language models to generate more audios as additional training data. Experiments on two datasets show that our model surpasses the previous best Textless SLU models and obtains a comparable performance with recent end-toend SLU models trained with the assistance of transcripts. We believe that our approach is generalizable to more fully utilize limited annotated data and thus improve the performance.

Acknowledgments

We thank all the reviewers for their insightful comments. This paper was partially supported by NSFC (No: 62176008).