WDMIR: Wavelet-Driven Multimodal Intent Recognition

1 Introduction

Intent recognition is a key aspect of human-computer interaction, and its core goal is to enable machines to accurately grasp user intent and thus provide users with better service [Zhang et al., 2019; Huang et al., 2023; Qiu, 2024]. Recently, multimodal intent recognition has been used to understand the user's intent in more complex scenarios. Compared with unimodal intent, multimodal information fusion can improve intent recognition accuracy by making joint decisions [Soleymani et al., 2017; Chen et al., 2021; Zou et al., 2022; Zhang et al., 2024; Sun et al., 2025].

∗Corresponding author.

Figure 1: In the task of multimodal intent recognition, videos and audio contain relatively few key pieces of information, necessitating the in-depth mining of complementary information.

Researchers are currently paying greater attention to multimodal intent detection research as it focuses more on intricate real-world situations. To this end, [Zhang et al., 2022a] introduces the first multimodal intention recognition baseline dataset, MIntRec, and conducts experiments on models such as MulT [Tsai et al., 2019], MISA [Hazarika et al., 2020], and MAG-BERT [Rahman et al., 2020] to establish a baseline for intent recognition for subsequent research. Subsequently, in order to effectively integrate information from different modalities such as text, video, and audio, researchers have designed various models, all of which have achieved certain results in the field of multimodal intent recognition [Zhang et al., 2022c; Sun et al., 2024; Huang et al., 2024; Zhou et al., 2024]. Although existing methods have made some progress in multimodal intent recognition, the accuracy of multimodal analysis of user's intent is still limited due to the fact that the potential correlation between different modalities has not yet been fully explored, as well as the insufficiency of the video and audio modalities in semantic feature extraction. As shown in Figure 1, when the speaker says, "Uh, I am so honored to receive this sprite," we might initially infer that the speaker is joking. However, by carefully observing the speaker's facial expressions and analyzing their tone, we can discern that the speaker's true intention is to flaunt. so much so that multimodal intention recognition currently presents the following two challenges: first, how to deeply mine the semantic information in video and audio modalities; and second, how to efficiently align and fuse the features of text, video and audio modalities.

To address the first challenge, we propose a waveletdriven approach. As far as we know, we are the first to

Video:

introduce wavelet transform to drive the fusion of video and audio data information. Wavelet transform decomposes signals into low-frequency and high-frequency components. The low-frequency component captures the global characteristics of the signal, reflecting smooth trends and largescale variations [Satirapod et al., 2001], while the highfrequency component focuses on local details, such as abrupt changes, fine structures, and rapid variations [Lahmiri, 2014; Li et al., 2023].

To address the second challenge, we designed collaborative representations and progressive fusion modules. These modules aim to enhance the alignment and integration between wavelet-driven non-verbal modalities and text modalities through cross-modal mechanisms, achieving a transition from bimodal to trimodal collaborative representations. Subsequently, the progressive fusion module is utilized for deep representation, leveraging complementary information at different stages to improve the accuracy of multimodal intent recognition. Our primary contributions in this study are:

• We design wavelet-driven multimodal intent recognition methods to achieve fusion of nonverbal modal features in the frequency domain to improve the model's ability to understand and recognize multimodal data.
• We achieve cross-modal collaborative alignment to integrate multimodal information, and through progressive fusion, mine complementary information to enhance recognition accuracy.
• Our experiments significantly improved each metric on MIntRec and MELD-DA, validating the validity and generalizability of the method.

2 RelatedWorks

2.1 Multimodal Intent Recognition

Multimodal intent recognition aims to extract the user intent from multimodal information. However, challenges remain when dealing with scenarios involving text, visual, and acoustic modalities. [Zhang et al., 2022a] introduced the multimodal intent recognition task and released a MIntRec dataset, which integrates textual, visual, and acoustic modalities to recognize user intent comprehensively. [Zhou et al., 2024] proposed a token-level contrastive learning method with modality-aware prompts that effectively integrates text, audio, and video modality features through similarity-based modality alignment and cross-modal attention. [Huang et al., 2024] introduced a shallow-to-deep interactive framework with data augmentation capabilities to address the modality alignment issue by gradually fusing multimodal features and incorporating ChatGPT-based data augmentation methods. [Sun et al., 2024] proposed a context-enhanced global contrastive method that alleviates biases and inconsistencies in multimodal intent recognition by using within-video and cross-video interactions and retrieval, combined with global context-guided contrastive learning.

2.2 Multimodal FusionMethods

Multimodal fusion aims to integrate information from different modalities (e.g., text, video, and audio) to better understand and process information frommultiple sources, thereby improving system performance [Zhang et al., 2022b]. For example, [Zadeh et al., 2017] proposes a tensor fusion network that learns intra- and inter-modal dynamic features in an end-to-end manner. [Tsai et al., 2019] addresses the problem of aligning multimodal sequences in different time steps by using directed pairwise cross-modal attention. [Rahman et al., 2020] Introduced multimodal adaptation gates to finetune BERT to address non-verbal modal input. [Hazarika et al., 2020] maps each modality to two different subspaces to learn common and modality-specific features.

2.3 Wavelet Transform

Wavelet transform is a commonly used time-frequency analysis technique in signal processing that can convert information from the time domain to the frequency domain. The decomposed information exhibits both global and local characteristics. In recent years, many researchers have applied wavelet transforms to various fields. [Li et al., 2023] introduced a wavelet fusion module for facial superresolution tasks, addressing the issue in existing Transformerbased methods where global information integration often neglects relevant details, leading to blurring and limiting highfrequency detail recovery. [Phutke et al., 2023] proposed an end-to-end blind image inpainting architecture with a wavelet query multi-head attention transformer block, effectively repairing images by using the wavelet coefficients processed to provide encoder features as queries, bypassing the damaged region prediction step. [Sabry et al., 2024] applies wavelet transform to lung sound signal analysis, addressing the issues of noise contamination and artifact removal in lung sound signals. By performing a multi-scale analysis of the lung sound signals, effective features are extracted, improving the accuracy of lung disease classification. [Frusque and Fink, 2024] introduced wavelet techniques for denoising audio information. Our method mainly uses wavelet transform to decompose video and audio information at multiple levels, to realize the fusion of video and audio features in the frequency domain, and to realize the analysis of non-verbal modal information in the frequency domain to enhance its representation.

3 Method

3.1 Task Description

Let text, video, and audio data from three modalities are taken as inputs, their features are fused to predict user intent. Specifically, let the feature of the text modality be t, the feature of the video modality be v, and the feature of the audio modality be a. By designing a multimodal fusion module F, these features are mapped into a unified representation.

Subsequently, a classifier Fintent maps the fused features to the intent category space, generating a predicted distribution

where t ∈ Rdt , v ∈ Rdv , and a ∈ Rda represent the features of the text, video, and audio modalities, respectively, with dt, dv, and da denoting the dimensions of each modality's features. H ∈ Rdh is the fused feature representation; dh is the dimension of the fused features; ˆ y ∈ Rc represents the predicted probability distribution over c intent categories.

3.2 Framework Overview

The overall framework of WDMIR is shown in Figure 2. It mainly consists of Wavelet-driven Fusion Module (WFM), Collaborative Representation Module (CRM) and Progressive Fusion Module (PFM). WFM can realize the feature fusion of video and audio in frequency domain by multilevel decomposition of video and audio in the frequency domain. CRM utilizes the cross-modal mechanism to realize the collaborative representation of text modality and wavelet-driven video and audio modality to realize the collaborative representation of two modalities to three modalities. PFM improves the accuracy of intent recognition using multilevel feature progressive fusion representation.

3.3 Feature Extraction

For text modality, in order to fully utilize the semantic information in the text [Liu et al., 2023], we use pre-trained model BERT [Lee and Toutanova, 2018] to encode the text and extract the last hidden layer as the feature representation of the text.

where t is the input conversation text, Ft is the text feature extracted through BERTEmbedding.

For the video modality, we follow the research approach in [Zhou et al., 2024] and choose the Swin-Transformer [Liu et al., 2021], which performs excellently in the field of computer vision, to sample the video frame by frame and perform pre-training, extracting the last hidden layer to represent the visual semantic information.

where fi is the i-th frame in each video segment, lv is the number of frames sampled from the video, and Fv is the visual semantic information extracted by Swin-Transformer.

For audio modality, we adopt the method [Zhang et al., 2022a] to pre-train the model using Wav2Vec 2.0 [Baevski et al., 2020] to extract the output of the last hidden layer as the audio feature representation.

where a is the audio sequence corresponding to the video clip and Fa is the audio feature extracted by Wav2Vec 2.0.

3.4 Wavelet-driven FusionModule (WFM)

WFM is able to decompose audio and video signals over sequences into the frequency domain, where fine-grained feature fusion of non-verbal modalities is achieved.Before further analyzing the video and audio features, we perform sequence alignment of the video and audio features using the CTCmodel [Graves et al., 2006].

where V and A represent the video and audio features after sequence alignment, respectively.

In order to facilitate the fusion of non-verbal modal information in the frequency domain. We choose Haar wavelet bases to perform 3-level 1-dimensional DWT transform on audio and video aligned in sequences to obtain the highfrequency and low-frequency components of video and audio information, which correspond to local and global features, respectively.

Where La and Lv are the low-frequency components of the audio and video features obtained through multilevel wavelet decomposition, respectively, and Hai and Hvi are the i-th high-frequency components of the audio and video features after multilevel decomposition, respectively.

To better extract global information from video and audio data, we employ a one-dimensional convolution with a kernel size of 3 to strengthen the representation of low-frequency components, further enhancing the model's understanding of global information.

where L′a and L′v are the low-frequency components of audio and video enhanced by one-dimensional convolution.

For the multilevel high-frequency components obtained from wavelet decomposition, we concatenate the components at each level sequentially to construct the total high-frequency components for each modality. This approach integrates high-frequency information from different levels, enhancing the model's ability to capture fine-grained features.

where Fhv and Fha are the total high-frequency components of video and audio, respectively.

To better facilitate the fusion of video and audio information, we use the BiLSTM model to map audio to the video space and video to the audio space, respectively.

where H′v is the visual feature obtained from the highfrequency component of the video through BiLSTM; H′a is the audio feature obtained from the high-frequency component of the audio through BiLSTM.

To better leverage the complementary information between modalities and enhance feature representation, we designed a frequency domain interaction module.

where HAV is the audio feature obtained from the interaction between audio and video, HV A is the video feature obtained from the interaction between video and audio, and ⊙ denotes the element-wise multiplication.

To better integrate the interaction-enhanced features with their respective low-frequency features, we perform an inverse wavelet transform to reconstruct the audio and video modalities, resulting in fused audio and video features.

where FAV is the reconstructed and enhanced audio modality feature, and FV A is the reconstructed video modality feature.

3.5 Collaborative RepresentationModule (CRM)

The text modality serves as the primary source of information for intent recognition. We treat the text modality as the main modality and perform pairwise interactions with the enhanced audio and video modalities separately. Cross-modal attention mechanisms are employed to achieve alignment and feature fusion. Subsequently, the audio and video modalities, weighted by the text features, are further interacted with the text modality, enabling deep fusion of the three modalities.

Considering that the text modality contains the primary information, we perform sequence alignment on the fused video and audio features through a cross-attention mechanism [Huang et al., 2024]. We choose the text modality Ft as the query Q, and the fused video FV A and audio modality FAV as the key and value K and V .

where Fvat and Favt denote the weighted video and audio features, respectively.

Then Ft as the query vector Q, Fvat as the key vector K, and Favt as the value vector V are used to realize the trimodal co-representation via cross-transformer [Tsai et al., 2019].

where Ftva denotes the three modal co-representation features of text, video, and audio.

3.6 Progressive FusionModule (PFM)

PFM improves the accuracy of multimodal intent recognition mainly through multilevel feature progressive fusion.To focus on the fused text, video, and audio model features from different visions. We stack Ft, Fvat and Favt to obtain the matrix Fm.

Then, the stacked Fm is enhanced through selfattention [Vaswani, 2017; Zhang et al., 2021] for feature enhancement.

where F = [F ′ t , F ′vat , F ′avt ] denotes the enhanced features.

We utilize LSTM to learn the encoded video and audio information, taking the hidden state of the last layer as output to prevent the loss of key features during the processing of video and audio characteristics, thereby enhancing the model's understanding of complex data.

where Flv and Fla denote the video features and audio features obtained by LSTM, respectively.

We concatenate the features obtained from multiple layers and map them to the output space through an MLP to get the final output ˆ y.

To make the output better approximate the true distribution, we use a cross-entropy loss function to measure the difference between the predicted and true values.

where L is the total loss function of the model. N is the number of samples and C is the number of intended categories. yij is the true label of the i-th sample in the jth category. ˆ yij is the probability distribution of the i-th sample belonging to the j-th category as predicted by the model.

4 Experiments

4.1 Datasets

We conduct experiments on two datasets, MIntRec [Zhang et al., 2022a] and MELD-DA [Saha et al., 2020]. MIntRec is a multimodal intent dataset containing text, video, and audio, with 2224 samples and 20 intent categories. It includes 1334, 445, and 445 samples for training, validation, and testing, respectively. MELD-DA is a multi-round emotion conversation dataset containing text, video, and audio, with 9988 samples and 12 emotion conversation behavior labels. It includes 6991, 999, and 1998 samples for training, validation, and testing, respectively.

4.2 Baselines

In our experiments, we will use state-of-the-art multimodal fusion methods as baselines: (1) MISA [Hazarika et al., 2020] projects each modality into two different subspaces to learn the fusion of common features and unique attributes of different modalities. (2) MulT [Tsai et al., 2019] potentially converts one modality to another for feature fusion by directing paired cross-modal attention. (3) MAG-BERT [Rahman et al., 2020] introduces a multimodal adaptation gate as an accessory to fine-tune BERT to enable it to handle non-verbal data for multimodal data fusion. (4) TCL-MAP [Zhou et al., 2024] uses the similarity of modalities to design a multimodal perceptual cueing module for modal alignment, and uses a cross-modal attention mechanism to generate modal perceptual cues for multimodal fusion. (5) SDIF-DA [Huang et al., 2024] enhances text data through ChatGPT, then designs interaction modules from shallow to deep and gradually aligns and fuses features between different modalities effectively. (6) CAGC [Sun et al., 2024] enhances the capture of global contextual features by mining the contextual interaction information within and between videos, thus effectively solving the problems of perceptual bias and inconsistency of multimodal representations.

4.3 EvaluationMetrics

Based on previous work [Zhou et al., 2024], we use Accuracy (ACC), Weighted F1 Score (WF1), Weighted Precision (WP), and Recall (R) as the evaluation metrics of the model. The impact of sample unevenness on model performance is reduced by a weighted average of the number of samples in each category by WF1 and WP.

4.4 Experimental Settings

In our experiments, bert-base-uncased 1 and wav2vec2-base960h 2 from the Huggingface are used as the pre-training models for extracting mentioned text and audio features. Video features are extracted from the Torchvision library by using swin b pre-trained on ImageNet1K. Adam [Loshchilov, 2017] as an optimization parameter throughout the experiment. The training batch size is 16, and the validation and test batch sizes are both 8.

4.5 Main Result

Our method is compared with the optimal method in §4.2, and the experimental results are shown in Table 1. From the analysis of the experimental results, it is clear that our approach has made significant progress in two main areas. First, on the MIntRec dataset, our method achieves 1.13%, 1.07%, 1.08%, and 0.99% improvement in four key performance metrics, namely, ACC, WF1, WP, and R, respectively, when compared to the best existing baseline method. These results strongly demonstrate the effectiveness of the method in handling multimodal intent recognition tasks in complex real-world scenarios. Second, on the multi-round sentiment dialog analysis dataset MELD-DA, our method also demonstrates better performance, improving 0.41%, 1.17%, 1.11%, and 1.22% in the four evaluation metrics of ACC, WF1, WP, and R, respectively, compared to the optimal baseline. This result confirms the effectiveness and robustness of our proposed method in the task of conversational emotion recognition. Overall, the experimental results fully demonstrate that MIntRec-WDMIR

1 https://huggingface.co/google-bert/bert-base-uncased

2 https://huggingface.co/facebook/wav2vec2-base-960h

Figure 3: MIntRec-WDMIR represents the confusion matrix evaluation results with wavelet-driven fusion module on the MIntRec dataset, where leading zeros before decimal points are omitted.

our method outperforms the existing state-of-the-art baseline methods on all evaluation metrics for two representative datasets, which in turn validates the effectiveness and generalizability of the method.

4.6 Ablation Study

To further explore the impact of different modules on the performance of the WDMIR method on the MIntRec, we perform ablation experiments on the Wavelet-driven Fusion Module, Collaborative Representation Module, and Progressive Fusion Module to determine the contribution of each module to the overall performance.

Wavelet-driven Fusion Module

To evaluate the performance of WFM, we removed it from the model. The experimental results are shown in Table 2: on the MIntRec dataset, the model's accuracy (ACC) dropped by 3.03%, weighted F1 score (WF1) decreased by 3.2%, weighted precision (WP) declined by 3.06%, and recall (R) decreased by 3.88%. On the MELD-DA dataset, MIntRec-NWDMIR

Figure 4: MIntRec-NWDMIR denotes the confusion matrix evaluation results obtained after removing the wavelet-driven fusion module in the MIntRec dataset, with leading zeros before decimal points omitted.

ACC dropped by 1.6%, WF1 by 1.66%, WP by 2.31%, and R by 1.68%. These results indicate that WFM can effectively extract fine-grained semantic information from nonverbal modalities in intent and emotion data. Moreover, the confusion matrix in Figure 3 and the visualization in Figure 4 on the MIntRec dataset suggest that WFM also contributes to the recognition of implicit intents.

Collaborative Representation Module

To assess the effectiveness of the collaborative representation module, we conducted an ablation study by removing Ftva, with the results shown in Table 2. The removal of Ftva led to a significant performance drop on both the MIntRec and MELD-DA datasets, particularly on MIntRec, where ACC, WF1, WP, and R decreased by 3.03%, 3.25%, 3.44%, and 3.59%, respectively. These results demonstrate that the collaborative representation module effectively integrates complementary information across multiple modalities, thereby enhancing the accuracy and robustness of intent recognition.

Progressive Fusion Module

To verify the impact of the progressive module on the overall performance of WDMIR, we sequentially removed Flv and Fla, as well as F ′vat and F ′avt . The experimental results are shown in Table 2. The results demonstrate that the progressive fusion module improves model performance on both the MIntRec and MELD-DA datasets. Specifically, Flv and Fla compensate for the loss of audio-visual information, while F ′vat and F ′avt enhance deep fusion between modalities. The ablation experiments further validate the critical role of the progressive fusion module in optimizing multimodal information complementarity and improving the overall performance of the model.

4.7 Performance with Single-ModalityMissing

We further analyzed the impact of removing a specific modality on the model's performance, as shown in Figure 5. The vertical axis of the figure depicts the model's performance metrics-ACC, WF1, WP, and R-on the MIntRec and MELD-DA datasets. The experimental results indicate that the absence of the text modality leads to a significant decline in performance, demonstrating that text plays a dominant role in multimodal tasks. Removing the video or audio modalities also causes performance degradation, but to a lesser extent, suggesting that they mainly provide auxiliary information.

4.8 F1-Score Analysis Across Intent Categories

To better evaluate the performance of WDMIR on MIntRec, we compare its f1-score with baselines to assess how each method performs across fine-grained intent categories. As illustrated in Figure 6, our approach demonstrates strong performance, particularly in the intent categories of "Flaunt", "Joke" and "Oppose" suggesting that WDMIR is adept at managing complex intent categories. WDMIR performs similarly to other methods in the "Care" and "Complain" categories, and even outperforms in some areas, indicating stable performance in common intent categories, likely due to the wavelet-driven nonverbal modal fusion. However, despite some improvements in certain intent categories, WDMIR's performance in categories like "Taunt", "Oppose" and "Joke" is not as strong as that of other methods, which may be linked to the influence of wavelet-driven nonverbal modal fusion. The F1-scores metric performed significantly below human levels on the "Taunt", "Oppose", and "Joke" intentions. This suggests that WDMIR still faces challenges in understanding emotions that are heavily dependent on context and humor, and that further improvements in understanding linguistic humor and contextual variation are needed.

5 Conclusion

In this paper, wavelet transform is introduced into the field of multimodal intent recognition for the first time, and a novel wavelet-based multimodal intent recognition method, WDMIR, is proposed. The method serializes multilevel decomposition of video and audio data by wavelet transform and realizes the extraction and fusion of video and audio features in the frequency domain. Through the collaborative fusion module, it realizes the collaborative representation from bimodal to trimodal, which improves the expression of deep features and better identifies the semantic information in multimodal data. Through the progressive fusion module, the whole different levels of feature representations are effectively represented to improve the accuracy of multimodal intent recognition. Comprehensive experimental results show that the proposed wavelet-driven approach can enhance the performance of the model in multimodal intention recognition tasks. This study not only demonstrates the potential of wavelets in multimodal learning but also provides a new perspective on feature extraction and fusion of multimodal data.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Grants No. 62337001, U23A20319, 62406303), the Key Technologies R&D Program of Anhui Province (No. 202423k09020039), the Anhui Provincial Natural Science Foundation (No. 2308085QF229), and the Fundamental Research Funds for the Central Universities. Additional support was provided by the Youth Science and Technology Talent Growth Project of Guizhou Provincial Department of Education (Grant No. KY[2022]054), the Guizhou Provincial Higher Education Undergraduate Teaching Content and Curriculum System Reform Project (Grant No. GZJG2024331), and the Guizhou Provincial Science and Technology Projects (Grant No. QKHJC[2024]Youth012).