DELTA: Disentangled Hierarchical Interaction and Adaptive Adjustment for Emotion and Intent Understanding in Multimodal Conversations

1 Introduction

Emotion and intent joint understanding in multimodal conversations [Singh et al., 2022; Liu et al., 2024; Liu et al., 2026] is an emerging task that aims to jointly infer emotional states and behavioral intents frommultimodal inputs, including audio, video, and text content. Unlike traditional methods that treat emotion and intent recognition as independent problems, MC-EIU emphasizes the semantic interaction between them, leveraging their interdependence to enhance joint understanding. This task holds immense potential for broad applications, such as dialogue systems [Danieli et al., 2015], recommendation systems [Li et al., 2024], mental health assessment tools [Francese and Attanasio, 2023], social robots [Wu et al., 2025], and customer service agents [Gong et al., 2022].

∗Corresponding author.

In recent years, the research focus has transitioned from independent emotion recognition [Li et al., 2023c; Xia et al., 2025] or intent recognition [Yang et al., 2024] towards the joint MC-EIU task. This shift has led to the development of several representative frameworks. For instance, Singh et al. [Singh et al., 2022] proposed EmoInt-Trans, a multimodal contextual Transformer for joint recognition. Liu et al. [Liu et al., 2024] introduced the EI2 network to model the correlation between emotion and intent, while Zheng et al. [Zheng et al., 2025] developed MEI-Pro to leverage progressive interactions. While these methods have achieved promising performance, certain limitations remain. Regarding multimodal processing, current models often lack explicit mechanisms to decouple redundant noise from essential semantics or to dynamically distinguish the varying contributions of distinct modalities. Consequently, irrelevant signals may interfere with feature learning. Moreover, concerning task synergy, most approaches primarily rely on shallow, single-round interaction mechanisms. Such designs may not fully capture the deep, iterative mutual reinforcement between emotion and intent, and risk propagating noise from one task to the other.

A primary challenge lies in the interference from multimodal noise and the neglect of dynamic learning variations [Zhu et al., 2019; Li et al., 2023c]. Multimodal data naturally contains noise and redundancy. These interfering signals degrade the quality of feature representations, making it difficult to extract consistent emotion and intent semantics. Moreover, existing methods often overlook that the contribution of each modality varies dynamically. Relying on a static fusion strategy may allow a dominant but irrelevant modality to overshadow critical subtle cues provided by others. Additionally, the training process is often dominated by simple samples, limiting the model's ability to generalize to hard samples with complex multimodal conflicts. Therefore, to achieve robust understanding, it is essential to effectively filter out noise and adaptively adjust the learning focus across both modalities and instances.

Another key challenge lies in modeling the deep and iterative interaction between emotion and intent [Singh et al., 2022]. In real-world dialogue, emotion and intent are not independent. As shown in Figure 1, they mutually influence each other. However, current approaches primarily employ single-round direct interactions, which may not fully capture this dynamic interplay. Worse still, these direct pathways risk becoming "noise corridors," where ambiguous signals or errors from one task could potentially propagate to the other. This lead to error accumulation instead of the desired mutual enhancement. To address these challenges and achieve more effective joint understanding, two fundamental questions need to be carefully considered: Q1: How can we effectively decouple redundancy and noise from essential semantics, and dynamically balance both modality contributions and the learning focus on samples of varying difficulty? Q2: How can we achieve robust interaction and mutual enhancement between emotion and intent while simultaneously filtering out potential noise?

To overcome the aforementioned limitations, we propose DELTA, a novel Disentangled Hierarchical Interaction and Adaptive Adjustment framework for MC-EIU. Firstly, to eliminate redundancy and noise, we introduce a Disentangled Feature Denoising (DFD) module. By leveraging orthogonal decomposition, we project raw inputs into modalityinvariant and modality-specific subspaces. This enables us to effectively filter out inherent redundancy and conflicting noise while retaining the essential information shared across modalities. Secondly, we devise a Dual-Level Adaptive Adjustment mechanism to calibrate learning dynamics. This strategy dynamically re-weights modalities based on their contextual relevance and modulates instance-level gradients. By doing so, we can effectively distinguish the varying contributions of different modalities and rectify the imbalance between simple and hard samples, preventing the model from being dominated by easy patterns. Finally, we replace shallow fusion with a Hierarchical Task-Bottleneck Interaction (HTBI) module. Acting as an iterative communication hub with progressive compression, HTBI facilitates a multiround, noise-filtered bidirectional flow of information. This process mimics the dynamic, step-by-step interplay between emotion and intent, allowing for deep interaction and mutual

• optimization while preventing noise propagation.
• The main contributions can be summarized as follows:
• We propose a Disentangled Feature Denoising (DFD) module combined with a Dual-Level Adaptive Adjustment mechanism, which effectively purifies multimodal representations, dynamically balances contributions across heterogeneous modalities, and adapts the optimization focus based on sample difficulty.
• We develop the Hierarchical Task-Bottleneck Interaction (HTBI) strategy, a progressive interaction paradigm that captures the deep-seated, iterative dependencies between emotion and intent, preventing noise propagation while fostering mutual reinforcement.
• Extensive experiments demonstrate that DELTA establishes new state-of-the-art results on standard benchmarks, proving the efficacy of our hierarchical and adaptive design.

2 RelatedWork

2.1 DisentangledMultimodal Representation Learning

Disentangled multimodal representation learning aims to decompose complex multimodal data into independent latent factors, typically separating modality-invariant and modalityspecific components [Tsai et al., 2018; Jiang et al., 2025]. [Hazarika et al., 2020] proposed MISA, which projects heterogeneous modalities into common and private subspaces using orthogonality constraints to reduce disparities. Building on this, recent advancements have incorporated adversarial training [Yang et al., 2022a] or distillation techniques [Li et al., 2023c] to further enhance feature independence and fusion efficacy. [Li et al., 2023a] propose a dual-level disentanglement mechanism and adaptive fusion strategies to enhance multimodal emotion analysis. [Fang et al., 2025] propose EMOE, which utilizes a mixture of modality experts and unimodal distillation to dynamically adjust modality importance and retain unimodal predictive capabilities. However, most existing methods primarily utilize disentanglement for feature alignment or general fusion, often overlooking the critical issue of inherent redundancy and conflicting noise within the data. In our DELTA framework, the primary goal of disentanglement is feature denoising. We employ an orthogonal decomposition strategy with explicit constraints to rigorously filter out redundancy and conflicting signals, ensuring that only purified, complementary semantics are retained to facilitate accurate joint emotion and intent understanding.

2.2 Emotion and Intent Joint Understanding in Multimodal Conversations

The task of Unified Multimodal Emotion-Intent Emotion and intent joint understanding in multimodal conversations (MCEIU) aims to simultaneously predict emotional states and behavioral intents from heterogeneous data, seeking a comprehensive grasp of user expressions. Recently, several studies have been dedicated to developing unified frameworks for this purpose. For instance, [Singh et al., 2022] proposed the EmoInt-MD dataset and proposed a multi-task transformer framework that utilizes shared contextual embeddings from neighboring utterances to predict both labels. Building on this, [Liu et al., 2024] introduced a novel framework, which explicitly models cross-modal interactions to capture the correlations between emotion and intent. More recently, [Yang et al., 2025] presented the BEAR framework to handle modality uncertainty while exploiting task dependencies. However, despite these advancements, existing methods often treat multimodal inputs as clean signals. They may not thoroughly eliminate inherent noise within modalities and tend to overlook the varying contributions of different modalities to the final prediction. Furthermore, most approaches rely on shallow or direct interaction mechanisms. They often lack the sufficient capability to capture the deep, iterative mutual reinforcement between emotion and intent, which limits their ability to fully leverage the implicit and subtle correlations shared between these two tasks.

3 Methodology

3.1 ProblemFormulation

Let D = {u1, u2, . . . , un} denote a multimodal dialogue consisting of n consecutive utterances, where speakers interact sequentially. For the i-th utterance ui (1 ≤ i ≤ n), the input involves three heterogeneous modalities: text (t), acoustic (a), and visual (v). We denote the extracted feature representations as xti ∈ Rdt , xai ∈ Rda , and xvi ∈ Rdv , respectively, where dm represents the feature dimension of modality m ∈ {t, a, v}. Each utterance ui is associated with an emotion label yemo i and an intent label yint i , belonging to the predefined category sets Yemo and Yint, respectively. Our objective is to jointly predict the associated emotion label yemo i and intent label yint i for every given utterance ui.

3.2 Overview

Figure 2 illustrates the overall architecture of our DELTA framework. DELTA first employs a Disentangled Feature Denoising module to project heterogeneous inputs into orthogonal subspaces, effectively eliminating redundancy. Subsequently, a Dual-Level Adaptive Adjustment mechanism dynamically calibrates modality weights and modulates instance gradients. Finally, the Hierarchical Task-Bottleneck Interaction module serves as a progressive communication hub, facilitating deep, noise-filtered mutual enhancement between emotion and intent for precise joint prediction.

3.3 Disentangled Feature Denoising

Multimodal data often contains prevalent redundancy and conflicting information across modalities, which potentially hinders the accurate understanding of core emotions and intents. To address this challenge, following [Li et al., 2023c], we propose a disentangled feature denoising module based on orthogonal decomposition. This module is designed to disentangle raw modal representations into modality-invariant subspaces and modality-specific subspaces, utilizing explicit constraints to reduce feature redundancy and filter out irrelevant noise.

For each input modal feature xmi (m ∈ {t, a, v}), we employ a shared Em inv encoder and three modality-specific encoders Em sp to disentangle it into modality-invariant features zm inv and modality-specific features zm sp :

where zm inv aims to capture the consistent emotion and intent semantics shared across modalities, while zm sp captures the unique information and variations specific to that modality.

To mitigate the redundancy between the two feature subspaces, we impose an orthogonality constraint by minimizing the cosine similarity between them, thereby forcing the model to learn complementary feature representations:

To ensure that the disentangled features retain all essential information from the original input, we concatenate them and reconstruct the original feature via a decoder Dm. Simultaneously, to guarantee the consistency of the private features, the reconstructed feature is fed back into the private encoder to obtain ˜ zm sp :

The corresponding reconstruction loss Lrec and consistency loss Lcon are defined as:

Although the above process achieves feature disentanglement, to further enhance the discriminability of the shared semantic feature zm inv , making samples with the same emotion label aggregate while separating those with different labels, we introduce a triplet loss:

where R denotes the set of triplets, with (a, p, n) representing the anchor, positive, and negative samples respectively, and µ is the margin parameter.

The total loss Ldec for this module is calculated as the weighted sum of the aforementioned terms:

where α represents the balancing coefficients.

Finally, we fuse the purified modality-invariant feature with the modality-specific feature to obtain the final denoised feature hmi , serving as input for subsequent modules:

Through such an orthogonal decomposition mechanism, we can effectively eliminate information redundancy and filter out irrelevant noise, while retaining key cues contributing to the understanding of emotion and intent. This process provides purer and more representative features for subsequent fusion and interaction.

3.4 Dual-Level Adaptive Adjustment

In multimodal dialogue understanding, the contribution of different modalities to emotion recognition and intent classification often varies dynamically with the context. Traditional methods typically treat all modalities equally, overlooking differences in their expressive capabilities, which may lead to a dominant modality overshadowing critical subtle cues from others. Furthermore, significant differences exist in learning difficulty across samples; the dominance of simple samples may limit the model's ability to generalize to hard samples. To address these issues, we propose an Dual-Level Adaptive Adjustment Mechanism that dynamically adjusts the learning process from two parts: Modality-level Dynamic Cooperation and Instance-level Gradient Adjustment.

Inspired by [He et al., 2025], we first introduce Modalitylevel Dynamic Weighting (MDW) to achieve a refined, context-aware fusion of emotion and intent features. Unlike static concatenation, this strategy generates task-specific importance weights based on the joint multimodal context. Specifically, for the decoupled features hmi (m ∈ {t, a, v}), we first concatenate them to capture global interaction information. Subsequently, we utilize two independent taskspecific Multi-Layer Perceptrons (MLPemo and MLPint) to project this context into importance scores smi,τ . These scores are then normalized via a Softmax function to obtain the final weights ωm i,τ :

where smi,τ denotes the logical score of modality m for task τ. The normalization ensures that ∑m ωm i,τ = 1. Based on these weights, we perform weighted integration to obtain the final emotion feature hemo i and intent feature hint i :

Complementing forward fusion, we employ Instance-level Gradient Adjustment (IGA) to balance simple and challenging samples. This strategy dynamically modulates encoder parameter updates by monitoring prediction confidence. We first calculate the prediction deviation Dm i for modality m to quantify the discrepancy between predictions and ground truth:

where Ii is the index of the true label for sample i, pmi represents the predicted probability for the target class, and Projm denotes the modality-specific projection layer. Next, we derive the adjustment weight W m i based on the relative deviation disparity across the batch:

where k iterates over the other modalities distinct from m, B represents the current training batch, and j is the sample index. To apply this, we expand W m i into a weight matrix ˆW m i ∈ Rdm×dm and modulate the gradient of the modality encoder parameters θm:

In this way, we can dynamically adjust the optimization focus at the instance level and the feature contribution at the modality level. This dual-level adjustment enables the model to fully leverage the unique advantages of heterogeneous data while preventing any single modality or simple sample pattern from dominating the learning process. This can enhance the joint understanding ability of emotion and intent.

3.5 Hierarchical Task-Bottleneck Interaction

Existing methods predominantly rely on single-round direct interactions, failing to account for the profound and iterative interplay between tasks. Consequently, they struggle to capture the intricate semantic dependencies between emotion and intent. Furthermore, traditional direct interactionmethods often allow noise to propagate between tasks. To address these limitations, inspired by [Zheng et al., 2025], we devise a Hierarchical Task-Bottleneck Interaction (HTBI) module. Specifically, we introduce a set of learnable "bottleneck tokens" acting as a communication hub to facilitate bidirectional information flow. Unlike simple direct interactions, the number of bottleneck tokens is progressively reduced at each layer. This design compresses and preserves critical interaction information and filters out irrelevant details layer-by-layer.

Specifically, as shown in Figure 3, the module consists of L iterative layers. In the l-th layer (l ∈ {1, . . . ,L}), we first determine the current bottleneck representation Bl. To achieve progressive information compression, the number of bottleneck tokens Kl is halved relative to the previous layer (i.e., Kl = ⌊Kl-1/2⌋). For the first layer, B0 is initialized with learnable parameters. For subsequent layers, we employ a Transformer layer to contextualize the previous bottleneck features Bl-1 and retain the top-Kl tokens:

where Transformer(·) denotes the standard encoder layer, and Truncate(·,Kl) represents the operation of slicing the first Kl tokens. This step acts as a semantic filter, producing a condensed summary of the task interaction history.

After obtaining the compressed bottleneck Bl, the interaction proceeds in two phases: aggregation and dissemination. We first formulate the standard Cross-Attention (CA) mechanism as: CA(Q, K, V ) = Softmax( QK⊤ √dk )V . In the aggregation phase, the bottleneck tokens Bl act as the Query, seeking complementary information from the task features. As illustrated in the MTCA block of Figure 3, we employ a parallel interaction strategy. Bl independently queries the emotion features hemo i,l-1 and intent features hint i,l-1 from the previous layer. The retrieved contexts are then summed to integrate cues from both tasks:

This aggregated information is fused into the bottleneck via a residual connection and normalization:

Through this parallel aggregation, the bottleneck hub effectively absorbs distinct yet complementary semantics from both emotion and intent domains without forcing them into a shared vector space prematurely.

Finally, in the dissemination phase, we utilize the refined bottleneck Bup l to update the original task features. Here, the task-specific features (emotion or intent) act as the Query, while the updated bottleneck Bupd l serves as the shared Key and Value. Each task retrieves enhanced interaction information from the common hub:

Through L rounds of such hierarchical operations, the emotion and intent features continuously interact and refine each other. The final features not only retain their specific semantics but also integrate the filtered key information from the counterpart task, simulating the dynamic mutual influence observed in real conversations.

3.6 Prediction and Optimization

Following L rounds of hierarchical bottleneck interaction, the final obtained features hemo i,L and hint i,L not only retain their specific semantics but also fully integrate interaction information through the bottleneck hub. Finally, we feed these enhanced features into their respective MLPs to generate the final probability distributions for emotion and intent:

By employ the Cross-Entropy loss between the predicted results ˆ y and the ground truth labels y, the total task loss Ltask is defined as the sum of the emotion prediction loss and the intent prediction loss:

where N denotes the number of training samples. The final total loss function Ltotal is formulated as follows:

4 Experiments

4.1 Experimental Setup

Implementation Details. For feature extraction, we obtain 768-dimensional representations for all modalities. Specifically, audio clips are resampled to 16 kHz via Librosa and encoded by a pre-trained wav2vec 2.0 model [Baevski et al., 2020]. Video frames are sampled at 1 Hz using FFmpeg and processed via the Video Swin Transformer [Liu et al., 2022]. Textual features are derived from BERT-base [Devlin et al., 2019]. During the training process, we select the Adam optimizer and a batch size of 32. We train the model for up to 100 epochs, implementing an early stopping strategy with a patience of 8. All experiments are conducted on NVIDIA A800 GPUs. Finally, we report the average results over five independent runs with different random seeds.

Datasets. We evaluate our proposed method on three datasets: MCEIU m (Mandarin) [Liu et al., 2024], MCEIU e (English) [Liu et al., 2024], and MINE [Yang et al., 2025]. Evaluation Metrics. We evaluate performance using Accuracy (ACC) and Weighted Average F1-score (WAF). For the multi-label intent task in the MINE dataset, we additionally report Micro F1 (M-F1) and Samples F1 (S-F1).

4.2 Performance Comparisons

We compare DELTA against representative baselines, including bc-LSTM [Poria et al., 2017], MISA [Hazarika et al., 2020], MFSA [Yang et al., 2022b], MMIN [Zhao et al., 2021], COGMEN [Joshi et al., 2022], JOYFUL [Li et al., 2023b], EmoInt-Trans [Singh et al., 2022], EI2 [Liu et al., 2024], and BEAR [Yang et al., 2025]. From Table 1, we can have the following observations.

First, DELTA consistently outperforms baselines across all datasets. Specifically, on MCEIU e, it achieves 52.71% and 57.02% accuracy for emotion and intent recognition. This is largely achieved by our DFD module, which employs orthogonal decomposition to rigorously decouple noise from essential semantics, overcoming the redundancy limitations of prior works.

Second, our model significantly outperforms methods that employ static fusion or treat modalities equally (e.g., MISA, MFSA). For instance, compared to MISA on MCEIU m, DELTA improves the intent accuracy by over 15%. The reason might be that these baselines ignore the dynamic variation in modality contribution and sample difficulty, allowing dominant modalities or simple samples to bias the learning process. Conversely, DELTA employs a Dual-Level Adaptive Adjustment mechanism. This enables the model to dynamically re-weight modalities based on context and calibrate gradients for hard samples, ensuring robust understanding.

Third, while the unified frameworks EI2 and BEAR also belong to the MC-EIU task, they still fall short compared to our method. For example, on MINE, DELTA improves the Intent S-F1 by 0.85% over BEAR. The reason might be that these methods primarily rely on shallow or direct interaction mechanisms, which fail to capture the deep iterative dependencies and risk propagating noise between tasks. Unlike these works, our HTBI module can act as a progressive communication hub, facilitating multi-round, noise-filtered bidirectional information flow, simulating the deep interplay between emotion and intent to achieve mutual enhancement.

4.3 Ablation Study

To thoroughly evaluate the effectiveness of the DELTA framework, we conduct ablation experiments on the MCEIU m dataset, covering task integration, modality importance, and component contributions. The results are presented in Table 2.

First, regarding task integration and modalities, the proposed joint learning framework significantly outperforms single-task baselines with an improvement of 3.68% in Emotion Accuracy, confirming the necessity of leveraging task interdependence. Among modalities, text contributes the primary semantics, but removing audio or visual cues causes a notable performance drop, validating that non-verbal signals are indispensable for robust understanding. Second, we analyze the contribution of each module. Removing the Disentangled Feature Denoising (DFD) module incurs an accuracy loss of approximately 1.5%, verifying its role in filtering redundancy. Similarly, the exclusion of the Modality Dynamic Weighting (MDW) and Instance-Guided Adjustment (IGA) mechanisms leads to distinct performance declines. For instance, the absence of MDW reduces the Emotion Accuracy to 57.89%, highlighting the importance of dynamically balancing modality weights. Finally, removing the Hierarchical Task-Bottleneck Interaction (HTBI) module causes the most significant impact among components, proving that our progressive bottleneck strategy is critical for capturing deep task interactions while preventing noise propagation.

4.4 Parameter andWeight Analysis

Sensitivity of Hyperparameters. We analyze the impact of the interaction depth L and the DFD balancing coefficients on model performance. As illustrated in the left panel of Figure 5 , accuracy improves steadily and peaks at L = 4, confirming that sufficient bottleneck iterations are essential for deep semantic refinement. However, a decline at L = 5 suggests that excessive depth may introduce redundancy and lead to overfitting. Furthermore, regarding the DFD module, we evaluate the loss weights αo, αr, αc, αt in Figure 4. Empirical results indicate the model is robust to their variations. Balanced coefficients are found to be crucial: they ensure the orthogonality constraint (Lorth) effectively filters noise without overwhelming the reconstruction (Lrec) and primary task optimization, thereby guaranteeing feature purity and training stability.

Dynamic Modality Weights. The evolution of weights assigned by the MDW mechanism across training epochs is visualized in the right panel of Figure 5. On one hand, the weights exhibit a stable distribution after the initial phase, validating that the model converges to a robust fusion strategy. On the other hand, the weight of the text modality is consistently higher than that of audio and video. This showcases the intrinsic dominance of textual semantics in conversation, while highlighting the necessity of MDW. It ensures non-verbal modalities are not ignored but assigned appropriate attention to provide complementary cues, achieving superior fusion compared to static methods.

5 Conclusion

In this paper, we propose a novel framework for MC-EIU called Disentangled Hierarchical Interaction and Adaptive Adjustment. We introduce a Disentangled Feature Denoising module based on orthogonal decomposition to purify heterogeneous representations, and design a Dual-Level Adaptive Adjustment mechanism to dynamically optimize learning focus from both modality and instance perspectives. Furthermore, we employ a Hierarchical Task-Bottleneck Interaction strategy to facilitate progressive, noise-filtered mutual reinforcement between tasks. Experiments demonstrate that DELTA consistently outperforms state-of-the-art methods on standard benchmarks. Future work will explore incorporating Large Language Models to further enhance the semantic reasoning capabilities of the framework.

Acknowledgments

This research was supported by the National Natural Science Foundation of China (Nos. 62271411, U22A2098), the Technological Innovation Team of Shaanxi Province (No. 2025RS-CXTD-009), the International cooperation project of Shaanxi Province (No. 2025GH-YBXM-017).