Multi-modal Anchor Gated Transformer with Knowledge Distillation for Emotion Recognition in Conversation

1 Introduction

Emotion plays a pivotal role in human communication, influencing not only the content but also the tone and context of interactions. Emotion Recognition in Conversation (ERC) aims to identify the emotional states expressed in each utterance within a dialogue. This task is essential for applications in areas such as conversational agents, healthcare systems, and recommendation engines. While emotions are traditionally expressed through text, they are also richly conveyed in the audio and visual modalities [Poria et al., 2017; Wu et al., 2025]. Figure 1 provides an illustrative example of multi-modal ERC, showcasing how information from various modalities can be integrated to improve emotion recognition.

∗Corresponding Author.

Recent research in ERC has primarily focused on individual modalities. Text-based models often leverage context modeling [Majumder et al., 2019; Song et al., 2022; Hu et al., 2023; Yang et al., 2024] or incorporate external knowledge [Zhong et al., 2019; Zhu et al., 2021; Lee and Lee, 2022; Wang et al., 2025]. Audio-based models make use of multi-task learning, attention mechanisms, and data augmentation strategies [Latif et al., 2023; He et al., 2025; Guo et al., 2025], while video-based models extract key frames to enhance emotion recognition [Wei et al., 2021; Poria et al., 2017]. However, relying solely on a single modality for emotion recognition can overlook crucial emotional cues embedded in other modalities, leading to suboptimal performance. This limitation has spurred increasing interest in multi-modal ERC. Existing multi-modal models typically extract frame-level features for each modality, align these features across modalities, and fuse them for emotion classification [Tsai et al., 2019; Guo et al., 2022; Zheng et al., 2023]. While effective, these approaches often treat all modalities equally, disregarding the varying significance of each modality in emotion recognition. Furthermore, the complex alignment process increases the computational burden, making these models less suitable for deployment in resource-constrained environments. To address these challenges, a more efficient and adaptive approach to modality representation and integration is needed.

Prompt-based learning has recently gained attention for its success in both natural language processing (NLP) and multimodal tasks. In ERC tasks focusing on textual data, welldesigned prompts can effectively guide models to extract relevant contextual information, thereby improving the quality of utterance-level features [Song et al., 2022; Son et al., 2022; Yun et al., 2024]. Additionally, knowledge distillation techniques have been widely explored to enhance the performance of student models by enabling them to mimic better the representations learned by teacher models [Lin et al., 2022; Li et al., 2023; Yun et al., 2024; Ma et al., 2024]. To overcome the challenges in multi-modal ERC, we introduce the Multi-modal Anchor Gated Transformer with Knowledge Distillation (MAGTKD), a framework designed to improve the integration of multi-modal information for emotion recognition tasks. Specifically, MAGTKD leverages context-aware prompts to extract high-quality utterance-level textual representations. These robust textual features are then used within a knowledge distillation framework to enhance the representation capacity of weaker modalities (e.g., audio and video), ultimately improving the fusion of multi-modal features for emotion recognition. In contrast to existing methods, which rely on frame-level feature interactions before fusion-thereby increasing computational complexity [Tsai et al., 2019; Guo et al., 2022; Zheng et al., 2023]-MAGTKD directly fuses utterance-level features post-interaction, addressing the computational burden while maintaining or even improving performance.

We evaluate MAGTKD on two widely-used benchmark datasets, IEMOCAP and MELD. Experimental results demonstrate that MAGTKD achieves state-of-the-art performance on both datasets, surpassing existing methods in both accuracy and efficiency.

The key contributions of this work are as follows:

• We propose MAGTKD, a novel framework for ERC that effectively integrates multi-modal features, taking into account the varying contributions of different modalities to emotion classification.
• MAGTKD significantly reduces model complexity compared to traditional frame-level feature fusion methods.
• MAGTKD sets new benchmarks for ERC, achieving superior performance on the IEMOCAP and MELD datasets.

2 RelatedWorks

2.1 Prompt Learning

Prompt Learning has emerged as an effective approach for leveraging pre-trained models by designing task-specific prompts to fine-tune and integrate them for downstream tasks, enabling improved modality representations. It has been widely adopted across various NLP tasks [Gao et al., 2021; Heinzerling and Inui, 2021; Xu et al., 2023]. Recently, researchers have begun exploring the application of prompt learning in multi-modal settings [Tsimpoukelli et al., 2021; Khattak et al., 2023; Zhu et al., 2023]. [Tsimpoukelli et al., 2021] presents a simple, yet effective, approach for transferring this few-shot learning ability to a multi-modal setting (vision and language). [Khattak et al., 2023] proposes Multi-modal Prompt Learning (MaPLe) for both vision and language branches to improve alignment between the vision and language representations. [Zhu et al., 2023] develop Visual Prompt multi-modal Tracking (ViPT), which learns the modal-relevant prompts to adapt the frozen pre-trained foundation model to various downstream multi-modal tracking tasks. With success in diverse NLP and multi-modal learning applications, we extend prompt learning to the emotion recognition task, aiming to harness its potential for enhancing emotional feature extraction and representation.

2.2 Konwledge Distillation

Knowledge Distillation (KD) aims to transfer knowledge from a large teacher network to a smaller student network. This knowledge transfer typically occurs at three levels: soft labels of the final layer [Hinton et al., 2015], intermediatelayer features [Romero et al., 2015], and the relationships between features across layers [Yim et al., 2017]. Based on the learning strategy, KD can be categorized into offline [Passalis and Tefas, 2018; Li et al., 2020] and online [Zhang et al., 2018; Chung et al., 2020] distillation. In offline distillation, the teacher model is pre-trained to guide the student model's learning. In contrast, online distillation involves simultaneous training of the teacher and student models with joint parameter updates. KD has demonstrated its effectiveness in transferring knowledge across modalities in multi-modal research [Albanie et al., 2018]. Motivated by this, we adapt KD techniques to the multi-modal ERC task, enabling efficient knowledge transfer between modalities to enhance emotion recognition performance.

2.3 Modal Fusion

In the domain of modality fusion, existing works predominantly focus on extracting frame-level features and performing feature interactions at this granularity. [Tsai et al., 2019] introduces the Multi-modal Transformer (MulT) to generically address the above issues in an end-to-end manner without explicitly aligning the data. [Zheng et al., 2023] extracts three modal frame-level features and uses an attention mechanism to perform alignment operations on the three modal features. However, frame-level feature alignment often introduces significant computational complexity. Unlike these approaches, our work adopts utterance-level feature extraction and designs a novel model for multi-modal feature fusion, effectively reducing complexity while maintaining strong performance in emotion recognition tasks.

3 Methods

To enhance the representation of each modality and achieve effective multi-modal fusion, we propose the MAGTKD model for the ERC task. Figure 2 illustrates the overall architecture of the proposed framework, with detailed descriptions provided in the following subsections.

3.1 Task Definition

Given a set of speakers S, utterances U, and emotion labels Y , a conversation consisting of k utterances is represented as [si, u1, ym, sj , u2, yn, ..., si, uk, ym], where si, sj ∈ S are speakers and ym, yn ∈ Y are one of the predefined emotion categories. If i = j, si and sj represent the same speaker. Moreover, uk ∈ U represents the k-th utterance. Each utterance uk = {tk, ak, vk} contains three modalities, where t, a, v represent text, audio and video, respectively. The goal of ERC is to predict to which emotion label ym the utterance uk belongs.

3.2 Feature Extraction

Figure 2 illustrates the extraction of modality-specific features using dedicated encoders for each input modality. In this section, we explain the feature generation process in detail:

Text: Following prior work [Lee and Lee, 2022; Song et al., 2022], we utilize prompt learning to model both the context and speaker information. For the text encoder, we adopt RoBERTa [Liu et al., 2019]. We construct the contextual representation Ck and the prompt Pk as follows.

where < mask > represents a special token, Ftk ∈ R1×d is embedding of < mask >, representing the aggregated emotion feature, and d is the hidden dimension of the < mask > token.

Audio: Self-supervised learning has achieved remarkable success not only in natural language processing but also in audio and video domains [Baevski et al., 2020; Baevski et al., 2022]. For the audio encoder, we employ Date2vec [Baevski et al., 2022], with the audio segment ak of the k-th utterance as input. The process of extracting audio features is formalized as follows.

where Fak ∈ R1×d is the embedding of ak, and d is the hidden dimension of audio features.

Video: Similar to the audio feature extraction process, we utilize Timesformer [Bertasius et al., 2021] as the video encoder. The process of extracting video features is formalized as follows.

where vk is the video input, and Fvk ∈ R1×d is the embedding of vk, and d is the hidden dimension of video features.

3.3 Knowledge Distillation

Unlike traditional knowledge distillation methods that utilize KL divergence, the multi-modal ERC task involves crossmodal knowledge distillation. We adopt a collaborative distillation strategy based on soft labels and intermediate-layer features, using Pearson correlation coefficients as our crossmodal measurement approach.

where p(u, v) is the Pearson correlation coefficient between two logit vectors u and v.

Soft Label-Based Distillation leverages the soft label outputs from the last layer of each modality encoder to compute the knowledge divergence across modalities at both the sample and feature levels. Pearson correlation coefficients are employed to measure the degree of knowledge disparity between different modalities. By reducing this disparity, the textual features transfer knowledge to the audio features. The process is formalized as:

where B is a training batch, C is the emotion categories, P t , P a ∈ RB×C are the prediction matrix of text and audio modality, respectively. τ is a temperature parameter to control the softness of logits.

Intermediate-Layer Feature-Based Distillation computes similarity matrices within a batch for the textual modality as the target matrix (via dot product between text modality features and their transpose). Similarly, a source matrix is computed for the audio and text modalities. Using the softmax function, we derive the target and source distributions. The KL divergence between these distributions is minimized to enable the transfer of knowledge from textual features to audio features. The process is defined as:

where Fi,j , F ′ i,j ∈ RB×B are the text-modal similarity matrix and the text-audio modal similarity matrix, respectively. Ti, Si are target and source distributions.

The overall loss function includes the above two losses and the cross-entropy loss:

where yi is true labels and pi is predict labels.

3.4 Multi-modal Anchor Gated Transformer

In the first stage, we utilize prompt learning and knowledge distillation to extract utterance-level features for each modality. However, directly concatenating these features for emotion recognition, in figure 3, results in degraded model performance. To address this issue, we propose a second stage that employs a Multi-modal Anchor Gated Transformer (MAGT) to effectively integrate features across the three modalities. Specifically, each modality serves as an anchor to aggregate complementary information from other modalities. Specifically, we first use the audio and video features as anchors to aggregate information from the other modalities. Given the strong performance of the text modality, the raw text features are used as an anchor to aggregate the audio and video features enriched by other modalities.

We construct the dataset at the conversation level, where utterance-level features from different modalities are arranged sequentially based on temporal order. For each utterance, speaker and positional embeddings are added. Speaker embedding uniquely maps each speaker in the dataset to a sequence ID, which is then embedded using an embedding layer.

where PE is positional embedding and SE is speaker embedding. suk represents speaker of utterance uk. Fm = (Fm1 , Fm2 , . . . , Fmk ) and m ∈ {t, a, v}. Hm is the positional- and speaker-aware utterance sequence representation for m modality.

Given the remarkable success of Transformers in NLP, the effectiveness of gating mechanisms for preserving salient features, and the superior performance of the text modality compared to audio and video in ERC tasks, we introduce an Anchor-Gated Transformer. This structure leverages a Transformer encoder with three inputs (Query Q ∈ Rlq×dk , Key K ∈ Rlk×dk , and Value V ∈ Rlk×dk ) to integrate features across modalities.

where m ∈ {t, a, v} and n ∈ {t, a, v} - {m}. Hm→m represents anchor modality aggregating its own information. Hn→m represents an anchor modality aggregating information from other modalities.

To enhance the emotional representation of each modality, we incorporate a gating mechanism to filter out irrelevant information and retain the most effective emotional features.

where Wn→m is a weight matrix, bn→m is a bias parameter, αn→m represents gate, and ⊗ is the element-wise product.

3.5 Emotion Classifier

The emotion classification task is performed using a linear layer. The features t′ , extracted from the Multi-modal Attention and Graph-based Transformer (MAGT), are transformed into the emotion label pi corresponding to each utterance ui.

The classification process can be formalized as follows:

where W ∈ RC×d and b ∈ RC are the weight matrix and bias vector of the linear layer, respectively. Here, C denotes the number of emotion classes, and d represents the dimension of the feature vector t′ . The softmax function ensures that the outputs are normalized probabilities across all emotion classes, and argmax selects the class with the highest probability as the predicted label pi.

3.6 Training

In the first stage, the utterance-level feature extraction is optimized using the loss function defined in Equation 14. In the second stage, the multi-modal fusion is performed using the knowledge distillation (KD) loss functions defined in Equations 9 and 12. These can be formalized as:

where i refers to either the audio or video modality.

The total loss function in the second stage is the sum of the cross-entropy loss and the two distillation loss functions, weighted by their respective coefficients. This can be expressed as:

where α and β are the coefficients for the distillation losses of the audio and video modalities, respectively.

4 Experiments

4.1 Datasets

In this section, we introduce two widely adopted benchmark datasets: MELD and IEMOCAP. Following descriptions for specific details of these two datasets:

MELD [Poria et al., 2019] is a multiparty conversation dataset containing over 1,400 dialogues and more than 13,000 utterances extracted from the TV show "Friends." This dataset includes seven emotion categories: neutral, surprise, fear, sadness, joy, disgust, and anger.

IEMOCAP [Busso et al., 2008] consists of 7,433 utterances and 151 dialogues, divided into five sessions, each involving two speakers. Each utterance is labeled with one of six emotion categories: happiness, sadness, anger, excitement, frustration, and neutral. The training and development datasets are randomly split from the first four sessions in a 9:1 ratio. The test dataset comprises the last session.

4.2 Experimental Setup

We evaluate the performance of our model on two datasets using Accuracy (Acc) and Weighted F1-score (W F1) as metrics. We designed a two-stage experimental process. The first stage focuses on extracting utterance-level features from different modalities, while the second stage involves multimodal feature fusion using a dataset constructed at the conversation level. The hyperparameter settings are shown in Table 1. All experiments are conducted on a single NVIDIA GeForce RTX 2080 Ti GPU.

4.3 Baselines

We compare our proposed model, MAGTKD, against classic baselines, including DialogueRNN [Majumder et al., 2019], DialogueGCN [Ghosal et al., 2019], MMGCN [Hu et al., 2021b], and DialogueCRN [Hu et al., 2021a], as well as stateof-the-art models such as A-DMN [Xing et al., 2022], DialogueINAB [Ding et al., 2023], SACCMA [Guo et al., 2024], Ada2I [Nguyen et al., 2024], and GraphCPC [Li et al., 2024].

4.4 Comparative Experiments

Table 2 compares our model with prior works on IEMOCAP and MELD. Our model achieves the best performance on both datasets, setting new state-of-the-art (SOTA) results. On IEMOCAP, we achieve 69.38% accuracy and 69.59% weighted F1 (W F1), outperforming GraphCFC by 0.99% and 1.56%, respectively. On MELD, our model achieves 66.36% accuracy and 65.32% W F1, with improvements of 5.17% and 6.83% over Ada2I, the previous SOTA. These results demonstrate our model's ability to effectively integrate multi-modal features and handle challenges in conversational emotion recognition through prompt learning, knowledge distillation, and advanced fusion techniques.

4.5 Visualization and Analysis

Figure 4 shows t-SNE visualizations of the feature representations from the IEMOCAP and MELD datasets. We visualize single-modal features (text, audio, and video) as well as features enhanced by knowledge distillation, where the text modality guides the audio and video modalities.

The visualizations indicate that the text modality has the strongest discriminative power across both datasets, followed by audio, while video shows the least discriminative ability. After applying knowledge distillation, the audio modality improves significantly, benefiting from the text modality's guidance. However, the video modality struggles to learn effectively, highlighting the challenge of transferring knowledge to modalities with weaker feature representations.

Furthermore, the audio modality in IEMOCAP shows stronger discriminative power compared to MELD, which results in better performance when learning from the text modality. In the next sections, we will provide quantitative results to further explore this trend.

4.6 Ablation Studies

Table 3 shows the results of ablation studies on feature extraction and multi-modal fusion. We evaluate individual modalities and the effects of knowledge distillation from the text modality to audio and video modalities.

The results highlight that knowledge distillation improves the performance of the audio modality but has little effect on the video modality. This aligns with the observations in Figure 4, where the video modality struggles to learn effectively from the text modality due to its weaker feature representation. On the other hand, the audio modality benefits more from the distillation process, particularly in the IEMOCAP dataset where it has a stronger feature representation compared to MELD.

For multi-modal fusion, we first tested the Concat method. The best performance was achieved by fusing the strongest modality features (text, distilled audio, and undistilled video). However, adding the distilled video modality degraded performance. We then tested the MAGT fusion method, which also performed best when fusing the strongest modalities. Notably, MAGT maintained stable performance even when the weakest modality, the distilled video, was included.

These results demonstrate that MAGT effectively integrates emotional cues from different modalities, even when some modalities contribute less useful information.

4.7 Complexity Analysis

We define the feature dimensions of the prior frame-level model as (S,L,D), and for the proposed utterance-level model as (C, U, D), where:

• S: number of samples,
• L: frame sequence length,
• D: hidden feature dimension,
• C: number of conversations,
• U: number of utterances per conversation.

Assuming C · U = S and L is consistent across modalities.

Spatial Complexity The frame-level model has spatial complexity:

while the utterance-level model has:

which reduces by a factor of L since C · U = S.

Temporal Complexity The frame-level model's temporal complexity is:

whereas the utterance-level model's complexity is:

Thus, the relative temporal complexity is:

For shorter dialogues (U ≪ L), the proposed model has a significant reduction in complexity. Additionally, spatial complexity is reduced by a factor of L. In summary, the proposed model is more efficient, with linear spatial and quadratic temporal complexity in U, compared to the frame-level model's quadratic temporal complexity in L.

4.8 Hyper-parametric Analysis

Figure 5 shows the effect of varying the hyperparameters α (audio distillation coefficient) and β (video distillation coefficient) on model performance for IEMOCAP and MELD. For IEMOCAP, changing α significantly affects performance when β is fixed, while adjusting β with a fixed α results in smaller variations. This suggests that the audio modality better benefits from knowledge distillation, whereas the video modality shows weaker learning. This aligns with the observations in Figure 4, where the audio modality has better feature discriminability than the video modality. In MELD, setting α = 0 and increasing β initially improves performance, but further increases lead to a decline. Similarly, setting β = 0 and varying α shows an initial performance boost followed by a decrease, confirming that the KD loss improves model performance but requires careful tuning.

4.9 Convergence Analysis

Figure 6 presents the convergence behavior of the model on IEMOCAP and MELD datasets. For IEMOCAP (a)-(c), as the number of epochs increases, both the total loss and the KD loss for audio and video modalities converge. The KD loss for the audio modality converges steadily, while the video modality experiences fluctuations before stabilizing, which is consistent with the lower discriminability of video features, as shown in Figure 4. Similar patterns are observed for the validation and test sets. For MELD (d)-(f), the trends are similar, with both the total loss and the KD losses for audio and video modalities converging as epochs increase. These results confirm the effectiveness of our model and the positive impact of knowledge distillation on training stability and performance.

5 Conclusion

The proposed MAGTKD model effectively addresses the challenges of multi-modal ERC by leveraging prompt learning to extract robust textual representations and employing knowledge distillation to enhance weaker modalities. The subsequent use of MAGT enables efficient aggregation of emotional information across modalities, resulting in stateof-the-art performance on both the MELD and IEMOCAP datasets. Future work will explore extending MAGTKD to handle more complex multi-modal scenarios, such as incorporating dynamic contextual information in real-time conversations or addressing challenges posed by highly imbalanced datasets.

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Grant no.62276265 and 62406326).