MATCH: Modality-Calibrated Hypergraph Fusion Network for Conversational Emotion Recognition

1 Introduction

Emotion recognition in conversations (ERC) aims to detect emotional states from conversational signals, providing crucial emotional cues for downstream tasks. ERC has attracted significant attention in fields such as social recommendation [Zhang et al., 2024], fake news detection [Mittal et al., 2020], and dialogue systems [Bertero et al., 2016]. Early ERC approaches primarily focused on text-based inputs, utilizing techniques like recurrent neural networks (RNNs) [Majumder et al., 2019], Transformers [Lian et al., 2021], and graph neural networks (GNNs) [Ghosal et al., 2019] to extract emotional features from text, achieving some success. Recently, with the increased availability of multimodal data, research has shifted towards multimodal emotion recognition in conversations (MERC), aiming to enhance emotional understanding by integrating and analyzing information across multiple modalities.

∗Corresponding Author (mingli@zjnu.edu.cn)

Existing MERC approaches can be broadly classified into aggregation-based methods and graph-based methods. Aggregation-based methods combine modality information using techniques such as concatenation, attention mechanisms, and tensor fusion to perform emotion prediction [Zadeh et al., 2017; Zadeh et al., 2018]. In contrast, graphbased methods capture modality interactions and contextual dependencies in conversations through node-level propagation and diverse edge designs [Ghosal et al., 2019; Hu et al., 2022; Shi et al., 2025]. While effective, graph-based methods struggle to model complex multivariate dependencies between utterances, leading to the development of hypergraphbased approaches. For instance, M3Net [Chen et al., 2023] captures the multivariate and multi-frequency characteristics of multimodal features by integrating hypergraph and frequency domain decomposition on graph, while HAUCL [Yi et al., 2024] uses a hypergraph autoencoder to learn adaptive hyperedge connectivity patterns that are relevant for emotion prediction.

Despite these advancements, there exist two key challenges: (i) Insufficient calibration of modality features: Raw modality features often encompass intricate and multifaceted information [Li et al., 2023a]; however, existing approaches either overlook deeper exploration of these features or simply categorize them into intra-modal and inter-modal information. This not only limits the utilization of modal features but may also result in the misinterpretation of emotional cues during prediction due to imprecise delineation of these feature types. (ii) Inadequate learning of conversational patterns: Real-world conversations exhibit inherent structural patterns across utterances. For instance, a person expressing happiness may simultaneously smile and use a high-pitched voice, which are correlated features that may appear in utterances across different speakers, influencing emotional states. Such conversational patterns reflect the dynamic pathways of emotion shifts across context and modalities, which are critical for MERC tasks. However, existing approaches struggle to effectively capture and represent these complex pathways, resulting in superficial emotional understanding.

As illustrated in Figure 1, in utterance #4, Phoebe says "We did have fun, didn't we?", accompanied by a smile and a high-pitched tone, which influences Rachel, who also smiles and uses a high-pitched tone. This expression of joy is difficult to identify from the text alone, as there is no direct contextual connection between the two. Similarly, in utterances #10 and #11, Phoebe's intonation and expression affect Rachel. Thus, the emotional state of utterance #11 shifts to neutral, rather than maintaining the previously expressed joy. These nuances of emotion across utterance, modality, and speaker require a deeper exploration and calibration of modality features to prevent interference from erroneous contextual and modality information. More importantly, they highlight emotional pathways between utterances, enhancing a deeper understanding while minimizing the impact of unnecessary contextual or modality information, emphasizing the need to explore conversational patterns.

To address these challenges, we propose a modalitycalibrated hypergraph fusion network, named MATCH. MATCH comprises two key components: conversational entity calibration and emotion-aligned hypergraph fusion. The entity calibration strategy focuses on the critical entities in ERC tasks, i.e., utterance and speaker, and performs a fine-grained calibration at the modality and context levels. This strategy yields a more refined multimodal representation compared to simply distinguishing between intra- and intermodal information. Subsequently, the calibrated features are used to construct a hypergraph that captures high-order semantic relationships. Moreover, we construct a line graph to extract conversational patterns that are challenging to be represented by hypergraphs. Together, these components enable MATCH to perceive both surface-level semantic and deeper emotional pathways, facilitating the generation of comprehensive emotional representations. In summary, our contributions are as follows:

• We propose MATCH, a hypergraph-based MERC model that delivers comprehensive emotional understanding by conversational entity calibration and emotion-aligned hypergraph fusion.
• We design a fine-grained conversational entity calibration strategy that enhances the utilization of multimodal features by calibrating utterance and speaker knowledge at both the modality and context levels.
• We propose emotion-aligned learning, which maximizes the role of hyperedges in MERC by hyperedge- and graph-level alignment. This facilitates the learning of surface semantics and deep emotional pathways, enhancing the emotional understanding.

2 RelatedWork

2.1 Multimodal Fusion

Multimodal fusion aims to produce a more comprehensive representation by integrating multimodal information through early fusion, decision fusion, and hybrid fusion strategies [Zhao et al., 2024]. Early fusion combines modalities into a joint representation [Mai et al., 2020], while decision fusion aggregates predictions from individual modalities using weighted summation or expert voting. Hybrid fusion blends the advantages of both approaches, offering greater flexibility [Duan et al., 2024; Tellamekala et al., 2023; Li et al., 2025d]. Recent advances in graph deep learning have enhanced modality interaction capture in MERC. However, a common issue in both graph-based and non-graphbased methods is the lack of modality calibration, as simply extracting inter- or intra-modal information is often insufficient for producing appropriate results [Joshi et al., 2022; Li et al., 2023b; Li et al., 2023a]. This challenge aligns with real-world conversations, where emotional states conveyed by an utterance are not solely expressed through modality information.

2.2 Hypergraphs in Emotion Recognition

Hypergraphs provide a powerful framework for modeling high-order interactions among multiple entities, going beyond the pairwise correlations captured by traditional graphs [Mill´an et al., 2025]. This flexibility makes them especially effective for representing complex relationships in a wide range of real-world datasets [Feng et al., 2019; Ju et al., 2024; Li et al., 2025c; Li et al., 2025b; Li et al., 2024;

Li et al., 2025a]. Recent studies have leveraged this capability to MERC tasks, aiming to capture high-order information through a multi-node connectivity paradigm [Chen et al., 2023; Yi et al., 2024]. However, an important challenge is that hyperedges, as part of the hypergraph, reflect its inherently valuable attributes [Wang et al., 2024; Chen et al., 2024]. Existing research has not explored how to leverage hyperedge information to enrich emotion representations [Lu et al., 2024], limiting the potential of hypergraphs for MERC tasks. Meanwhile, the issue of information imbalance between modalities persists, causing some hyperedges to propagate weak semantic knowledge, resulting in significant bias in emotional prediction.

3 Methodology

In this section, we provide preliminaries and a detailed introduction to each component of the proposed MATCH, as depicted in Figure 2.

3.1 Task Definition

Let C = [c1, c2, · · · , cN] denote a set of N conversations and U = [u1, u2, · · · , uM] represent a set of M utterances, respectively. Each conversation ci consists ofMutterances and involves Q ≥ 2 speakers. An utterance ui, is represented as a triplet ui = {uai , uvi , uti } , where uai ∈ Rda , uvi ∈ Rdv and uti ∈ Rdt represent the acoustic, visual, and textual features of ui, respectively. MERC aims to predict the emotional label ˆ yi for each utterance uk based on its corresponding triplet representation.

3.2 Preliminaries

Definition 1. Graph follows the paradigm of pairwise node connections. A traditional graph (or in short, a graph) can be defined as: G = (V, E) where V and E denotes a set of vertices and edges, respectively. Xg ∈ R|V|×dh is the feature matrix, dh is the dimension of features, the adjacency matrix of G is Λg ∈ R|V|×|V| .

Definition 2. Hypergraph extends beyond the pairwise connection paradigm, offering a high-order representation structure. A hypergraph can be defined as: Gh = (V, Eh,W) where V is a vertices set initialized with feature Xh ∈ R|V|×dh and Eh is a hyperedges set which contains multiple vertices {v1, · · · , vn}. Λ ∈ R|V|×|Eh| is the corresponding incidence matrix.

Definition 3. Line Graph is derived from hypergraph Gh to capture the structured information within Gh. A line graph can be defined as: Gl = (Vl, El), where each vertex vi ∈ Vl is a vertex-hyperedge pair {(v, e) | v ∈ e, v ∈ Vh, e ∈ Eh} from Gh. Edge set El and adjacency matrix Λl ∈ {0, 1} is defined by the relation with Λl(vl, vl) = 1 if either v = v′ or e = e′ for vl = (v, e), vl = (v′ , e′ ) ∈ Vl.

3.3 Utterance Encoding

For visual and acoustic features, we use two separate fully connected layers to obtain their respective representations.

where cζi denotes the representation for utterance uζi . θζ FC are learnable parameters. For textual features, a bidirectional GRU is employed to enhance contextual coherence and obtain the corresponding representations:

where hti is hidden state of the i-th utterance.

3.4 Conversational Entity Calibration

For MERC, entity information in a conversation carries varying levels of emotional cues. Unlike previous work that defines all possible relationships in a dialogue as entities (e.g., between speakers or within the same speaker), we categorize entity information into two types: (i) utterance information and (ii) speaker information. After utterance encoding, we obtain the contextual information cξi corresponding to each utterance. For speaker information, we also utilize a bidirectional GRU to extract it.

where sξi denotes the speaker information corresponding to uξi and ˆhξi is the hidden state.

Previous work has focused solely on decoupling utterance information [Li et al., 2023a], which proves insufficient in scenarios where the emotional states of speakers vary dynamically. To address this, we calibrate both two conversational entities, thereby constructing a more precise emotional learning space. Given the inherent differences in information density across modalities, we first leverage text to augment the acoustic and visual features. Take speaker representations sξi for instance:

where ˆ svi denotes the enhanced visual representations. sim(★,★) is the cosine similarity function and τ1 is a temperature parameter. The enhanced acoustic representations ˆ sai can be captured in the same manner.

The speaker information often overlaps across both contextual and modality levels, which can obscure clear emotional cues. Therefore, we calibrate speaker information at both levels to ensure a more accurate and coherent emotional representation. Specifically, we enhance recognition performance by adjusting the distance between semantically similar representations in the semantic space through contrastive learning. First, we add Gaussian noise as an effective way to mitigate the inevitable loss of information noted in [Wang et al., 2023] and re-normalize the features when projecting modality data into the semantic space:

where noise θξ , ξ ∈ {a, v, t} is sampled from zero-mean gaussian distribution. Then, the modality- and contextuallevel calibration losses can be defined as:

where ˜ sm,ξ i = Tm(˜ sξi ) and ˜ sc,ξ i = Tc(˜ sξi ). T (★) denotes the projector, τm, τc are temperature parameters. im+, ic+ denote the positive modality- and context-level list for i-th sample. In this manner, the modality and contextual cues are distributed across different locations in the representation space, making it easier to calibrate the deeper semantics conveyed by each speaker. Combining Equation. (4)-(7), we can obtain the calibrated representations as:

We apply the same process to the utterance information cξi to obtain the calibrated utterance representations cξi . The final calibrated representation for the i-th utterance is:

where η is a hyperparameter to control the weight of calibrated speaker information. The overall calibration loss is:

3.5 Emotional-aligned Hypergraph Fusion

Graph structures are effective in ERC tasks due to their ability to model non-Euclidean data. However, traditional graph neural networks have an inherent limitation, i.e., message propagation occurs point-to-point, which leads to a loss of context during the learning process. In contrast, hypergraphs overcome this by forming hyperedges that connect multiple nodes, enabling the capture of multivariate conversational relationships that are difficult to model with traditional graphs.

Existing hypergraph-based approaches face a significant challenge: the underutilization of hyperedge representations, leading to shallow and incomplete emotion understanding. These approaches primarily focus on node-level representations, while node information is eventually aggregated into hyperedge representations and propagated back to the nodes.

The weak semantic nature of the hyperedges dilutes their impact on individual nodes, causing semantic imbalance. For instance, nodes connected to textual modality hyperedges typically exhibit stronger semantics than those linked to visual or acoustic hyperedges.

To this end, we introduce the line expansion [Yang et al., 2022] to unleash the power of hypergraph for MERC. First, we categorize emotional features into two main types: (i) semantic information learned by hypergraph, and (ii) conversational pattern learned by line graph. These two types of information complement each other, revealing emotional pathways between utterances and enabling the generation of a more comprehensive understanding of emotion. Additionally, we introduce emotion-aligned learning to ensure consistent understanding across these two perspectives.

Hypergraph Semantic Learning. We begin by constructing conversational hypergraph Gh = (V, Eh) to capture highorder semantic nuances across utterances. Following [Chen et al., 2023], we define two types of hyperedges (e1, e2) ∈ Eh: one connecting nodes within the same context and the other connecting nodes within the same modality. Each item in triplets of utterances {uai , uvi , uti } is treated as nodes v ∈ V, | V |= 3 ×M, which are initialized with calibrated representations xξi . Subsequently, we have the corresponding hypergraph incidence matrix Λ ∈ R|V|×|E| . Based on this, we apply a hypergraph convolutional network to capture the fundamental semantic representations shared between utterances. Formally:

where Dv and De denote the node degree matrix and hyperedge degree matrix, respectively. σ represents a nonlinear activation function. W is a learnable matrix. Considering that the low density of semantics in non-textual modalities may be further amplified during the propagation, we adopt an emotion-aligned learning strategy to alleviate this issue, which will be presented in the following sections.

Conversational Pattern Learning. Although multimodal representations learned through hypergraphs can capture emotion states to some extent, meaningful emotion consensus can also emerge between unpaired utterances in complex dialogues through cross-modal information, i.e., the pathways of emotion transfer within the context. These emotion pathways are reflected in the information flow across hyperedges, while hypergraph is more sensitive to node-level details. To address this, we introduce line graphs to further learn the conversational pattern. First, we construct the line graph adjacency matrix ˆΛ ∈ R|R|×|R| based on Λ ∈ R|V|×|E| , where R = 6 ×M. Subsequently, we transform the hypergraph node embedding X ∈ R|V|×dh into the line graph feature vector through a node projector Pv to explore the conversational pattern:

Let ˆX be the line graph feature at the first layer, that is, ˆX(0) = ˆX, the following convolution on line graph is:

where ˜Λ = ˆΛ + 2I, Wle is a learnable matrix.

Let ˆX(K) denote the node representations obtained from the line graph learning module. To preserve semantic consistency, we apply the back-projector P ′v , which re-projects the line node representations into the hypergraph space. This reprojection is performed based on the reciprocal of the edge degree, formally expressed as:

In this way, the structural information embedded in the hypergraph is re-expressed as vertices in the line graph through the convolution and back-projection processes. This enables the model to capture and understand the deeper, hidden conversational patterns.

Emotional-aligned Learning. Hypergraphs and line graphs provide complementary perspectives for emotion understanding. However, weak semantic hyperedges reduce their ability to effectively convey emotion. To address this, we propose emotion-aligned learning that enhances the effectiveness of semantic transfer by optimizing the distribution discrepancy through hyperedge- and graph-level alignment. Hyperedge-level alignment optimizes the gap between weak and strong semantic hyperedges, enhancing the representational capacity of acoustic and visual features through the incorporation of textual features. Based on Equation. (11), the hyperedge representations can be defined as follows:

where B denotes the hyperedge representations, let Bξ denotes the representations of three modalities. The corresponding distributions are:

where τemo is a temperature parameter.

We introduce the Kullback-Leibler (KL) divergence loss to minimize the distribution gap, formally:

In this manner, the hyperedges Bζ enrich their representations by iterating through the strong semantic representation, preventing the propagation of invalid information.

Note that, both line graph and hypergraph provide meaningful emotional representations, and the graph-level alignment aims to enhance knowledge transfer between line graph and hypergraph, formally:

where X and X are the distributions derived from Equation (16) corresponding to X and X.

3.6 Training Objective

Finally, we apply a fully connected layer to obtain the prediction labels ˆ yi. Formally:

The main ERC task loss can be defined as:

where η2 is a hyperparameter. The overall loss is defined as:

where γ1, γ2, γ3 are hyperparameters. Their sensitivity is studied in Section 4.6.

4 Experiments

4.1 Datasets

We evaluate the MATCH on two benchmark datasets, IEMOCAP and MELD. The detailed statistics are shown in Table 1.

IEMOCAP [Busso et al., 2008] contains video data from dyadic conversations with ten speakers, with utterances classified into six emotion categories: Happy, Sad, Neutral, Angry, Excited, Frustrated. Following [Hu et al., 2021], we use the first four sessions for training, the last for testing, and randomly select 10% of the training set for validation.

MELD [Poria et al., 2019] consists of video data from multi-party conversations in TV show "Friends", with utterances classified into seven emotion categories, i.e., Neutral, Surprise, Fear, Sadness, Joy, Disgust, Anger. We use the predefined splits for training and evaluation.

4.2 Baselines

We compare our proposed MATCH with ten baseline models including aggregation-based methods like DiaRNN [Majumder et al., 2019] CTNet [Lian et al., 2021] CauAIN [Zhao et al., 2022] CMERC [Tu et al., 2024b], graph-based methods like DiaGCN [Ghosal et al., 2019] MM-DFN [Hu et al., 2022] AdaIGN [Tu et al., 2024a] PCGNet [Tu et al., 2024c], and hypergraph-based methods like M3Net [Chen et al., 2023] HAUCL [Yi et al., 2024].

4.3 Implementation Details

All experiments were conducted on an NVIDIA RTX A6000 GPU using the torch-geometric package. We used a batch size of 16 for both datasets. For IEMOCAP, the learning rate was set to 1e-4 with a dropout rate of 0.4, while for MELD, the learning rate was set to 5e-4 with a dropout rate of 0.3. Additional parameter settings are provided in Table 2.

4.4 Results and Discussion

As shown in Table 3, MATCH outperforms existing approaches on both datasets. Specifically, ACC improves by 0.92% and W-F1 increases by 0.57% on IEMOCAP. Similarly, on MELD dataset, ACC improves by 0.31%, and WF1 improves by 0.02%. Our method also achieves competitive F1 scores across most emotion categories, with notably optimal performance on "Neutral" and "Excited," surpassing state-of-the-art results.

An obvious drawback of aggregation-based approaches is the high coupling of contextual information across multiple time steps. While CTNet addresses this by introducing different GRU units and employing an attention mechanism to enhance discriminability, it still struggles to capture multiple complex conversational relationships. In contrast, MATCH improves upon these methods by calibrating entity information at both the contextual and modality levels, enhancing the quality of utterance representation and mitigating the interference of natural noise in the utterance, thus improving the subsequent fusion process.

Graph-based methods capture speaker and contextual interactions through edges, improving perception of complex relationships within conversations. Hypergraph-based models, particularly, offer superior emotional understanding due to the capacity to yield high-order relationships between multiple connected nodes. While M3Net and HAUCL leverage this property of hypergraphs, HAUCL adaptively constructs hyperedges via a hypergraph variational autoencoder, reducing the misleading influence of redundant hyperedges on node information transfer. M3Net, by contrast, enriches node-level information through multi-frequency decomposition. However, both methods primarily focus on node-level information, neglecting the role of hyperedges in contextual understanding. Our MATCH enhances hypergraph representations through line expansion, capturing both fundamental semantic expressions and deeper conversational patterns. The emotion-aligned learning further minimizes the information discrepancies between the two, enabling more effective emotion comprehension.

Interestingly, we find that MATCH outperforms CauAIN and PCGNet, which rely on external knowledge including personality traits and commonsense. This indicates that MATCH achieves comprehensive emotion understanding solely through multimodal data, without the need for external knowledge.

4.5 Ablation Study

We conduct ablation experiments on MATCH to validate each component's effectiveness.

Effect of Conversational Entity Calibration. As shown in Table 5, removing the entity calibration module significantly degrades MATCH's performance, as it relies solely on initial features which contain coupled or redundant information, hindering reliable predictions. This unreliability is effectively alleviated by introducing entity calibration. While M3Net attempts to mitigate coupling through multifrequency decomposition, it does not fully distinguish the contributions of contextual and modality information, leading to suboptimal performance. We apply the proposed entity calibration strategy to several baselines, with the results in Table 4 showing improvements in all cases. This indicates that most baselines have limitations in utilizing multimodal features, and our entity calibration strategy provides a foundation for extracting deeper emotional cues.

Effect of Emotion-aligned Hypergraph Fusion. As MATCH's core module, emotion-aligned hypergraph fusion is designed to capture high-order semantic cues and diverse dialogue relationships. Without it, MATCH struggles to perceive complex relationships, limiting its ability to facilitate effective cross-modal interactions essential for MERC. Furthermore, deeper conversational patterns are missed, preventing the accurate conveyance of similar emotions through contextual and modality relationships, and resulting in the loss of emotional pathways. This lack of deeper information severely impacts the model's ability to generate accurate emotion judgments.

Effect of Conversational Pattern Learning. We introduce line expansion to enhance the hypergraph's ability to represent high-order semantics, uncovering meaningful conversational patterns and emotional pathways, emphasizing the positive role of hyperedges in MERC. Without this, the node-focused propagation mechanism of hypergraph fails to effectively transmit hyperedge information, leading to redundant semantics being mistakenly propagated to connected nodes. Additionally, learning conversational patterns offers more contextualized emotional insights and effectively preserves implicit semantic transfer within conversations. Integrating conversational patterns with global semantic understanding through graph-level alignment enables a more comprehensive affective understanding.

4.6 Hyperparameter Sensitivity Analysis

We analyze key hyperparameters γ1, γ2, γ3, and the number of hypergraph (L) and line graph (K) layers on IEMOCAP validation set. Figure 3 demonstrates that the impact of these hyperparameters on the overall W-F1 results follows a trend of initial improvement, which then declines and stabilizes as their values increase. Excessively large parameter values reduce their effectiveness, indicating challenges in achieving an optimal balance between primary and auxiliary tasks. Figure 4 illustrates that performance improves initially, but declines as the number of hypergraph or line graph layers increases. An excessive number of layers fails to produce more meaningful emotion representations.

5 Conclusion

In this paper, we introduce MATCH, a hypergraph-based framework for MERC. MATCH enhances utterance representations by calibrating dialogue entities at context and modality perspectives. It captures high-order substructural interactions in the hypergraph through line expansion, extracting deep emotional cues while preserving fundamental semantics, enabling effective emotional pathway learning. Besides, emotion-aligned learning improves knowledge transfer at hyperedge and graph levels. Experiments demonstrate that MATCH outperforms state-of-the-art methods on two datasets without relying on external knowledge, fully leveraging the potential of hypergraphs in MERC.

Acknowledgments

This work was supported in part by the "Pioneer" and "Leading Goose" R&D Program of Zhejiang (No. 2024C03262), and by the National Natural Science Foundation of China (No. U21A20473, No. 62172370, No. 62176244, No. 62032022, No. U23A20388, No. 62320106007).

Contribution Statement

Ming Li (email: mingli@zjnu.edu.cn) is the corresponding author of this work.