Hybrid Relational Graphs with Sentiment-laden Semantic Alignment for Multimodal Emotion Recognition in Conversation

1 Introduction

Emotions significantly shape human interactions, decisions, and overall well-being [Baltruˇsaitis et al., 2019]. With the growing expectation for machines to perceive, understand, and express emotions like humans, detecting emotions from conversations using multimodal information has become increasingly crucial [Geetha et al., 2024].

Multimodal emotion recognition in conversation (MERC) aims to utilize text, acoustic, and visual modalities within a conversational context to accurately identify the emotion expressed by the speaker in each utterance. Typically, MERC is treated as a classification task, where researchers often start with efficient multimodal information processing and fusion, followed by applying a classifier for emotion detection, as demonstrated in [Li et al., 2023a; Hu et al., 2024]. Recently, some studies have explored the use of text generation for emotion recognition. For example, [Hu et al., 2022b] unified sentiment analysis and emotion recognition by designing text labels and employing Transformer decoders [Vaswani et al., 2017] to progressively generate both the results and confidence scores. Based on this, [Li et al., 2023b] introduces UniSA, a unified multimodal generative framework designed to effectively handle various emotion-related detection tasks.

∗Corresponding Author

Alternatively, to address the complex relationships and dependencies among diverse modalities, graph-based methods utilize graph structures to capture intricate interactions and offer a comprehensive understanding of sentiment. Some standard graphs define specific rules to explicitly identify the relationships between nodes. [Li et al., 2024] establishes connections between nodes from different modalities; for instance, a textual node is linked to all visual nodes. [Nguyen et al., 2023] connects nodes within a specific window of the same modality, and nodes representing different modalities of the same utterance. Additionally, [Yi et al., 2024] replaces the aforementioned relationships with two types of corresponding hyperedges, which can connect any number of nodes, thereby naturally encoding relationships of higher arity.

However, they have limitations in graph structure and overlook certain aspects of sentiment encoding: (1) Sentiment features in the conversation history are ignored. The classification task generates discrete labels that do not inherently convey emotional content. For instance, when "0" is assigned to represent happiness, no emotion-related information can be derived from the label "0" in isolation. Consequently, the emotional features detected in the conversation history are not effectively utilized in predicting the emotions of subsequent utterances. However, incorporating historical emotions can provide benefits(e.g. Utterances 6 & 8 in Figure 1). (2) Implicit relationships between utterances in the conversation graph are overlooked. Blindly connecting all nodes within the same conversation may introduce noise by linking semantically unrelated nodes (Utterances 2 & 8, etc.), thereby influencing emotion recognition accuracy. Conversely, establishing connections solely based on explicit relationships that match predefined rules may lead to the neglect of implicit relationships (Utterances 4 & 8, etc.), which are equally crucial for comprehensive emotional analysis. (3) Variations in implicit relationships across different modalities are disregarded. In fact, there are distinct differences in the implicit relationships between nodes across various modalities. For example, two speakers may appear simultaneously in a scene and interact physically, but not verbally. This creates a relationship within the visual modality, while no such relationship exists in the textual or auditory modalities. Applying the relational structure of one modality uniformly across others may result in the introduction of spurious connections or the loss of crucial contextual information. Existing research has yet to provide a solution to these discrepancies.

To address the challenges outlined earlier, we propose a novel encoder-decoder framework, HRG-SSA, which combines Hybrid Relational Graphs(HRG) with a Sentimentladen Semantic Alignment strategy(SSA). This approach treats MERC as a text generation task, thereby endowing the emotional labels in the conversation history with genuine textual emotional semantics. SSA employs a two-stage contrastive learning process. Initially, it aligns the various modal features of the conversation history with emotions, and subsequently, it aligns the semantics of each sentiment-laden modal feature. This progressive alignment enhances the model's ability to fuse multimodal information during the encoding process. Inspired by [Zhao et al., 2022], HRG employs a modality-adaptive connection prediction module that infers implicit relationships between nodes within a modality based on their semantic similarity. It then integrates these inferred relationships with explicit ones to construct a more coherent and comprehensive conversation graph structure, thereby enhancing the interaction and fusion of features within each modality. Ultimately, a decoder incorporating dynamic fusion layers is employed to effectively integrate multimodal information, enabling the generation of the emotional polarity for the current utterance. The main contributions of this paper can be summarized as follows:

• We propose an SSA strategy that uses progressive contrastive learning to integrate sentiment cues into contextual references, facilitating multimodal semantic alignment. Additionally, a novel modality-adaptive HRG infers implicit connections within modalities, completing
• the dialogue graph and improving modality fusion.
• We propose an end-to-end generation framework HRGSSA that incorporates HRG and SSA into the encoding phase to enhance multimodal information representation. The decoder subsequently employs dynamic fusion layers to seamlessly integrate multimodal inputs and generate the speaker's emotional tendency.
• Experiments on the IEMOCAP and MELD datasets demonstrate that our method outperforms state-of-theart approaches in accuracy and weighted F1 score, highlighting its effectiveness in MERC.

2 RelatedWork

2.1 Transformer-BasedModels for MERC

Drawing inspiration from Transformer architectures, most of recent studies focus on designing cross-modal attention mechanisms to effectively fuse various modalities [Rahman et al., 2020; Huang et al., 2023; Sun et al., 2023]. [Hubert et al., 2019] categorized the challenges in multimodal data modeling into two types: misalignment within modalities and across modal feature spaces. They proposed the endto-end Multimodal Transformer (MulT) to address these issues and showed the effectiveness of the attention mechanism in modal fusion. CENet enhances each word's representation by incorporating long-range emotional cues implicit in the visual and auditory modalities [Wang et al., 2022]. SDT employs a Transformer-based model with intra-modal and intermodal attention mechanisms to capture interactions and dynamically learn modality weights through hierarchical gating [Ma et al., 2023]. [Maji et al., 2023] designed a cross-modal transformer block to capture interactions and temporal dependencies between audio and text, while using self-attention to prioritize key sentiment information from the fused features. Recent studies have explored text generation for emotion recognition using transformer decoders. They unified the emotion-related classification tasks into an end-to-end framework by unifying their label formats [Hu et al., 2022b; Li et al., 2023b]. Although these methods have made progress in MERC, none have recognized the potential of emotional labels in dialogue history to enhance emotion prediction in subsequent interactions.

2.2 Graph-BasedModels for MERC

Sequential modeling has an upper limit in extracting features from conversation context, prompting some researches to explore the graph structure within dialogues. These approaches typically treat each modality of a utterance as a node and establish connections based on meaningful relationships [Hu et al., 2024]. Early studies relied on the adjacency relationships inherent in the conversational structure to establish edges. For example, a common approach is to connect the nodes within the same modality for a given conversation and link nodes across different modalities within the same utterance [Hu et al., 2021a; Nguyen et al., 2023]. Moreover, some studies exploit speaker-specific relationships to establish connections between the corresponding nodes [Lian et al., 2023; Lee and Choi, 2021]. Furthermore, some researchers have exploratively leveraged external knowledge and rules to extract relationships within the context. For example, M3GAT employs the spaCy toolkit to construct a dependency tree of text utterances, leveraging the syntactic dependencies between words to build edges [Zhang et al., 2023a]. DER-GCN constructs a multi-relational sentiment interaction graph that incorporates relationships between speakers and heterogeneous elements derived from events [Ai et al., 2024]. MKEIGN integrates textual and visual common-sense knowledge into the edge representation to improve the modeling of conversation graph relations by external knowledge [Tu et al., 2024a]. The introduction of hyperedges effectively shortens the distance between interactions across different modalities, making the hypergraph an innovative approach for modeling complex relationships in MERC [Chen et al., 2023; Yi et al., 2024]. Exploratively, [Pei et al., 2024] proposed a multi-track graph convolutional network that separates message passing by category semantics to prevent heterophilic mixing and mitigate oversmoothing and oversquashing in deep GNNs. However, these methods mainly focus on explicit utterance-level relationships, overlooking both the implicit connections within each modality and the variations in implicit relationships across modalities.

3 Methodology

In this section, we provide a detailed overview of each module in the proposed HRG-SSA, as illustrated in Figure 2.

3.1 ProblemFormulation

A conversation consists of a sequence of utterances: C = {u1, u2, ..., uN}. N is the number of utterances. Each utterance ui consists of three modalities: text, audio, and vision, along with a corresponding speaker: ui = {uti , uai , uvi ; Si}. uti ∈ Rdt , uai ∈ Rda and uvi ∈ Rdv , dt, da, dv denote represent the dimensionalities of the text, audio, and visual modalities, respectively. The goal of MERC is to predict the emotion yi expressed by the speaker Si in each utterance ui, based on the conversation history: Ch = {u1, u2, ..., ui-1}.

3.2 Feature Extraction and Encoding

Following [Chen et al., 2023], we employ DenseNet [Huang et al., 2017] or 3D-CNN [Yang et al., 2019] to extract features from the visual modality, and utilize OpenSmile toolkit [Eyben et al., 2010] to extract features from the audio modality. For the text modality, we uniformly utilize the pre-trained T5-base [Raffel et al., 2020] encoder for feature extraction.

Unimodal Encoder: To ensure consistency in the dimensionality of the representations across modalities, we employ two projection layers to perform preliminary spatial alignment for the audio and visual features, as detailed below:

where uηi represents the extracted feature, resulting in fixedsize representations hηi ∈ Rdh , η ∈ {a, v}; Wη and bη are learnable parameters. It is worth noting that these two modalities are only encoded at the utterance level.

The text modality includes both utterance-level and context-level encoding, which are utilized for graph initialization and the cross-attention module of the decoder, respectively. We integrate speaker identity information by adding a "speaker:" prefix to each utterance, followed by an end-ofsequence token "</s>" as a suffix. For instance, the Utterance 3 in Figure 1 appears as "Ross: You chipped in?!

</s>". We use the representation of the token "</s>" as the utterance-level encoding, while retaining the context-level representations:

where C th = {ut1 , ut2 , ..., uti-1 }, indexof("</s>") represents the index of "</s>".

Sentiment Encoding: To effectively leverage the sentiments within the conversation history, we structured the sentimental labels of each utterance into a standardized textual format. The specific format is as follows: "Ross: surprise & negative </s>", and we use si to denote that of utterance i. This structured representation ensures consistent integration of emotional context throughout the dialogue. Encoding it with the T5 encoder yields a representation rich in emotional semantics, enabling a more nuanced understanding of the dialogue's sentiment:

3.3 Initial Fusion and Sentiment Alignment

After encoding, we employ the cross-attention (CS-Attention for short) [Vaswani et al., 2017] mechanism to facilitate the initial fusion between modalities at the utterance level. For the text modality, we use hti as the query, while the concatenation of hai and hvi serves as both the key and value for attentional interactions.

Ht = CS-Attention(ht , hav , hav )

where ∥ denotes the concatenate operation; ·T represents the transpose operation of a matrix; W Q t ,W K t ,W V t are learnable parameter matrices; d is the dimension of hidden states.

In order to improve fusion efficiency and align with the conclusion that text plays a dominant role in multimodal affective computing [Zhang et al., 2023a; Zhang et al., 2023b], we use hti as the key and value for acoustic and visual modalities. And we utilize hai and hvi as the queries for attentional interactions, respectively.

where η ∈ {a, v}; W Q η ,W K η ,W V η are learnable parameter matrices.

Primary Sentiment Alignment(PSA): In this module, we use historical sentiment to align contexts with their corresponding sentiment expressions. Multiple sentiments are often present in a conversation, and aligning the semantics of an utterance with its true emotional state, while distancing it from other emotions, enhances the model's attention to sentimental cues, thereby improving subsequent emotion prediction. In prior work, we encoded both the speaker's utterances and their associated emotions to obtain hsi . In the current module, we treat each modal representation Hη i , η ∈ {t, a, v} and its corresponding hsi as a positive pair, while pairing hηi with the representations of other hsj , j ̸= i as negative pairs. This contrastive learning strategy facilitates semantic alignment across modalities. Suppose there are n utterances in the dialogue, and the comparative loss is as follows:

where the target utterance is not involved; sim(H η i , hsi ) denotes the cosine similarity between two vectors; log(·) denotes the logarithmic function; and τ is a temperature parameter. By minimizing this loss, modality features are effectively aligned with the correct sentiment representation space.

3.4 Hybrid Relational Graphs

Graph Structure: We represent successive conversation contexts using a hybrid relational directed graph G = (V, E), where each node v ∈ V represents a unimodal utterance and the set of edges E includes both explicit and implicit relationships between the nodes. All nodes are initialized with their corresponding hidden states Hη i . And, the directed graph structure ensures that subsequent nodes cannot be used for emotion prediction of preceding nodes, preserving the temporal causality of the conversation.

Explicit relations remain consistent across modalities; we separately link neighboring utterances (including the utterance itself) and those from the same speaker. This approach preserves the temporal relationships underlying the pairwise connections and enables the tracking of each speaker's emotional evolution. Implicit connections(IC) capture the hidden pathways of information propagation within each modality by employing a detector that evaluates the correlations between nodes. These connections complement explicit relationships and result in modality-specific edge structures.

Specifically, based on the hidden states of nodes in each modality, the implicit unobserved relationships of modality adaptation are computed. A single-layer feedforward neural network, along with a mask matrix, is employed to calculate the scores sηij between node i and j, thereby inferring those implicit relationships.

where η ∈ {t, a, v}; W T η is a learnable parameter matrix; LeakyReLU is a nonlinearity activation function; and mask ∈ Rn∗n is a square matrix with -inf in the upper triangle and zeros in the lower triangle, thus preventing backward dependency. Then, a softmax function is applied to calculate the attention values αη ij for each node pair.

where exp(·) represents the exponential function. If the value exceeds the average score of node j predecessor, an implicit connection is established between node i and j (i<j):

where A represents the adjacency matrix.

Graph Update: The update of node representations within the graph structure follows the graph attention network [Veliˇckovi´c et al., 2018] framework, comprising two steps: directed edge weight calculation and information aggregation. First, we take each node representation Hη i as input and apply the self attention mechanism [Vaswani et al., 2017] to compute the attention distribution between a specific node and its previously connected nodes.

where We is a learnable linear transformation, aimed at providing sufficient expressive power to map the input features into higher-level representations; W Q e and W K e are learnable parameter matrices for attention mechanism; Nj denotes the neighborhood of node j in the graph.

Based on the attention weights computed above, we update each node by aggregating the features of its neighboring nodes. Additionally, we stack L layers of graph updates to facilitate information exchange across multi-hop nodes:

where η represents the modality; l denotes the index of the layer; σ(·) denotes the sigmoid function. We use the output of the final layer as the encoded representation of the hybrid relationship graph, denoted as HηL i .

Multimodality Semantic Alignment(MSA): Through the previous operations, the utterance node representations have been enriched with emotional context, and sufficient information fusion within the modality has been achieved. In this module, we will once again employ contrastive learning to perform cross-modal semantic alignment. The three modalities from the same modality form positive pairs pairwise, while negative pairs are formed between different utterance nodes. The comparative loss is as follows:

where n denotes the number of nodes in the graph; ξ indicates two other modalities besides η. By minimizing this loss function, the semantics of multiple modalities for the same node are brought into closer alignment.

Up to this point, the previous PSA module and the current MSA module progressively refine the role of sentiment cues, further promoting cross-modal alignment.

3.5 Decoder

This framework employs text generation tasks for emotion classification. We organized the sentimental labels into the fluent text format: "In the last round of the above dialogue, the speaker Ross's emotion is <bos emotion>surprise

<eos emotion>, and sentiment is <bos sentiment>negative <eos sentiment>.", where the special tokens with pointed brackets are placeholders for label extraction.

MultiModal Dynamic Fusion: Specifically, we utilize the pre-trained T5-base Decoder as the decoder and introduce several enhancements to optimize its performance. As mentioned previously, we have multiple inputs: htC , HηL , and HD(representing the hidden states of the decoder input). Since the classic transformer decoder can only process two inputs with a single cross-attention layer, we introduce a multimodal dynamic fusion structure by adding an additional cross-attention layer in series.

First, we use ˆ HD, obtained from the masked self-attention layer, as the query, while the conversation history representation htC serves as the key and value, passing through the first cross-attention layer. This approach allows us to capture event interactions and sentiment cues from the conversation history:

where W Q c , W K c and W V c are learnable matrices. Next, we use the concatenation of HηL as both the key and value, with HDC as the query, to extract sentiment cues from the multimodal representation through the second cross-attention layer:

where W Q m , W K m and W Vm are learnable matrices. Once the output is obtained, it is mapped onto the vocabulary through a feedforward network (FFN), and the probability distribution is generated using softmax, consistent with the original Transformer decoder:

where t denotes the t-th time step and Ws is learnable matrix. We adopt a cross-entropy loss to optimize this generation task:

Training Objectives: The final training objective is a weighted fusion of the three loss functions described above:

where the hyperparameter weight λ controls the significance of the contrastive loss.

4 Experiments

4.1 Experimental Settings

Datasets: We evaluate the performance of our model on two popular datasets for MERC: IEMOCAP [Busso et al., 2008]

Table 1: Results on IEMOCAP and MELD. Bolded values indicate the best results, and underlined values denote the second-best.

and MELD [Poria et al., 2019]. The former consists of videos of two-person conversations between 10 actors, while the latter is a multimodal dataset for multi-party conversations, taken from the Friends TV series. The distribution of both datasets is presented in Table 2. Note that since IEMOCAP does not provide labels for emotional stance, we assigned positive, negative, or neutral based on their emotional labels.

Implementation Details: All experiments were conducted on a single NVIDIA RTX A6000. We used a two-stage optimization strategy, first fine-tuning non-pretrained parameters (10 epochs) and then optimizing all parameters (10 epochs), with learning rates set to 5e-5 and 3e-5, respectively. Adam optimizer is used for training. The values of the remaining key hyperparameters are shown in the Table 3.

Baselines and Metrics: To evaluate the performance of our approach, we compare it with the following state-of-theart methods: (1) Transformer-Based models: UniMSE [Hu et al., 2022b] and UniSA [Li et al., 2023b]; (2) Graph-Based models: DialogueGCN [Ghosal et al., 2019], MMGCN [Hu et al., 2021b], MMDFN [Hu et al., 2022a], M3Net [Chen et al., 2023], AdaIGN [Tu et al., 2024b], DER-GCN [Ai et al., 2024], HAUCL [Yi et al., 2024]. For details on the baseline methods, please refer to section 2. Evaluation is conducted using the most widely adopted metrics in this field: accuracy (Acc) and weighted F1 score (w-F1).

4.2 Performance Comparison

Table 1 presents the results on the IEMOCAP and MELD datasets. The suffix "gen" indicates that predicted historical emotions are used in the experiments; otherwise, groundtruth emotions are used. The comparison between them suggests that higher prediction accuracy of historical emotions leads to improved future predictions. And our proposed HRG-SSA outperformed all the baseline models. Compared to HAUCL, HRG-SSA achieves improvements of 6.29% in ACC and 6.48% in w-F1 on IEMOCAP, and 0.34% in ACC and 0.86% in w-F1 on MELD. The model shows greater improvement on IEMOCAP, likely due to its simpler setup with fewer emotion categories and two-person dialogues. In contrast, MELD is more complex, with challenges like imbalanced emotion labels, leading to smaller gains.

4.3 Ablation Study

To more comprehensively evaluate the effectiveness of our proposed HRG-SSA method, we conducted ablation experiments to examine the contributions of PSA, MSA, and IC, as well as the impact of each modality on emotion recognition accuracy. The results are presented in Table 4.

In the table, "w/o PSA" and "w/o MSA" indicate the removal of the two losses, LPSA and LMSA, respectively, while "w/o IC" denotes the exclusion of all implicit connections. The results demonstrate that removing any of the aforementioned modules leads to a decrease in emotion recognition accuracy, highlighting the positive contribution of each module. Moreover, removing PSA or MSA alone causes a significant performance drop, indicating that the two modules complement each other and validating the effectiveness of the progressive semantic alignment strategy.

The results in Table 4 indicate that the model performs well with only the text modality. Adding audio improves accuracy, while the inclusion of visual information reduces it. We believe that speech and text share some semantic overlap, enabling them to complement each other. However, the visual information in the datasets depicts a large scene, with the speaker's facial expression being only a small part of it. This may hinder the model's ability to focus effectively, leading to a decrease in accuracy. The collaboration of the three modalities results in a significant improvement, highlighting the subtle collaborative mechanism among them and offering us a more comprehensive and multidimensional perception.

4.4 Sensitivity Analysis

We selected two key parameters in HRG-SSA for sensitivity analysis on the IEMOCAP and MELD datasets, as illustrated in Figure 3. The left subfigure illustrate the impact of the number of stacked graph layers on model performance. Both graphs exhibit a trend of increasing and then decreasing performance, suggesting that both overly shallow and overly deep graph stacks hinder effective node information aggregation. Performance is notably poor when the number of layers is 0 (i.e., without the hybrid relationship graphs), emphasizing its importance. Additionally, the optimal performance of IEMOCAP and MELD is achieved at 1 and 3 layers, respectively. This can be attributed to the intrinsic properties of the datasets. IEMOCAP, which comprises two-person dialogues, inherently establishes implicit relationships through multi-hop node interactions, thus diminishing the necessity for deeper nesting. In contrast, MELD is a multi-party conversation dataset and contains a certain amount of invalid spoken content, raising the cost of information mining. The right subfigure show the effect of the loss weight λ and both datasets achieve the best performance at 0.001.

4.5 Visualization

Since we believe that text plays a key role in the MERC task, we take the example shown by Figure 1 in section 1, and draw a hybrid relationship graph constructed by its textual modality, as well as the distribution of attention among the nodes, as shown in Figure 4. The figure provides a clear visual representation of the model's mechanism. The red dashed line with arrows highlights the implicit connections mined by HRGSSA, and "0" indicates no edges between nodes. Our model accurately identifies the full range of emotions by leveraging implicit connections and more rational attention allocation, while the baseline model HAUCL misclassifies U5 as Angry.

5 Conclusion

In this paper, we propose an end-to-end framework for MERC based on comparative learning and graphs. This framework aims to effectively utilize historical sentiment and emphasize implicit relationships within modalities. Our proposed method HRG-SSA complements the dialog graph by mining implicit relationships within each modality, thereby facilitating the alignment of sentiment-rich multimodal information. This improves the model's ability to extract sentiment cues and enhances accuracy. In the future, we will further explore the variability factors within each modality to further improve the performance of the model by constructing adaptive multimodal heterogeneous maps, as well as enhancing the interpretability of the model.

Acknowledgements

This research was supported by the National Natural Science Foundation of China (Nos. 62471403, 62261136549, 62271411, U22A2098), the Technological Innovation Team of Shaanxi Province (No. 2025RS-CXTD-009), the International Cooperation Project of Shaanxi Province (No. 2025GH-YBXM-017), the Fundamental Research Funds for the Central Universities (Nos. G2024WD0151, D5000240309).