Speaker-Guided Encoder-Decoder Framework for Emotion Recognition in Conversation

1 Introduction

Emotion recognition in conversation (ERC) aims to identify the emotion of each utterance in a conversation. Due to its potential applications in creating empathetic dialogue systems [Zhou et al., 2018], social media analysis [Li et al., 2019], and so on, it has been concerned by a considerable number of researchers recently.

Unlike vanilla emotion recognition of sentences, emotions expressed in a conversation are dynamic and can be affected by many factors such as surrounding conversational context, and speaker dependencies [Poria et al., 2019b]. Consequently, ERC models require a strong ability to model context, select crucial information, and capture speaker dependencies, which is more challenging. Among all the factors, speaker information is vital for tracking the emotional dynamics of conversations, which consists of two important aspects: intra- and inter-speaker dependencies [Poria et al., 2019a]. Intra-speaker dependency, also known as emotional inertia, deals with the aspect of emotional influence that speakers have on themselves during conversations [Kuppens et al., 2010]. In contrast, inter-speaker dependency refers to the emotional influences that the counterparts induce into a speaker [Poria et al., 2019c]. Exploring intra-speaker dependency and inter-speaker dependency jointly is conducive to comprehending the complex dialogue context that implies various emotions.

∗Corresponding author.

Numerous approaches have been proposed for ERC with a focus on intra- and inter-speaker dependencies modeling. They can be divided into two categories: static speakerspecific modeling [Zhang et al., 2019; Ghosal et al., 2019; Li et al., 2020; Shen et al., 2021a] and dynamic speakerspecific modeling [Majumder et al., 2019; Hu et al., 2021]. For the static speaker-specific modeling methods, they utilize speaker information attaching to each utterance to specify different connections between utterances. In this way, the injected prior speaker information is static and shows the limited ability to facilitate conversational context modeling. Intuitively, the speaker state that is affected by the intra- and inter-speaker dependencies, changes dynamically during a dialogue. Different states of a certain speaker greatly affect various emotional expressions. Thus, it is necessary to implement dynamic speaker-specific modeling.

As shown in Figure 1(a), existing dynamic speaker-specific modeling methods [Majumder et al., 2019; Hu et al., 2021]

design speaker-specific recurrent neural networks (RNNs) to capture self-dependencies between adjacent utterances of the same speaker. In this scheme, utterances are classified based on speaker identity. However, sequences composed of separate utterances attaching to the same speaker are disjunctive and incomplete, which hinder the comprehension of the context. Moreover, overreliance on the speaker identity hinders the modeling of dynamic inter-speaker dependency. In this way, the model can't explore dynamic interactions between different speakers, which greatly affects the psychology of speakers, and then results in different emotional expressions. Nonetheless, how to explore intra- and inter-speaker dependencies that change dynamically is still unresolved.

To alleviate the problem, we design a novel speaker modeling scheme that explores intra- and inter-speaker dependencies dynamically. Besides, we propose a speaker-guided encoder-decoder framework (SGED), which can combine with various existing methods. The encoder module consists of two parts: conversational context encoder (CCE) and speaker state encoder (SSE). We use different existing methods which ignore dynamic speaker dependencies modeling as the CCE. For the SSE, we design a novel speaker modeling scheme to generate a dynamic speaker state sequence that depends on the speaker's previous state (intra-speaker dependency) and other neighboring speakers' states (inter-speaker dependency). For the decoder module based on RNN, we use speaker states to guide the decoding of utterances' emotions step by step. Extensive experiments indicate the effectiveness of SGED. In summary, our contributions are three-fold:

• To fully exploit the complex interactions between speakers that are vital for context comprehension, we propose a speaker-guided encoder-decoder framework (SGED) for ERC, formulating the modeling of speaker interactions as a flexible component.
• We are the first to explore dynamic intra- and inter-speaker dependencies jointly through a subtly designed speaker state encoder, which explores complex interactions between speakers effectively.
• Experimental results on three benchmark datasets demonstrate the effectiveness and high scalability of SGED.

2 RelatedWork

Recently, ERC has received increasing attention from the NLP community. Unlike traditional emotion recognition tasks, ERC models should be endowed with the ability of speaker interaction modeling and conversational context modeling.

2.1 Speaker InteractionModeling

According to whether the speaker-specific information is modeled dynamically during the dialogue, speaker modeling methods can be divided into two categories: static and dynamic speaker-specific modeling. For the static methods, some of them [Ghosal et al., 2019; Zhang et al., 2019] treat dialogue as a graph and inject prior speaker information as different relations between utterances or treat speakers as nodes in the graph. DialogXL [Shen et al., 2021a] changes the masking strategies of self-attention to capture intra- and inter-speaker dependencies. HiTrans [Li et al., 2020] exploits an auxiliary task to classify whether two utterances belong to the same speaker to make the model speaker-sensitive. However, these methods inject prior static speaker information into the model, neglecting that speaker states change dynamically according to the varying intra- and inter-speaker dependencies along with the dialogue. By contrast, DialogueRNN [Majumder et al., 2019] and DialogueCRN [Hu et al., 2021] utilize RNNs to update speakers' states in a dynamic manner. Nonetheless, they only consider intra-speaker dependency, ignoring inter-speaker dependency because of the limited speaker modeling scheme. Thus, we propose a speaker-guided encoder-decoder framework, which explores dynamic intra- and inter-speaker dependencies jointly.

2.2 Conversational Context Modeling

According to the methods of conversational context modeling, existing models can be mainly divided into graphbased methods [Ghosal et al., 2019; Zhang et al., 2019; Shen et al., 2021a; Zhong et al., 2019] and recurrencebased methods [Majumder et al., 2019; Hu et al., 2021; Jiao et al., 2019; Hazarika et al., 2018]. Recently, DAGERC [Shen et al., 2021b] is proposed to model the dialogue in the form of a directed acyclic graph, which allows the advantages of graph-based methods and recurrence-based methods to complement each other. Besides, some methods have exploited external commonsense knowledge to enrich the context representation learning [Zhong et al., 2019; Ghosal et al., 2020]. We use different methods as the context encoder, proving the effectiveness and high scalability of our proposed speaker-guided encoder-decoder framework.

3 Preliminary

3.1 ProblemDefinition

A conversation containing N utterances is denoted as C = {u1, u2, ..., uN}. Each utterance ui consists of ni tokens, namely ui = {xi1, xi2, ..., xini }. There are S speakers s1, s2, ..., sS(S ≥ 2) in a dialogue. Utterance ui is uttered by speaker sϕ(ui) where the function ϕ maps ui to its corresponding speaker. If ui isn't the first utterance spoken by sϕ(ui) in the conversation, we denote uψ(ui) as the last previous utterance expressed by the same speaker where 0 < ψ(ui) < i. We denote {uj |∀j, ψ(ui) ≤ j < i} as the local information for ui. ERC task aims to predict the emotion label yi for each utterance ui in the conversation C.

3.2 Textual Feature Extraction

By convention, we use the pre-trained language model to extract utterance-level representations. Following Shen et al [2021b], we fine-tune the pre-trained language model RoBERTa-Large [Liu et al., 2019] on each ERC dataset first and then freeze its parameters while training our proposed framework. Concretely, we first append a special token [CLS] at the beginning of a given utterance ui to obtain the input sequence {[CLS], xi1, xi2, ..., xini }. Afterward, we extract a representation of ui from the [CLS]'s embedding of the last layer with a dimension of 1,024.

4 Methodology

As illustrated in Figure 2, there are three components in our proposed framework SGED: (1) conversational context encoder; (2) speaker state encoder; (3) speaker-guided decoder.

4.1 Conversational Context Encoder

We take the existing baseline as the conversation context encoder of SGED, which is easily replaceable. 1 For a given dialogue C, we generate the utterance representation vector sequence H = [h1, h2, ..., hN ] as follows:

where Encoder means the replaceable conversational context encoder.

4.2 Speaker State Encoder

Afterward, we generate a speaker state vector for each utterance based on the utterance representations from the conversational context encoder. The dynamic speaker state has two factors: intra-speaker dependency and inter-speaker dependency. We take the sequence of utterance representations as input and model the two kinds of dependencies simultaneously to obtain the final speaker state vectors sequentially.

Intra-Speaker Dependency Modeling. For a given utterance ui which is uttered by speaker sϕ(ui), we take the last previous speaker state of sϕ(ui) and previous context ci = [h1, h2, ..., hi] ∈ Rh×i as input, modeling the intra-speaker dependency implied in the dialogue.

First, we utilize the last previous speaker state vector of sϕ(ui) and the representation vector of ui to obtain the query vector qintra i as follows:

1 In the experiments, we take different existing methods, namely bc-LSTM+att [Poria et al., 2017], DialogueGCN [Ghosal et al., 2019], and DAG-ERC [Shen et al., 2021b], for the conversational context encoder.

where vψ(ui) ∈ Rh×1 means the last previous speaker state vector of sϕ(ui); || means the concatenation operation; Wintra q ∈ Rh×2h and bintra q ∈ R1 are model parameters.

Then, we use qintra i as the query and previous context ci as the key and value to implement the attention mechanism, generating the intra-speaker state vector of ui as follows:

where ⊙ is an element-wise product operation; W1 ∈ R1×h and b1 ∈ R1 are model parameters; αintra i ∈ R1×i is the attention score to aggregate the previous context; vintra i ∈ R1×h is the intra-speaker state vector of ui.

Inter-Speaker Dependency Modeling. Simultaneously, we treat the representation vector of ui and the speaker state vectors of local information {uj |∀j, ψ(ui) ≤ j < i} as input, modeling the latent inter-speaker dependency.

We adopt the following methods to generate the interspeaker state vector of ui:

where ki = {vj|∀j, ψ(ui) ≤ j < i} ∈ Rh×l denotes the speaker state vectors of previous local information; ⊙ means the element-wise product operation; Winter q ∈ Rh×h , binter q ∈ R1 , W2 ∈ R1×h , and b2 ∈ R1 are model parameters; vinter i ∈ R1×h is the inter-speaker state vector of ui.

Finally, we fuse the intra-speaker state vector vintra i and the inter-speaker state vector vinter i to generate the final speaker state vector of ui:

where vi ∈ R1×h is the speaker state vector of ui. Note that if ui is the first utterance expressed by speaker sϕ(ui) in the dialogue, we let vi = hi. For the given dialogue C, we obtain the speaker state vector sequence V = {v1, v2, ..., vN}, which encodes speaker-specific intra-speaker dependency and inter-speaker dependency dynamically.

4.3 Speaker-Guided Decoder

To fully utilize the vital speaker-specific information implied in the conversation, we use the speaker state vector vi to guide the decoding of ui's emotion. In addition, inherent relations between emotion tags also facilitate ERC [Wang et al., 2020]. Therefore, we also use the embedding of the predicted emotion of ui-1 to instruct the decoding of ui's emotion. We subtly design a speaker-guided emotion decoder based on (gated recurrent unit) GRU to decode each utterance's emotion in a given dialogue sequentially.

Given an utterance ui, we first match the speaker state vector vi with the utterance representation vector hi as follows:

where Wm ∈ Rh×h and bm ∈ R1 are trainable parameters for feature transformation; mi ∈ R1×h is the generated match vector.

Afterward, we concatenate the match vector mi and the embedding ei-1 of the predicted emotion of ui-1 and then feed it to GRU:

Finally, we take the concatenation of hi ∈ Rh×1 and oi ∈ R1×h as the final representation of ui, and feed it to a feedforward neural network for emotion prediction:

where Wo ∈ Rh×2h , Wz ∈ RL×h , bo, bz ∈ R1 are trainable parameters; L is the class number of the dataset; ˆ yi is the predicted label of ui; E is the look-up table of emotion embeddings which is initialized randomly at the beginning; ei is the embedding of the predicted emotion of ui.

To train the model, we use cross-entropy loss function:

where M is the number of dialogues in the training set; Nj is the number of utterances in the j-th dialogue; yj,t is the ground-truth label; θ is the collection of trainable parameters.

5 Experimental Settings

5.1 Datasets

We evaluate our proposed framework on three ERC benchmark datasets. The statistics of them are shown in Table 1.

IEMOCAP. A two-party conversation dataset for ERC [Busso et al., 2008], which includes 6 types of emotion, that is neutral, happiness, sadness, anger, frustrated, and excited. Since this dataset has no validation set, we follow [Shen et al., 2021b] to use the last 20 conversations in the training set for validation.

MELD. A multi-party conversation dataset collected from the TV series Friends [Poria et al., 2019a]. The emotion labels are: neutral, happiness, surprise, sadness, anger, disgust, and fear.

EmoryNLP. A multi-party conversation dateset collected from Friends, but varies from MELD in the choice of scenes and emotion labels [Zahiri and Choi, 2018]. There are 7 types of emotion, namely neutral, sad, mad, scared, powerful, peaceful, and joyful.

Although all of the datasets contain multimodal information, we only focus on the textual information for experiments. Following existing work [Shen et al., 2021b], we choose weighted-average F1 for evaluation metrics.

5.2 ComparedMethods

We compare our method with the following baselines.

KET [Zhong et al., 2019] is a knowledge-enriched transformer that extracts commonsense knowledge based on a graph attention mechanism. COSMIC [Ghosal et al., 2020] incorporates different elements of commonsense and leverages them to learn self-speaker dependency. DialogueRNN [Majumder et al., 2019] uses speaker-specific GRU to model intra-speaker dependency and generate speaker states sequentially in a dynamic manner. Another dynamic speaker-specific modeling method is DialogueCRN [Hu et al., 2021], which utilizes speaker-level context to model dynamic intra-speaker dependency and designs a multi-turn reasoning module to imitate human cognitive thinking. bc-LSTM+att [Poria et al., 2017] is a contextual LSTM network without speaker information modeling, which is based on an attention mechanism to re-weight the features. DialogueGCN [Ghosal et al., 2019] is a graphbased method that treats speaker dependencies as static information and designs different relations between utterances. DAG-ERC [Shen et al., 2021b] models a dialogue in the form of a directed acyclic graph and utilizes speaker identity to establish connections between utterances.

We use the same feature extractor RoBERTa [Liu et al., 2019] as provided by Shen et al., [2021b]. We replace the textual feature in the previous methods with our extracted features first. Then, we use previous methods without dynamic speaker-specific information modeling (RoBERTa, bc-LSTM+att, DialogueGCN, DAG-ERC 2 ) as the conversational context encoder of the proposed speaker-guided encoder-decoder framework, investigating the effectiveness of our speaker modeling scheme.

2 Note that we use the one-layer DAG-ERC as the context encoder, making the parameters fewer than the original DAG-ERC with multiple layers. In this way, the potential impact of increasing parameters provided by the decoder is eliminated. Our method with fewer parameters even surpasses the original DAG-ERC with more parameters, proving the effectiveness of our method more convincingly.

5.3 Implementation Details

We utilize the validation set to tune parameters on each dataset and adopt Adam [Kingma and Ba, 2015] as the optimizer. The learning rate and batch size are selected from {1e-5, 5e-5, 1e-4, 5e-4} and {8, 16, 32}, respectively. The dimension of hidden vector h is set to 300 and the dimension of label embedding is set to 100. The feature size for the RoBERTa extractor is 1,024. We use the fine-tuned RoBERTa features provided by Shen et al., [2021b]. Each training process contains 60 epochs. The results of our proposed SGED framework reported in the experiments are all based on the average score of 5 random runs on the test set.

6 Results and Analysis

6.1 Overall Results

Overall results of our proposed method and the baselines are reported on Table 2. The results demonstrate that methods combined with our speaker-guided encoder-decoder framework achieve significant improvement.

As shown in Table 2, based on the same feature extractor RoBERTa, the framework proposed by us outperforms the other dynamic speaker-specific information modeling methods (DialogueRNN, DialogueCRN) in general. Moreover, SGED + RoBERTa which encodes each utterance independently in the context encoder, achieves comparable results with DialogueRNN+RoBERTa, proving the effectiveness of the SGED proposed by us. Taking different types of existing methods as the conversational context encoder, SGED which combines the novel speaker modeling scheme achieves significant improvement all the time. It indicates the high scalability and flexibility of SGED. Moreover, SGED + DAGERC* reaches the new state-of-the-art on the three benchmark datasets, even surpasses COSMIC which utilizes external commonsense knowledge.

MELD and EmoryNLP. On these multi-party conversation datasets, we observe that our SGED framework makes varying degrees of promotion based on different conversational context encoders. For the recurrence-based method bcLSTM+att and context-independent method RoBERTa, the promotion brought by the SGED framework is about 0.55% on average. However, for the graph-based methods DialogueGCN and DAG-ERC, our SGED framework achieves an increase of 1.66% on average. It is because the speaker state encoder of SGED is established based on the conversational context encoder. Thus, the context comprehension ability of the conversational context encoder affects the subsequent dynamic speaker-specific information modeling, and finally influences the decoding of emotion. The stable improvement brought by SGED indicates that the proposed dynamic speaker modeling scheme helps to explore crucial dependencies between speakers and benefits ERC.

IEMOCAP. On the two-party conversation dataset, we observe that the improvement brought by SGED has little correlation with the type of context encoder, which is different from the results on the multi-party conversation datasets. Besides, the improvement of SGED on IEMOCAP is 0.83% on average, which is slightly lower than the improvements on MELD and EmoryNLP. We argue that the dependencies between speakers in multi-party conversations are more complex and intractable. Therefore, it is more necessary to exploit SGED to model speaker-specific states dynamically for multi-party conversations.

6.2 Ablation Study

To investigate the effects of different modules in SGED, we conduct several ablation studies on EmoryNLP and IEMOCAP. We remove 1) intra-speaker dependency in speaker state encoder (SSE), 2) inter-speaker dependency in SSE, 3) SSE, 4) speaker-guided decoder (SGD), and 5) SSE and SGD, respectively. The results are reported on Table 3.

Effect of Different Dynamic Dependencies between Speakers. As shown in Table 3, the result on EmoryNLP declines when removing either intra-speaker dependency or inter-speaker dependency in the SSE. However, SGED without inter-speaker dependency performs worse than without intra-speaker dependency on IEMOCAP. It proves that both dynamic intra- and inter-speaker dependencies are important for multi-party conversations while dynamic inter-speaker dependency is more important for two-party conversations. Moreover, the result drops sharply when eliminating the entire speaker state encoder (SSE) that explores both intra- and inter-speaker dependencies in a dynamic manner, which indicates that the two kinds of dynamic dependency are complementary to each other.

Effect of Speaker-Guided Decoder. To evaluate the effectiveness of the speaker-guided decoder (SGD), we remove it and only use the speaker state vector for prediction. As shown in Table 3, it is clear that the speaker-guided decoder proposed by us boosts model performance a lot. Removing the SGD not only has a great impact on multi-party conversations (1.28%) but also two-party conversations (1.23%), which proves that the aggregation of speaker state feature and semantic feature benefits emotion recognition in conversation.

Effect of the Novel Speaker Modeling Scheme. Our proposed speaker modeling scheme consists of speaker state encoder (SSE) and speaker-guided decoder (SGD). As reported on Tabel 3, removing SSE and SGD simultaneously results in a significant drop. It indicates the effectiveness of the proposed speaker modeling scheme, which can capture dynamic speaker interactions and promote emotion prediction.

6.3 Discussion

Different Number of Speakers. As shown in Figure 3 (a), we illustrate the results of dialogues with the different number of speakers on MELD, achieved by DialogueGCN and our SGED framework which takes DialogueGCN as the conversational context encoder. We observe that our proposed method exceeds DialogueGCN on dialogues with different speaker numbers except 6. Compared with DialogueGCN, our method achieves the greatest promotion on dialogues with 5 speakers. We will take an example with 5 speakers for the case study and conduct an error analysis on dialogues with 6 speakers.

Case Study. In Figure 3 (b), we present an abridged dialogue that has 5 speakers originally and our method recognizes most emotions correctly. For person A, we recognize u3 as anger correctly though it is a short utterance that is intractable. We argue that the intra-speaker dependency of A captured by our method facilitates the recognition. Moreover, although person A's previous utterances (u1, u2, u3) all express anger, our method recognizes u7 as happiness which is obviously different from anger, based on the inter-speaker dependencies among person C (u5), B (u6), and A (u7). For person B, we recognize u8 as happiness correctly, due to the trade-off of inter-speaker dependency between person A (u7) and B (u8) and intra-speaker dependency of B captured by our method. It indicates that SGED models dynamic speaker states effectively with integration of intra-speaker dependency and inter-speaker dependency.

Error Analysis. In Figure 3 (c), we present the confusion matrix of conversations with 6 speakers excluding the diagonal, achieved by our method. From the heat map, neutral samples are more confused with non-neutral (e.g., happiness) samples, which can be alleviated by adding cues from other modalities. In addition to this, distinguishing similar emotions (e.g., happiness and surprise) still disturbs our method.

7 Conclusion

In this paper, we propose a speaker-guided encoder-decoder framework (SGED) for ERC, which captures dynamic intraand inter-speaker dependencies to benefit the understanding of the complex context. In this framework, we propose a novel speaker state encoder (SSE) and design a speakerguided decoder (SGD). SSE is competent at exploring intraand inter-speaker dependencies dynamically. Moreover, we treat different existing methods as the conversational context encoder of SGED, showing its high scalability and flexibility. Experiments on three benchmark datasets prove the superiority and effectiveness of our method. Especially, our method achieves a more significant improvement on the multi-party conversation datasets, indicating its strong ability to model dynamic multi-speaker states during a dialogue.