CauAIN: Causal Aware Interaction Network for Emotion Recognition in Conversations

1 Introduction

Emotion recognition in conversations (ERC) aims at predicting the emotion for each utterance in conversations. Due to its key role to achieve empathetic systems and wide range of applications in opinion mining, social media analysis, health care and other areas, ERC has received increasing attention in the natural language processing (NLP) community.

The key challenge in ERC is posed by emotional dynamics [Poria et al., 2019a], which refers to emotional influences during the interaction between speakers. Early studies have been devoted to cope with this challenge by modeling intraand inter-speaker dependency with recurrent neural networks (RNN) [Hazarika et al., 2018; Majumder et al., 2019; Ghosal et al., 2020] and graph neural networks (GNN) [Zhang et al., 2019; Ghosal et al., 2019; Ishiwatari et al., 2020; Li et al., 2021].

However, such attempts for intra- and inter-speaker dependency modeling are limited in capturing deeper and richer clues of emotional dynamics, for the reason that the emotion cause, which exactly triggers the target emotion, is ignored. We believe that the model can better understand human emotions if it has the ability to associate the emotion cause with the target utterance.

∗Corresponding author

For causal utterance that expresses reasons for the speaker to feel such an emotion, we categorize them as either intracause utterance or inter-cause utterance. The former one refers to those appearing in the speaker's own dialogue turns, which means the cause of the emotion is due to a stable mood of the speaker that inherited from previous emotional states, while inter-cause utterance is present in the other speaker's turns and the emotion of the target speaker is influenced by the other speaker. An example is shown in Figure 1 to illustrate emotion recognition with the help of identifying intraand inter-cause utterances in conversations. The situation is that PersonA seeks PersonB for help about the presentation and we choose #7 as the target utterance to be classified. It is obvious that utterance #7 does not contain any explicit emotion indicator words and almost sound neutral on the surface. Thus, clues for its carrying emotion happy could only be inferred from the context. And #2 and #6 are inter-cause utterances that contribute significantly to the evoked emotion happy of utterance #7, where PersonB is willing to do PersonA a favor and promise a date tomorrow night. Also, parts of PersonA's happy are from the continuous emotional states conveyed by intra-cause utterance #3.

In this paper, to extract richer clues for emotional dynamics, we propose Causal Aware Interaction Network (CauAIN) to explicitly model intra- and inter-speaker dependency from a perspective of emotion cause detection. Since there are no ERC datasets with emotion cause information annotated, commonsense knowledge is leveraged as casual clues for emotion cause detection in conversations. To be more specific, six if-then relation types from ATOMIC (The Atlas of Machine Commonsense) [Sap et al., 2019] are considered according to their causal relations. xReact, xEffect and xWant are viewed as the intra-cause clues, while oReact, oEffect and oWant are explained for inter-cause clues. In order to identify causal utterances of a certain emotion expressed by the target utterance, we devise a two-step causal aware interaction consisting of causal clue retrieval and causal utterance traceback. To begin with, different intraand inter-cause clues are evaluated in terms of effects they have on the target utterance. Then results of the previous step could be viewed as gate values for intra- and inter-cause utterance traceback. In the final classification layer, causal aware representations are mixed for emotion recognition.

To evaluate the performance of the proposed model, we conduct extensive experiments on three datasets: IEMOCAP, DailyDialog and MELD. State-of-the-art performance is achieved on all of the three datasets. Further, we also demonstrate the exact emotion cause discovered by our proposed model.

The main contributions of this work are summarized as follows:

• We propose a novel model CauAIN to fully understand the conversational context with intra- and inter-cause utterance detection. It is the first attempt to explore the emotion cause for emotion recognition in conversations.
• We devise a simple but effective method of automatically detecting causal utterances with commonsense knowledge as causal clues, which does not depend on any annotation of emotion cause.
• Results of extensive experiments on three benchmark datasets demonstrate the effectiveness of the proposed model and our method of detecting exact emotion cause.

2 Methodology

First, we define the problem of the ERC task. Given a conversation with N consecutive utterances {u1, u2, · · · , uN} and M speakers {s1, s2, · · · , sM}, the goal of this task is to predict the emotion label ei of each utterance ui spoken by si. The architecture of our proposed model CauAIN is shown in Figure 3, which consists of four parts: Causal Clue Acquisition, Casual Clue Retrieval, Causal Utterance Traceback and Emotion Recognition. We will elaborate each one of them in the rest of this section.

2.1 Causal Clue Acquisition

Intra- and Inter-Cause Clue Representation

Due to the limitation of annotated emotion cause in existing ERC datasets, we turn to ATOMIC for causal clues offering.

What is ATOMIC? ATOMIC is an atlas of everyday commonsense reasoning and organized through textual descriptions of inferential knowledge, where nine if-then relation types are proposed to distinguish causes vs. effects, agents vs. themes, voluntary vs. involuntary events, and actions vs. mental states.

Why we choose ATOMIC? Previous works [Sap et al., 2019; Turcan et al., 2021] demonstrate that neural networks are capable of anticipating the likely causes and effects of previously unseen events with the help of rich inferential knowledge provided by ATOMIC. According to this, we extend such unseen events under the circumstance of dialogue and explore six relation types which are all categorized as "effects" based on their casual relations. To be more specific, xReact, xEffect and xWant provide the intra-cause clues that represent influences or results generated by utterances from the speaker him/herself. In addition, oReact, oEffect and oWant mean what effects exert on others or what others would like to do and feel after receiving the current utterance. Thus, rich inter-cause clues could be revealed if we take such three relation types into consideration. Figure 2 illustrates causal clues corresponding to the aforementioned six relation types.

How to acquire causal clue from ATOMIC? To acquire intra- and inter-cause clues contained in ATOMIC, we adopt the generative commonsense transformer model COMET [Bosselut et al., 2019] which is trained on ATOMIC. Given the input event (which is referred as an utterance ui under the circumstance of dialogue) and the selected relation type, COMET would generate descriptions of "then" with the format of if-then reasoning. For example, taking ui and the relation type oReact as inputs, a reasoning sequence "If ui, then others would feel" could be derived from COMET. We concatenate ui and a relation with mask tokens such as (ui [MASK] oReact ) to construct the input of COMET. Following [Ghosal et al., 2020], the hidden state representation from the last encoder layer of COMET are taken as causal clue. Thus, for each ui in this work, three pieces of intra-cause clue generated from COMET are concatenated and mapped to the dimension of 2dh with a linear unit. So do the other three pieces of inter-cause clue. We denote them with Clueintra i ∈ R2dh and Clueinter i ∈ R2dh .

Utterance-level Representation

We employ the widely-used pretrained model RoBERTa [Liu et al., 2019] to extract utterance-level feature vectors. Specifically, for each utterance ui = {w1,w2, · · · ,wL}, we concatenate a special token [CLS] to the beginning of the utterance. Then the sequence {[CLS], w1, w2, · · · , wL} are fed to fine-tune the pretrained RoBERTa model with an utterancelevel emotion classification task and the [CLS] token from the last layer is passed through a pooling layer to classify it into its emotion class.

After the process of fine-tuning, to derive the utterancelevel feature vector ci corresponding to the [CLS] token, we pass each utterance in the same input format as {[CLS],w1,w2, · · · ,wL}:

where ci ∈ Rdm and dm is the dimension of hidden states of tokens in RoBERTa. Following [Ghosal et al., 2020], [CLS] tokens from final four layers are averaged to obtain the utterance-level feature vector for each utterance.

Conversational Representation

Under the circumstance of conversational setting, the emotion of an utterance usually depends on the context of the whole conversation. Thus, based on utterance-level features ci, we apply a bi-directional Gated Recurrent Unit (GRU) to model sequential dependencies between adjacent utterances and the conversational representation hi can be computed as:

where hi ∈ R2dh represents the hidden state vector at time step i and dh is the dimension of a GRU cell output.

2.2 Causal Aware Interaction

To attain richer clues for emotional dynamics in conversations and explicitly interact intra- and inter-speaker dependencies, we devise the two-step causal aware interaction, including causal clue retrieval and causal utterance traceback, to enrich context representation with emotion cause.

Causal Clue Retrieval

To explore how much the emotion cause of the target utterance depends on intra- or inter-cause utterances, we should retrieve intra- and inter-cause clue and assign weighted scores to them. For intra-cause clue retrieval, we focus on influences or effects from the same speaker and the retrieval score could be computed as:

where fq(x); fk(x); fe(x) are all linear transformations. maskintra i,j makes sure the target utterance hi retrieve utterances from the same speaker as it to perform intra-cause clue retrieval. It is also worth noting that maskintra i,j guarantee the correct temporal order of retrieval process, which meets the nature of causality that cause could not be found according to causal clues from the future.

where ϕ maps the index of the utterance into that of the corresponding speaker.

The process of inter-cause clue retrieval pay attention to clues contained in utterances from other speakers.

Once retrieval scores from intra- and inter-cause clue are obtained, we should consider them comprehensively on the same scale. The joint value that controls howmuch information should be gathered from intra- or inter-cause utterance can be computed by:

Causal Utterance Traceback

In the step of causal utterance traceback, the model could be aware of different weights to focus more on utterances related to the emotion cause according to results derived from causal clue retrieval. The joint value is decomposed into two parts from the turns of intra- and inter-speaker according to Eq. 4 and Eq. 6, which is denoted as αintra and αinter .

Then causal-aware context representations incorporated with intra-cause utterance and inter-cause utterance could be obtained by:

where S(i) is the set of utterances with the same speaker as utterance ui and O(i) stands for the set of utterances with the speaker different from utterance ui. Further, emotional information included in the causal clue should also be taken into consideration:

where fk(x) is a linear transformation. And final causalaware representation is concatenated by:

2.3 Emotion Recognition

Finally, taking the above causal-aware representation as input, an emotion classifier is applied to predict the emotion of the utterance.

where We and be are trainable parameters.

Cross entropy loss is utilized to train the model and the loss function is defined as:

where E is the number of emotion class and yj i stands for the ground-truth emotion label of the utterance i.

3 Experiments

3.1 Dataset

We conduct experiments on three benchmark datasets from IEMOCAP [Busso et al., 2008], DailyDialog [Li et al., 2017] and MELD [Poria et al., 2019b]. Statistics of the three datasets are shown in Table 1.

IEMOCAP is a dyadic conversation dataset between ten speakers. Each utterance is annotated with one of the following six emotion labels: happy, sad, neutral, angry, excited and frustrated.

DailyDialog contains two-way dialogues covering topics about the daily life. There are seven emotion labels in this dataset: anger, disgust, fear, joy, neutral, sadness and surprise. The dataset has over 83% neutral labels.

MELD is a multimodal dataset with multi-speaker conversations. It is collected from the TV show Friends and the emotion classes belong to anger, disgust, fear, joy, neutral, sadness and surprise.

3.2 Baselines and ComparisonModels

We compare our proposed model with the following methods: ICON [Hazarika et al., 2018] adopts two GRUs to attain utterance representations between two parties. Then a global GRU is utilized for inter-speaker dependency modeling.

DialogueRNN [Majumder et al., 2019] devises three states including global state, party state and emotion state to model intra- and inter-speaker dependencies with GRUs.

DialogueGCN [Ghosal et al., 2019] uses graph convolutional network to model intra- and inter-speaker dependency over the graph structure.

IEIN [Lu et al., 2020] argues that emotions of the utterance are interactive and utilizes the predicted emotion labels to explicitly guide the interaction of emotional dynamics among utterances.

RGAT [Ishiwatari et al., 2020] enhances relation modeling ability of relation-aware graph attention network with the proposed relational position encoding.

COSMIC [Ghosal et al., 2020] models more refined speaker states and utilizes commonsense knowledge to understand emotional dynamics better.

DialogXL [Shen et al., 2021] devises four different types of attention to make the model aware of intra- and interspeaker dependency.

DialogueCRN [Hu et al., 2021] proposes to fully understand the conversational context from a cognitive perspective and designs reasoning modules to integrate emotional clues.

SKAIG [Li et al., 2021] constructs a novel graph to explore psychological states of speakers and graph transformer is used to propagate the interactive information over the graph.

KI-Net [Xie et al., 2021] concentrates on direct utteranceknowledge interaction and involves additional affective information with an auxiliary task.

3.3 Implementation Details

For utterance-level feature extraction, we fine-tune RoBERTa Large model for a batch size of 32 and Adam optimizer is adopted with learning rate of 1e-5. Thus, the dimension of utterance-level feature vector dm is 1024. For all representations in the following parts of CauAIN, dh is set to 300. We train CauAIN with Adam optimizer in a learning rate of 1e-4.

The weighted F1 score is selected as the evaluating metric for IEMOCAP and MELD. Following previous works, we report the micro F1 score excluding utterances annotated with neutral and macro F1 score for DailyDialog. All of our results are averaged on 5 runs.

4 Results and Analysis

4.1 Overall Results

Illustrated in Table 2, our proposed model achieves state-ofthe-art results on all three datasets.

IEMOCAP and DailyDialog. IEMOCAP contains rich information of conversational context with the average conversation length up to 50 turns and the phenomena of emotional dynamics is more frequently observed in this dataset. Benefiting from emotion cause detection to extract deep and rich clues for emotional dynamics, CauAIN achieves new stateof-the-art scores of 67.61 on IEMOCAP. Even if some similar types of external knowledge from COMET is used, we think the reason why CauAIN performs better than COSMIC is that the step-by-step sequential interaction between utterances and lack of considering emotion cause hinder the understanding of rich emotional clues contained in the conversational context. In addition, the improvement of performance over SKAIG demonstrates that emotion cause could provide more direct and richer emotional clues for emotion recognition than psychological information does. Also, our model achieve the best performance on DailyDialog in terms of the macro-F1 score, which demonstrates CauAIN could partly alleviate the influence of data imbalance.

MELD. We observe that CauAINgains slight improvement over recent baseline models on MELD. The reason may be that MELD is a multiple-speaker dataset with short conversations. Thus, utterances that are not spoken to the current speaker would bring noise and useless information for emotion cause detection. It reminds us to further incorporate the discourse structure of conversations and explicitly model the conversation threads in the setting of multi-party conversations.

4.2 Ablation Study

We conduct ablation studies to verify the effectiveness of different components proposed in CauAIN. Results are shown in Table 2.

Effect of Emotion Cause

To investigate the impact of emotion cause for emotion recognition, we remove either the intra- and inter-cause related parts in the model. Specifically, causal clue from ATOMIC is discarded and we do not perform the two-step causal aware interaction. The performance has a certain degree of decline on all three datasets, which is displayed in the second-to-last row and the third-to-last row in Table 2. This suggests that both intra- and inter cause play a significant role in offering rich emotional clues for emotion recognition. Also, it manifests the effectiveness of explicit and thorough intra- and inter-speaker dependency modeling with the aid of intra- and inter cause detection. Besides, it is worth mentioning that the margin of dropped results perform a notable difference on MELD, where only considering inter cause would result in attending more useless contextual information that confuses the model. It is in accordance with the analysis in section 4.1 that additional information of conversation thread should be considered for multi-party conversations.

Effect of Causal Clue

To verify the effectiveness of the generated causal clue, we first analyze its impact on emotion cause detection. Since there is no emotion cause label annotated on IEMOCAP, we randomly select 100 utterances from the test set for human evaluation. Utterances corresponding to top three weighted values from the output of causal clue retrieval αjoint are chosen as the candidate of emotion cause. And we ask human annotators to judge whether true casual utterances belong to the candidate set. Accuracy is adopted as the evaluation metrics. With the aid of causal clue, the emotion cause related modules of our proposed model finally achieve an accuracy of 60%. When discarding the generated causal clue, the process of causal aware interaction degrades to the plain interaction between conversational utterances and the accuracy of emotion cause detection is reduced to 53%. Although the performance of modules related to emotion cause are not significant enough, the introduction of causal clue does help detect exact causal utterances in conversations and the whole process does not rely on any emotion cause annotation.

Further, the impact of causal clue on emotion recognition is shown in the last row in Table 2. The model without causal clue means that we do not introduce the generated causal clue from COMET and utterance interaction is just performed between pure context. From the previous analysis, it could be observed that the introduced causal clue help the model detect more accurate emotion cause. And the dropped results on ERC datasets not only demonstrates that the performance of emotion recognition is benefit from incorporating exact emotion cause, but also shows the effectiveness of enriching context representation with latent emotional information contained in the causal clue. And we believe that both emotion recognition and emotion cause detection would perform better if the golden label of emotion cause is available.

4.3 Case Study

In Figure 4, we exemplify a case from the test set of IEMOCAP to demonstrate the deep and rich emotional information brought by casual utterances when it comes to detect the emotion of the target utterance. The situation is that the woman lose her baggage in the airport and ask an official staff for help. It could be easily observed that no direct emotion descriptor words are contained in the target utterance #20. Thus, its carrying emotion Frustrated should be inferred from the conversational contexts. By performing intra-cause clue retrieval and intra-cause utterance traceback, utterances #1 and #16 are located as reasonable intra-cause utterances, where utterance #1 sets the initial emotional state of worry and #16 emphasizes the importance of baggage for the woman. Meanwhile, the aggravation of the woman's frustration is caused by inter-cause utterances #15 and #17 that the man has no idea about the lost baggage and could not offer effective help for the woman. These exact causal utterances and latent emotional information contained in causal clue provide our model with deeper and richer clues for right predictions.

5 RelatedWork

Emotion Recognition in Conversations. Recent works on ERC devote to model intra- and inter-speaker dependency to cope with emotional dynamics. We summarize them into two categories according to whether external resources such as commonsense knowledge or auxiliary tasks are incorporated.

For those without introducing external resources, various types of deep neural network are leveraged for context modeling. 1) RNN. Majumder et al. [2019] consider global state, party state and emotion state of speakers and utilize three GRUs to model intra- and inter-speaker dependency. Hu et al. [2021] explore cognitive factors of speakers and devise a cognitive reasoning module to iteratively capture emotional clues contained in the context. 2) GNN. Ghosal et al. [2019] model the interaction among speakers with information of nodes and edges propagating over a graph. Further, Ishiwatari et al. [2020] refine types of edges considered in the graph and propose relational position encoding to enhance relationaware graph attention network.

More recently, many works turn to external resources for additional knowledge to guide contextual modeling. Ghosal et al. [2020] explore commonsense knowledge to better understand aspects of conversation such as personality, events, mental states and intents. Based on this, Li et al. [2021] utilize external knowledge to model psychological interactions between utterances. Besides, an auxiliary task named sentiment polarity intensity prediction is introduced to involve more affective information directly [Xie et al., 2021].

Emotion Cause in Conversations. Poria et al. [2021] define two sub-tasks of recognizing emotion cause in conversations and provide a corresponding dialogue-level dataset named RECCON.

6 Conclusion

In this paper, in order to capture deeper and richer clues for emotion dynamics and explicitly model intra- and interspeaker dependency , we propose novel Causal Aware Interaction Network (CauAIN) for emotion recognition in conversations. To be more specific, we explore the effectiveness of incorporating emotion cause when recognizing the emotion of the target utterance. Commonsense knowledge is leveraged as causal clues to help automatically extract causal utterances and alleviate the limitation brought by lack of emotion cause annotation. Then two-step causal aware interaction including causal clue retrieval and causal utterance traceback is devised to detect the intra and inter emotion cause corresponding to the target utterance. And causal aware contextual representation is obtained for the emotion recognition. Experimental results on three benchmark datasets demonstrate the effectiveness of proposed CauAIN and its ability to detect exact emotion cause.

For future work, we would like to improve the performance of causal aware model under the circumstance of multi-party conversations because useless information frommulti speakers is not effectively filtered in our current method. Moreover, conversation resources with emotion and emotion cause annotated would be explored to jointly perform recognition of emotion and emotion cause at the same time.

Acknowledgments

We thank the anonymous reviewers for their insightful comments and suggestions. This work was supported by the National Key RD Program of China via grant 2020AAA0106501 and the National Natural Science Foundation of China (NSFC) via grant 62176078.