Towards Facilitating Empathic Conversations in Online Mental Health Support: A Reinforcement Learning Approach (Extended Abstract)

1 Introduction

Online mental health support platforms such as TalkLife (talklife.com) are used by millions of users for expressing emotions, sharing stigmatized experiences, and receiving peer support. These platforms might help improve access to mental health support as mental health care remains a global challenge with widespread shortages of workforce [Olfson, 2016] and other barriers like stigma [White and Dorman, 2001]. A key component of providing successful support is empathy, the ability to understand or feel the emotions and experiences of others [Elliott et al., 2011]. Quantitative evidence shows that empathic interactions have strong associations with symptom improvement in mental health support [Elliott et al., 2018] and are instrumental in building therapeutic alliance and rapport [Bohart et al., 2002; Robert et al., 2011]. Yet, highly empathic conversations are rare on online support platforms [Sharma et al., 2020].

Empowering peer supporters on online support platforms with feedback and training, for example through machine-inthe-loop writing systems [Clark et al., 2018; Tanana et al., 2019], has the potential to help supporters express higher levels of empathy and in turn improve the effectiveness of these platforms [Miner et al., 2019; Sharma et al., 2020]. Traditional methods for improving empathy (e.g., in-person training) do not scale to the millions of users of online support platforms. However, computational methods that can support peer-supporters by suggesting ways to modify existing conversation utterances to make them more empathic may help meet this need and indirectly benefit support seekers.

In this paper, we introduce Empathic Rewriting, a new task that aims to transform low-empathy conversations to higher empathy (Figure 1). For example, given a post from a support seeker "I can't deal with this part of my bipolar. I need help." and a low-empathy response "Don't worry! Try to relax. Anyone you can talk to?", we want to increase empathy in the response by transforming it to "Being manic is no fun. It's scary! I'm sorry to hear this is troubling you. Try to relax. Anyone you can talk to?"; the rewritten response should communicate more empathy through an understanding of feelings and experiences ("Being manic is no fun. It's scary") and display of felt emotions ("I'm sorry to hear this is troubling you").

Learning such transformations is challenging and requires a deep understanding of empathy while maintaining conversation quality through text fluency and specificity to the conversational context. Here, we propose PARTNER, 1 a deep reinforcement learning (RL) model for empathic rewriting (Section 3). We design an RL agent which learns to add new empathic sentences to posts or replace existing sentences in posts with more empathic ones. The agent operates on a pair of seeker post and the original response post (which rarely is highly empathic [Sharma et al., 2020]) and makes edits to the response at the level of a sentence by simultaneously (a) identifying positions in the original response post where changes are required, and (b) generating empathic sentences for insertion or replacement at the identified positions (Section 3.2). We model this agent using a policy network based on a transformer decoder model adapted from GPT-2 [Radford et al., 2019]. We build upon existing large-scale pre-training of GPT-2 on conversations, as done in DialoGPT [Zhang et al., 2020], and modify it to perform the two simultaneous actions of identifying positions and generating empathic sentences for empathic rewriting (Section 3.3). Through carefully constructed scoring functions, we reward transformations that increase empathy in posts while maintaining text fluency, context specificity, and diversity (Section 3.4).

Our experiments demonstrate that PARTNER can effectively increase empathy in posts in fluent, specific, and diverse ways and outperforms baselines used in related text generation tasks by > 35% in empathy improvement (Section 4). We view our approach and findings as a key step towards building AI systems for facilitating empathic conversations on online mental health support platforms, but these insights may generalize beyond mental health to other conversational settings on web-based platforms.

2 ProblemDefinition and Goals

2.1 Empathic Rewriting

We introduce empathic rewriting, a new task that aims to transform low-empathy conversational posts to higher empathy. In contrast with empathic dialogue generation [Rashkin et al., 2019], where the objective is to generate empathic posts from scratch, this task requires making changes to existing posts to make them empathic. This is more consistent with realistic use-cases in difficult, high-stakes settings such as online support systems, which are likely to augment, rather than replace humans [Miner et al., 2019]. Formally, let Si be a seeker post and Ri be a corresponding response post. We aim to transformRi into its more empathic counterpart ˆRi.

1 emPAthic RewriTing in meNtal hEalth suppoRt

2.2 Goals

For empathic rewriting to be useful in improving supportive conversations, the rewriting process should achieve specific goals related to empathy, conversation and natural language generation quality, and purposeful and precise feedback:

Theoretically-grounded empathy. Computational research typically defines empathy as reacting with emotions of warmth and compassion [Buechel et al., 2018]. However, psychotherapy research emphasizes aspects of empathy related to communicating cognitive understanding of feelings and experiences of others [Selman, 1980]. For empathic rewriting to be useful and potentially adopted in online mental health support, we need to design methods grounded in psychology and psychotherapy research. Here, we adopt the theoretically-grounded framework of empathy designed by Sharma et al. [2020] as reward signals (Section 3.4)

Context specificity and response diversity. Consider a rewriting approach that transforms every response to a generic but empathic response (e.g., "That must have been really hard for you"). While this approach may seem to "solve" empathic rewriting, it suffers from the issues of low specificity and diversity, which are critical for obtaining purposeful transformations. In this work, we learn rewriting actions that simultaneously achieve the goals of context specificity and response diversity using reinforcement learning (Section 3.4).

Text fluency and sentence coherence. In addition, only generating empathic words or phrases may not be sufficient. Without appropriate measures, the rewriting process may lead to an ungrammatical, non-fluent final response with noncoherent sentences. In this paper, we avoid such responses through carefully constructed reward functions (Section 3.4).

Rewriting for feedback and training. An important way in which the task of empathic rewriting can be used is for providing feedback and training to people through machinein-the-loop writing systems [Clark et al., 2018]. For humans to adopt such feedback, however, the rewriting process should make changes that are precise and minimal. Here, we train a reinforcement learning agent which learns when to stop making changes through a special "stopping" action (Section 3.2).

3 PARTNER: Empathic Rewriting Using Reinforcement Learning

Here, we present PARTNER, a reinforcement learning model for the task of empathic rewriting (Figure 2). Conceptually, our agent leverages context from the seeker post which it uses for making specific empathic changes. Alongside, it operates on the response post, looks for areas where empathy could be improved, and works on those improvements in fluent, coherent, specific, and diverse ways.

3.1 State: Seeker Post &Fixed-Length Contiguous Spans of Response Post

Our agent simultaneously operates on seeker post and fixedlength contiguous spans of response post. Formally, let Ri contain n sentences Ri,1, ...,Ri,n. At each step, we focus on a contiguous window of k sentences starting from the jth sentence Ri,j:j+k = Ri,j, ...,Ri,j+k-1. Then, our state s ∈ S is denoted by the pair (Si, Ri,j:j+k).

3.2 Actions: Sentence-Level Edits

Our agent takes actions at the level of a sentence, i.e. it either inserts new sentences or replaces existing sentences with newer ones. A deletion operation is equivalent to replacing a sentence with an empty string. Our agent can make word-level changes by replacing the original sentence with a slightly different sentence containing only word-level edits. In a state (Si, Ri,j:j+k), our agent simultaneously takes two actions - (a1) select a position in Ri,j:j+k for insertion or replacement, (a2) generate a candidate empathic sentence. The action space A1 of a1 consists of 2k+2 actions - k+1 positions for insertions, k positions for replacements, and one special action to stop the agent from making any further changes.

3.3 Policy

At its core, our policy has a transformer language model consisting of a stack of masked multi-head self-attention layers, based on GPT-2 [Radford et al., 2019]. It takes as input an encoded representation of our state (Si, Ri,j:j+k) and generates the action a = (a1, a2).

(a1) Selecting a position for insertion or replacement. A k sentence window Ri,j:j+k has k+1 positions for insertions and k positions for replacement. Then, our task is to select one of these 2k + 1 positions. For selecting this position, we first encode the input string "Si <SPLIT> Ri,j:j+k" using the transformer block of GPT-2. We then pass this encoded representation through a linear layer to get the prediction ˆpi of the position for insertion or replacement.

(a2)Generating a candidate sentence. We first encode our input string through GPT-2. We then compute a probability distribution over vocabulary tokens by transforming the encoded representation into a vocabulary-sized vector through a softmax layer. Finally, we use top-p sampling [Holtzman et al., 2020] over this probability distribution to generate the desired candidate sentence Ci,j.

3.4 Rewards

Our reward function, r = we∗re+wf ∗rf +wc∗rc+wm∗rm, aims to increase empathy in posts and maintain text fluency, sentence coherence, context specificity, and diversity:

Change in empathy (re). We reward actions that increase empathy of Ri and penalize actions that decrease empathy of Ri. Let fe(·) be a function that measures empathy of posts. Then, the change in empathy reward, re, is defined as fe( ˆRi)-fe(Ri), where fe(·) is estimated using the empathy classification model of Sharma et al. [2020].

Text fluency (rf ). We aim to prevent actions that lead to outputs that are highly empathic but lack text fluency. Therefore, we reward actions that lead to fluent outputs and penalize actions resulting in non-fluent outputs. Here, we operationalize text fluency as the inverse of perplexity of the generated ˆRis.

Sentence coherence (rc). While the candidate sentence that is added to the original response Ri might be highly empathic and fluent, it may not be well-suited for the response, leading to incoherent sentences in the transformed response ˆRi. Here, we design a reward function, rc that measures the average sentence coherence probability between the candidate sentence and existing sentences in the response.

Mutual information for specificity and diversity (rm). In the process of empathic rewriting, the final rewritten response may become generic (e.g., "I understand how you feel") thereby affecting the overall conversation quality. In order to ensure specificity to the seeker post and diversity of responses, we exploit the idea of maximizing mutual information between seeker post and the rewritten response post [Li et al., 2016].

4 Experiments and Results

4.1 Warm-Start Using Supervised Learning

We use the pre-trained weights of DialoGPT [Zhang et al., 2020] for initializing our transformer language model. In addition, we use a warm-start strategy using supervised learning on a parallel dataset of (low empathy, high empathy) pairs.

4.2 Dataset: TalkLife

We use a dataset of mental health conversations from the TalkLife (talklife.com) platform. The dataset contains 10.9M conversational threads and 26.9M pairs of (seeker post, response post) interactions from 642K users.

(C) Qualitative Example

4.3 Results

Automatic metrics. We find that empathic rewriting through PARTNER achieves the largest change in empathy (35% more than the next best baseline, MIME [Majumder et al., 2020]) and is more specific than all baselines (Table 1). Also, MIME generates empathic outputs (+1.21 change in empathy) but the generations have low diversity (86% less than PARTNER) indicating similar responses for most seeker posts. BART [Lewis et al., 2020] generates outputs with lowest perplexity, highest diversity, and lowest edit rate, which is consistent with substantial improvements to language models in recent years [Brown et al., 2020]. However, to our surprise, the rewritten responses through BART receive an overall drop of 0.06 in empathy, indicating that the model is unable to perform the task of empathic rewriting well and typically generates non-empathic, fluent, diverse text.

Human evaluation. Since automatic evaluation in language generation is often not robust, we perform a human evaluation on our key metrics (empathy, fluency, and specificity) through A/B testing. We recruit six graduate students in clinical psychology with expertise in empathy and mental health support and ask them to compare outputs from PARTNER against other baseline models, ablations given the same input. We find that rewritten responses from PARTNER are preferred for empathic and specific responses over all baselines (Figure 3B). DialoGPT is judged more fluent but generates responses following similar templates (e.g., "I'm sorry you.... I hope you....").

BLEU scores. We create a small parallel dataset of 180 pairs of corresponding low and rewritten high empathy response posts with rewritings from people having substantial expertise in empathy, mental health, and therapy (six graduate students in clinical psychology; none are co-authors). We compare outputs of PARTNER and baselines based on this ground truth using the BLEU metric [Papineni et al., 2002] (Figure 3A). We find that the outputs from PARTNER are closest to expert rewritings (86% better than the next best baseline, BART).

5 Discussion and Conclusion

The burden of mental illness globally is overwhelming, and common mental disorders are some of the most debilitating illnesses worldwide [Collins et al., 2011]. Existing mental health resources and interventions are ill-suited to the size of the need. Online mental health support platforms that make use of peer supporters is one route to scaling up support, but the biggest challenge is to effectively train or scaffold the peer supporters. Our empathic rewriting approach represents a foundational proof-of-concept of how computational methods may help peer supporters online.

Rewriting human-generated responses may be an effective approach to balancing the benefits and risks of using artificial intelligence in mental health settings. By combining human knowledge of context and experience, our approach can both provide feedback to online peer-supporters with actionable, real-time examples, and provide support seekers with more empathic responses. Importantly, this machinein-the-loop approach can help mitigate some of the risks related to toxicity and safety of AI systems in settings of suicidal ideation, self-harm, or insensitive comments related to race/ethnicity/gender [Li et al., 2020; Luxton et al., 2012].

(B) Human Evaluation

Acknowledgements

We would like to thank TalkLife and Jamie Druitt for supporting this work and for providing us access to a TalkLife dataset. T.A., A.S., and I.W.L. were supported in part by NSF grant IIS-1901386, NSF grant CNS-2025022, NIH grant R01MH125179, Bill & Melinda Gates Foundation (INV-004841), the Office of Naval Research (#N00014-211-2154), a Microsoft AI for Accessibility grant, and a Garvey Institute Innovation grant. A.S.M. was supported by NIH NCATS grants KL2TR001083 and UL1TR001085 and the Stanford Human-Centered AI Institute.