Estimating Causal Effects of Tone in Online Debates

1 Introduction

Debates on online forums or social media sites provide observational data for studying discourse. Current understanding draws upon theories such as linguistic accommodation, which states that dialogue participants change and vary their wording styles to mirror one another [Gallois and Giles, 2015; Giles and Baker, 2008]. Statistical methods applied to social media have shown evidence of linguistic style accommodation [Danescu-Niculescu-Mizil et al., 2011], power dynamics [Danescu-Niculescu-Mizil et al., 2012b] and varying persuasiveness of argumentation styles [Tan et al., 2016].

In this paper, we focus on online debates. We ask the causal question of how the tone used to reply in a debate affects changes in linguistic style and sentiment. To illustrate the setting, consider a snippet of a debate between two users, A and B, on a given topic. User A posts her opinion on the topic to which user B replies with a nasty tone. User A writes a second post, responding to B's post. The goal is to examine the change in A's sentiment or linguistic style between her first and second post. For example, we may observe that between her first post and her second post in response to B, A's negative sentiment increased. We study how A's sentiment might have changed had B been nice instead of nasty in the reply. We consider such sequences of three posts within debates and cast the tone of the first reply as the treatment. Formally, we estimate the average treatment effect of reply tone on changes in sentiment and linguistic style.

The challenge for this estimation is that the ideologies encoded in A and B's posts, and A's initial sentiment affect both B's reply tone and A's subsequent response. For example, consider a debate between A and B on gun control. Examples of opposing ideologies that influence the debate are strong opposition to gun violence versus strict interpretations of the constitution. Ideological differences or innate negativity from A provoke both B's nasty tone and A's subsequent reactions. Valid causal inference requires adjusting for these confounders when estimating the treatment effect. While sentiment analysis tools are available for extracting posts' sentiment, modeling the latent ideologies that underpin a particular debate requires careful consideration. This paper proposes representations of ideologies learned from debates to adjust for confounding.

In recent social media analyses, adjusting for attributes such as discussion topic, post authors, timing of posts and posting frequency has been useful to understand post likeability, antisocial behavior and emoji use [Tan et al., 2014; Jaech et al., 2015; Cheng et al., 2015; Pavalanathan and Eisenstein, 2015]. The adjustments are typically performed by only comparing posts that have similar values of the confounder. Our approach requires adjusting for the underlying facets of a debate, an unobserved and multi-dimensional confounder. We use a generative model of text to learn latent representations of posts to this end.

Main Idea. The goal of this paper is to estimate the causal effects of tones used in debate replies on other users' change in linguistic style and sentiment. We identify treatments (tone) and outcome representations which capture the change in sentiment and linguistic style across sequences of posts. We use three plug-in estimators of average treatment effects: regression, inverse propensity weighting (IPW), and augmented IPW. To adjust for confounding in these estimates, we find latent representations of posts that capture the underlying ideological viewpoints of the debate. Our contributions include:

• Formulating the problem of estimating the effects of tone on subsequent dialogue within the framework of causal inference.
• Learning latent representations of ideologies in debates from generative language models to represent confounders.
• Validating the consistency of estimated effects using three different estimators and examining multiple tones of reply in online debates.

We study 4FORUMS.COM, an online debate forum corpus that includes annotations for multiple reply tones including nasty versus nice, or emotional versus factual [Walker et al., 2012b]. Through comparisons against a naive confounder representation and studies across reply tones, we examine the implications of various modeling choices. With these findings, we highlight guidelines for estimating treatment effects using text from social media. We also find that factual replies significantly affect how users' vary their linguistic style and sentiment between posts.

2 RelatedWork

Prior work on online debate forums primarily focus on supervised prediction tasks. Debate text and reply structure between users has been used to predict stance, sentiment and reply polarity [Abbott et al., 2011; Walker et al., 2012a; Hasan and Ng, 2013; Sridhar et al., 2015; Misra and Walker, 2013; Rosenthal and McKeown, 2015]. Related work has also used the Change My View forum on Reddit.com to predict persuasiveness from styles of argumentation and characterize logical fallacies [Tan et al., 2016; Wei et al., 2016; Habernal et al., 2018].

Similarly, unsupervised methods have been applied to analyze dialogue. Statistical models have been proposed to quantify linguistic accommodation both on Twitter and U.S. Supreme Court arguments [Danescu-Niculescu-Mizil et al., 2011; 2012b]. In contrast, we formulate an approach based on causal inference.

Existing work on applying causal inference methods to social media focuses on controlling for confounding, or inferring treatments and outcomes from text. One line of work controls for observable confounders such as topic [Tan et al., 2014; Jaech et al., 2015], timing of posts [Jaech et al., 2015] and the post author [Tan et al., 2014]. Another line of research uses social media posts to estimate the effect of exercise on mood by inferring both exercise habits and mood from text [Dos Reis and Culotta, 2015; Olteanu et al., 2017]. In a different line of work, embeddings of text have been used as proxy confounders to study causal effects on paper acceptance [Veitch et al., 2019].

3 Technical Background

We review estimating the average treatment effect (ATE) for binary treatments from observational data. We have n iid observations called units, i = 1 . . . n. Each unit is treated or not, and we denote this treatment assigment Ti ∈ {0, 1}. We say Yi(1) is the potential outcome if we treat unit i (set Ti = 1), and analogously, Yi(0) if we do not treat i (set Ti = 0). The average treatment effect (ATE) compares potential outcomes:

However, we only observe one outcome for each unit, conditioned on its assigned treatment Yi(Ti)|Ti. If we compute the ATE above by simply averaging over treated and untreated populations, the estimate will typically be biased because Y (0), Y (1) are not independent of the assigned treatments T. Put simply, knowing the treated and untreated units gives us information about their outcomes.

In observational studies, this bias occurs because variables Z, called confounders, may affect both the treatment and outcome. If we observe the confounders for each unit, Zi, then we have Y (0), Y (1) ⊥ T|Z, the condition called ignorability. In this case, the ATE, which we denote ψ, is identifiable as a parameter of the observational distribution by a theorem called adjustment:

In plain English, the ATE is: how do treated and control units differ in outcome when we average over the varying rates at which units receive treatment? We refer to work by Pearl and Rubin for an in-depth treatment of causal inference [Pearl, 2009; Rubin, 2005].

3.1 Estimators for ATE

Drawing from extensive work on estimating ATEs, we present three estimators for ψ. We will return to using these estimators in our empirical study. The first estimator fits expected outcomes Q(Z, T) = E[Y |Z, T] from the observations, e.g., with linear regression. The corresponding ATE estimate, ˆ ψMLE is:

The second estimator reweights observed outcomes using the propensity score, P(T = 1|Z). The resulting inverse propensity weighting (IPW) estimator is:

The final estimator, augmented-IPW (AIPW), interpolates between the two estimators [Robins et al., 1994; Van der Laan and Rose, 2011]. It has been shown that the AIPW estimator satisfies double robustness: it retrieves consistent estimates if either the propensity score or outcome model is correct even if the other is misspecified. It is:

All three estimators rely on measurements of confounders Z. We will see that in debate threads, we must recover the confounders from high-dimensional text. A key idea of this paper is to use text data to find a representation for Z.

4 Dataset

To estimate the ATE of tone, we use the 4FORUMS.COM corpus collected and annotated as part of the Internet Argument Corpus [Walker et al., 2012b]. 4FORUMS.COM has been used to predict users' stances on topics, disagreements between users, and sarcasm use [Lukin and Walker, 2013; Walker et al., 2012a; 2012c; Sridhar et al., 2015].

4FORUMS.COM is a collection of debate discussions, each belonging to a topic such as "evolution" or "climate change." For some pairs of posts called quote-response pairs, 4FORUMS.COM includes annotations about the reply, obtained using Amazon Mechnical Turk. A quote-response pair is a post and its reply where the replier quotes the original poster and responds directly to the quoted statement. The response is annotated by multiple annotators along four dimensions which we refer to as reply types: nasty/nice, attacking/reasonable, emotional/factual, questioning/asserting. Each reply type has two opposing polarities (e.g., nasty or nice) which we refer to as its tone. The annotation score for each type ranges from -5 to 5, where negative values correspond to the antagonistic tone such as nasty or attack and positive values map to tones such as reasonable or factual.

We select the four debate topics with the most quoteresponse annotations: "abortion", "gay marriage", "evolution", and "gun control". Each debate topic has on roughly 1200 quote-response annotations. Following prior work, we use the mean score across annotators and discard annotations with a mean score between -1 and 1 [Misra and Walker, 2013; Lukin andWalker, 2013; Sridhar et al., 2015]. In the next section, we formalize the use of these annotations as treatments to estimate causal effects.

5 Problem Statement

To study causal effects in debate threads, we first introduce post triples. A post triple ti = (p1i , p2i , p3i ) is an ordered sequence of three posts where each post pji belongs to the i-th triple and appears j-th in the sequence. The author of post pji is denoted by aji . The triples we consider have the property that a1i = a3i , i.e., the same user authors the first and last posts. Based on the discussion in which the triple appears, the triple ti has a debate topic τi. We will refer to p1i as the original post and to p2i as the reply post.

Each triple we study has a quote-response annotation for p2i towards p1i . Given the reply type α of the annotations and its mean score, we binarize the values by considering those ≤ -1 as 0 and ≥ 1 as 1. Replies are thus converted to binary negative or positive tone, such as nasty or nice. For each triple ti and reply type α, the tone of reply post p2i toward p1i gives the treatment assignment Ti = Rα i ∈ {0, 1} for the triple.

The next problem is to quantify the outcome of interest: changes between p3i and p1i after receiving reply p2i . We rely on the Linguistic Inquiry and Word Count (LIWC) tool [Pennebaker et al., 2007]. LIWC is a dictionary which maps an extensive set of English words to categories that capture both lexical and semantic choices. Several text classification and statistical analysis tasks have represented posts with counts of LIWC categories [Anand et al., 2011; Abbott et al., 2011; Walker et al., 2012a; Danescu-Niculescu-Mizil et al., 2011].

We first combine LIWC categories into groups which we call category types that measure positive sentiment, negative sentiment, and linguistic style. Fig. 1 shows each category type. For the sentiment groupings, we select the categories related to positive and negative emotion as listed on the LIWC website. For linguistic style, we use the categories identified in prior work for linguistic style accommodation [DanescuNiculescu-Mizil et al., 2011].

Given a category type, the frequency of words in pji belonging to each category gives a vector representation of the post. We can construct such vector representations for p1i and p3i . Formally, for a category type c and reply type α, the outcome Y c,α i for triple ti is the Euclidean distance between the vector representations for p1i and p3i . This strategy suggests many possible vector representations of posts including word embeddings [Mikolov et al., 2013].

We state the ATE estimatation problem for these debate triples. For all configurations of c and α, we estimate:

This estimates the mean difference in text changes between users receiving a positive-tone reply and those receiving negative-tone ones. The main challenge is to find a representation for Z that captures plausible confounders in debates.

6 Constructing Confounder Representations

In a post triple of interest, the debate topic, latent ideologies of each author within the topic, and sentiment of the original author are confounders. That is, these variables plausibly influence both the treatment (reply tone) and the outcome (change between posts). Prior work has shown that text in political debates can be mapped into a lower dimensional space that corresponds to the moral or ideological facets of that debate topic [Johnson and Goldwasser, 2018; Misra et al., 2015; Iyyer et al., 2014; Boydstun et al., 2013]. Unsupervised approaches have been used to discover word-clusters that correspond to these frames directly from text [Iyyer et al., 2014]. Here, we fit an unsupervised generative model of text to learn ideology representations.

We use the latent Dirichlet allocation (LDA) topic model [Blei et al., 2003] to recover unobserved ideologies. The observations wij are counts of word j in document i. The generative process is:βk ∼ Dirichlet(γ) (6)

Each of the k latent topics βk is a distribution over the words in the vocabulary. Each document-level latent variable θi is a distribution over topics. For a document i, each word wij is drawn by sampling a topic from the document's distribution over topics and then sampling a word from that topic.

The posterior expected values E[θi] and E[βj] converge to the optimal values of θi and βj. This convergence property allows us to substitute E[θi] as a confounder for each post. LDA is fit using variational inference.

We fit LDA for each debate topic τ with the observed word counts across posts from that topic. By conditioning on τ, the confounder representation incorporates both the debate topic and finer-grained ideology. The inferred mean proportions over latent topics E[θi] is the embedding for each post. For each triple ti, we concatenate the embeddings for posts p1i and p2i to include in the confounder Zi. Since the embeddings aim to approximate ideologies, including both p1i and p2i embeddings helps to further deconfound the effect of users' opposing or similar views on tone and word change.

To represent the sentiment encoded in p1i , we use the same LIWC category types as we do for the outcomes. As before, we compute the frequency of words in p1i that belong to each category. This gives us a vector representation of sentiment which we include in Zi.

7 Empirical Results

The difficulty in validating causal effects, particularly in debates, is that there is no ground truth. Typically, causal estimation procedures are validated using simulated data but for text, realistic generative models do not exist. Thus, one of the paper's contributions is developing an evaluation strategy for text-based causal inferences. Our approach is three-pronged: 1) we assess the predictive performance of the key ingredients for estimation, the propensity score and expected outcome models; 2) we manually inspect the latent ideological topic; 3) we compare causal effects across multiple estimators and against a naive confounder representation.

We found that: 1) causal estimation using our confounder representation reduces bias in the ATE estimates compared to using a naive confounder; 2) if we had not compared multiple estimators and instead used a single estimator like the high-variance IPTW (a common practice), we would have incorrectly reported effects; 3) the estimates suggest that emotional/factual and questioning/asserting tones elicit changes in linguistic style and emotion while nice/nasty or reasonable/attacking tones show no effect. Code and data to reproduce all results are available. 1

7.1 Methods andMetrics

Using the latent confounder representation proposed, the goal is to estimate the ATE ψ of reply tone Rα on outcome Y c,α. In the empirical results below, we estimate ψ using three estimators ( ˆ ψMLE , ˆ ψIPW, ˆ ψAIPW ) for all configurations of LIWC category type c and α that yield different treatments and outcomes. We report the unadjusted estimate, [E[Y |T = 1] - E[Y |T = 0], which will be biased. We compare against a naive representation, Z - Debate Topics Only, which only uses the debate topic τi without finer-grained ideologies.

1 github.com/dsridhar91/debate-causal-effects

We fit the propensity score P(T = 1|Z) (used by ˆ ψIPW, ˆ ψAIPW ) with logistic regression using the observed treatments and constructed confounder representations. We fit the expected outcomes, Q(Z, T = 0), Q(Z, T = 1) (used by ˆ ψMLE , ˆ ψAIPW ) with linear regression.

Experimental Setup. Besides the processing of quoteresponse pair annotations described in the Data section, we prepare the posts to fit LDA. We obtain all unigram tokens after lemmatizing and removing stop words from posts across all discussions for a given topic. We retain only those tokens which occur in more than 2% but in fewer than 80% of posts. For each topic, this yields a document-term-frequency matrix of roughly 30, 000 posts and 400 remaining terms after preprocessing. We fit LDA with k = 50 topics. For each reply type, the ATE is averaged over roughly 1500 triples. For the AIPWestimator ˆ ψAIPW , we use a variant proposed to improve finite sample performance [Van der Laan and Rose, 2011].

7.2 Predictive Performance

The first step to validating ATEs is to verify that the models used in various estimators fit the observations well. Table 1 gives the root mean squared error (RMSE) and F1 for the expected outcomes Q(Z, T) and propensity scores P(T = 1|Z), respectively. We perform five-fold crossvalidation on the triples. We report performances for all reply types α but for the outcome model which depends also on the LIWC category type, we show scores for positive sentiment outcomes for conciseness. Our code will reproduce the other RMSEs.

For the positive sentiment expected outcomes, our confounder Z-Full predicts with lower RMSE than Z-Debate Topics Only. The F1 score is also slightly improved by using Z-Full over Z-Debate Topics Only when predicting emotional/factual. While this is reassuring, for causal inference, unbiased estimation is more important than predictive performance. Below, we investigate causal estimation, where ZDebate Topics Only shows undesirable consequences.

Attacking/Reasonable

Emotional/Factual

3.6

2.4

2.7

5.1

0.81

0.69

3.0

2.2

2.7

5.1

0.81

0.72

Questioning/Asserting

3.1

4.4

0.80

2.9

4.1

0.79

Table 1: The expected outcome models (shown here for positive sentiment) using Z-Full generally improves over using Z-Debate Topic

Only. The propensity score model performs comparably in both cases. We evaluate the models using 5-fold cross-validation. For expected

outcomes, we report RMSE and F1 for the propensity score.

Figure 2: Top words across latent topics from each debate topic chosen to illustrate ideologies captured by LDA. The latent topics are suggestive of viewpoints like morality and faith versus science and evidence.

ATE (and Standard Error) for Nasty/Nice

7.3 Latent Ideologies

Even if the ideology representations inferred using LDA are useful for predictive performance above, we carefully inspect the latent topics found by LDA. Since we assumed that ideology is a confounder, we want the latent topics to approximate ideologies. We inspected the top ten words associated with each latent topic across all debate topics. Fig. 2 shows these words for two latent topics from every debate topic as illustrative examples of the ideological views found. For example, in gun control debates, LDA finds topics that align with constitutional rights to bear arms and in evolution debates, there are contrasting topics that align with creationist versus scientific views. Our code includes simple visualization to inspect all latent topics.

7.4 ATE Estimation

We use Z-Debate Topic Only and Z-Full and apply the three estimators ˆ ψMLE , ˆ ψIPW and ˆ ψAIPW . We compare these estimates against the unadjusted estimate. Table 2 shows the ATEs (and standard error) for the nasty/nice reply type. When confounders are missing from adjustment, we expect the estimate to be closer to the biased, unadjusted estimate. Indeed, the results show that using Z - Debate Topics Only, omitting sentiment and ideology, consistently yields estimates which are closer to the unadjusted estimate than using Z - Full. This is a key finding: estimation bias is reduced with the finergrained confounder representation. After adjusting for Z - Full, the effects on dialogue outcomes from nasty/nice tones are not significant.

Table 3: Factual and asserting tones result in significantly decreased linguistic style changes. Factual tones may provoke decreased change in sentiment. We report ATE (and standard error) with bolded numbers when the ATE is significantly greater than zero.

In Table 3, we proceed with our confounder representation Z - Full and study the remaining reply types: attacking/reasonable, emotional/factual, questioning/asserting. The ˆ ψMLE and ˆ ψAIPW estimators find significant effects, particularly for factual and authoritative tones. The ˆ ψIPW estimator yields similar ATE estimates but suffers from high variance. This is another key finding that comparing multiple estimators provides a form of validation: the propensity score-based ˆ ψIPW estimator is known to have high variance, and without the two remaining estimators, we may have concluded that no significant effects occur.

The ˆ ψMLE and ˆ ψAIPW estimators suggest that factual and asserting tones cause decreased changes in linguistic style: users' second posts remain closer to their original posts on average across triples. Further, these estimators suggest that both positive and negative sentiment changes are decreased when the tone is factual. The results suggest that users change their overall sentiment less when responding to factual arguments instead of emotionally charged ones. However, users may also maintain their original linguistic styles more when responding to factual or asserting tones. This finding on the role of factual and asserting tones may point to in-depth followup studies on persuasion and argumentation in debates.

Finally, the empirical studies reveal unexpected findings about causal estimation of treatments effects in debates. Interestingly, the choice of outcome representation matters: Table 3 in particular shows that changes to linguistic style are affected more than sentiment changes. A single outcome representation which had concatenated all LIWC categories or used word embeddings may have yielded different results.

8 Discussion

We study treatment effects in debates by estimating unobserved confounders from sequences of posts. We examine these interpretable embeddings in debates to find that they match known ideological views. The exercise of estimating treatment effects yields results of two flavors: 1) evidence that factual replies cause decreased change in linguistic style and sentiment, and 2) guidelines for practioners to estimate treatment effects from social media text.

We highlight areas of future study. In this work, we focus on ideology and sentiment as confounders. It is interesting to consider possible confounding from posts' timing and position in a discussion thread. A fruitful area of research is learning deep confounder (and outcome) representations for text while maintaining model interpretability, which we show in this paper is important for validating findings. Finally, validating causal effects in questions of social science research remains an open problem. Simulating outcomes from text is a line of future work.

Acknowledgements

This work is supported by NSF grants CCF-1740850 and IIS1703331.