A Murder and Protests, the Capitol Riot, and the Chauvin Trial: Estimating Disparate News Media Stance

1 Introduction

The thirteen month period spanning the murder of George Floyd by police officer Derek Chauvin on May 25, 2020, the Capitol Riot on January 6, 2021, Chauvin's conviction for the Floyd murder on April 21, 2021, and his sentencing on June 25, 2021 were momentous events for policing in the United States in part due to the sustained media attention given to police practice, conduct and function. These four events, however, cast the police in diametrically different roles. The Floyd murder and Chauvin's conviction and sentencing focused attention on police use of excessive force, especially against Black Americans, whereas the Capitol riot cast police in the role of valiant warriors in the defense of the nation's political foundation.

In this paper, using advanced natural language processing (NLP) methods, we analyze the different responses of three major media outlets, Fox News, CNN and MSNBC to these seminal events. The differences that we document are consistent with the partisan divide in the viewership of the three outlets. According to a Pew sponsored survey 1 , 93% of Republicans or those leaning Republican name Fox News as one of their preferred sources of political news whereas only 6% of Democrats or those leaning Democrat name Fox as a preferred source. For CNN and especially MSNBC, the percentages flip, CNN and MSNBC are preferred sources, respectively, for only 4% and 1% for Republicans/lean Republicans and for Democrats/lean Democrats they are, respectively, 79% and 95% preferred sources.

∗Daniel S. Nagin and Ashiqur R. KhudaBukhsh are joint corresponding authors.

1 https://www.pewresearch.org/fact-tank/2020/04/01/americans -main-sources-for-political-news-vary-by-party-and-age/

While the key focus in this paper is to address the specific task of estimating aggregate media stance on policing, our work has valuable potential for advancing AI for social good. In particular, we demonstrate how advanced NLP and machine learning technologies can help policy makers, content producers, and general public better understand the political disparity and partisan divide in the content creation of major media outlets. The mode of analysis that we demonstrate could just as well be applied to other contentious issues such as abortion, immigration, and gun control that divide contemporary America. Casting the task of estimating stance as a natural language inference (NLI) task [Bowman et al., 2015], our study proposes a robust, scalable, and measurable indicator - media entailment ratio - for measuring the disparate stance of different major media outlets toward seminal societal events. It also provides a sustainable lens through which we can better understand, access, and evaluate the news content and public opinion. While focusing on three high-profile policing and justice events fromMay 2020 to June 2021, the proposed methods and analyses can also be generalized to other contexts for social good.

To summarize, our contributions are the following:

• Social: We analyze the different responses of three major news outlets to momentous events for policing using sophisticated NLP methods. To our knowledge, our analysis is the first to show that across cable networks coverage of politically salient events responds quickly and dramatically to the partisan preferences of their viewership.
• Method: Casting the problem of aggregate stance mining as a natural language inference task [Bowman et al., 2015], we present an active learning pipeline drawing from existing sampling strategies [Sindhwani et al., 2009; Scheffer et al., 2001; Palakodety et al., 2020] and a novel strategy, dubbed inconsistency sampling.
• Resource: We release a sentence-level, domain-specific text entailment data set on policing consisting of 2,276 anno-

tated instances 2 .

2 Data Set

We consider news transcripts from three major US cable news networks: Fox News, CNN, and MSNBC. We scrape all transcripts available on their official web sites uploaded between May 25 2019 and July 24 2021. We divide this time period of 26 months into four non-overlapping temporal slices:

1. Pre-George Floyd (denoted by Tpre-Floyd ): baseline time period starting on May 25 2019 and ending on May 24 2020.
2. George Floyd's murder, protest, and civil unrest (denoted by TFloyd): starts on May 25 2020, the day George Floyd was brutally murdered, and ends on Dec 31 2020.
3. Capitol insurrection (denoted by Tcapitol): starts on January 1 2021 and ends on March 7 2021, a day before the trial of Derek Chauvin begun.
4. Chauvin's conviction and sentencing (denoted by Ttrial): starts on March 8 2021, the day Chauvin's trial begun and ends on July 24 2021, one month after Chauvin's conviction.

Preprocessing. Next, we preprocess these transcripts and remove all text corresponding to video clips indicated by "(BEGIN VIDEO CLIP)" and "(END VIDEO CLIP)" marks and any non-utterance, such as time information (e.g., [12:15:05]) and special marks (e.g., "(COMMERCIAL BREAK)"). Overall, we obtain 34,710 transcripts (CNN: 28,163, Fox news: 3,617, and MSNBC: 2,930 transcripts). We also remove the speaker information (e.g., HANNITY, COOPER or MADDOWS) preceding any sentence since their presence may bias machine learning models' predictions triggering shortcut learning [Geirhos et al., 2020].

We observe that compared to MSNBC and Fox, the news transcript archiving of CNN is substantially more detailed and comprehensive. The temporal distribution of the news transcripts is summarized in Figure 1. As shown in Figure 1, the relative fraction of uploaded transcripts (with respect to an individual news network) remained comparable throughout all four time slices.

Next, we construct three corpora, one from each news network transcripts, consisting of all sentences that contain the token police (denoted by Dfox , Dcnn, and Dmsnbc). We argue that this is a high recall approach to filter content discussing policing in the US without biasing the data set by filtering by topic or event related keywords. To support this, we note that MSNBC and Fox News have roughly equal number of sentences obtained through this filter. However, the number of sentences that mention Floyd is strikingly different (MSNBC: 713 sentences, Fox news: 198 sentences).

2 Data and additional resources are available at the project page: https://github.com/Suji04/IJCAI2022-Media-Stance.

2.1 Separability Tests

When we manually inspect these transcripts, we see systematic differences. Here are three examples from Dmsnbc, Dfox, and Dcnn, respectively:

1. Unarmed black person committing no crime shot and killed by police.
2. Number one, it's the media activists class that took a traffic - tragic incident and cast all police as corrupt which caused the Ferguson effect.
3. We've all seen the image of that capitol police officer who led that mob away from the Senate chamber.

However, human biases on contentious social issues is well-documented [Breitfeller et al., 2019]. Unlike human beings, machines are less likely to have labeling biases. We ask ourselves how do we quantify this difference (if any) using automated methods? Quantifying differences between largescale text data sets is an emerging field and existing methods often require a vast amount of data [KhudaBukhsh et al., 2021]. We instead conduct a simple experiment using classification accuracy as a proxy for text separability. Our instances are individual sentences with the token police from different networks, and the labels are the news networks. Our classifier takes a sentence with the token police from news transcripts as input, and outputs the predicted source news network. Our intuition is if the linguistic signals present in two networks' reporting on policing are distinguishable, the classifiers will be able to predict the source news network with high accuracy. We randomly sample equal number of examples from any two news networks and construct binary classifiers to predict the news network label. Note that, if the reporting has little or no separable linguistic signal, the classifier will not perform much better than chance (50% accuracy on an evenly balanced test set).

It may well be the case that Fox News typically uses long sentences as opposed to short sentences used by MSNBC or CNN. To verify if any such quark present in the data set is influencing the classification accuracy through triggering shortcut learning [Geirhos et al., 2020], as control, for each news network pair, we train another classifier on a training set of randomly selected sentences from news transcripts i.e. without the restriction that the sentences must contain the token police. The control allows us to compare how randomly selected sentences from two different news networks are distinguishable as opposed to sentences with the token police.

Table 3 summarizes our separability results. We first compare the prediction accuracy on sentences with the token police in them. We notice that the classification tasks involving Fox news yielded substantially higher accuracy than the classification task involving the pair ⟨CNN,MSNBC⟩. Hence, CNN and MSNBC's portrayal of police could be linguistically more akin to each other than Fox news. We find that distinguishing between randomly chosen sentences is considerably more challenging than distinguishing between sentences related to police. The classification accuracy on randomly chosen sentences does not differ by much regardless of the choice of the network pair. Hence, the general nature of news coverage of Fox is less distinct than the network's coverage of police related events. In Table 2, we consider the classification tasks involving the network pairs ⟨CNN, Fox⟩ and ⟨Fox,MSNBC⟩ and list a few illustrative examples where the classifier correctly predicts the network label with high confidence.

Our classification results indicate that Fox's portrayal of policing is considerably different from CNN and MSNBC. In Section 5, we present critical insights into how these portrayals are different and provide methodological details and necessary technical background in Section 4.

3 RelatedWork

Beyond the rich social science research on differential policing and media portrayals [Chaney and Robertson, 2013; Lawrence, 2000; Brunson, 2007], previous attempts to quantify the differential treatment of police towards the white and black communities have employed computational linguistic methods on the transcripts of videos captured from police body cameras [Voigt et al., 2017]. Voigt et al. [2017] established a linguistic framework to estimate the level of respect in the police officer's speech and found that officers speak with significantly less respect to black people than white. In contrast, our work focuses on differential media portrayals of police in prominent US cable news networks and how such portrayals reflect on news consumer responses.

Several recent lines of work have analyzed discussions around controversial issues on mainstream and social media and how political bias influences framing [Demszky et al., 2019; Sap et al., 2020; Roy and Goldwasser, 2020]. Demszky et al. [2019] provide an NLP framework to quantify and analyze political polarization in 4.4M tweets on 21 mass shooting events. A weakly supervised learning approach was proposed in Roy and Goldwasser [2020] to analyze the nuanced frames present in news articles on contentious topics (abortion, immigration, and gun control). In Mendelsohn et al. [2020], the authors analyzed the presence of dehumanizing language in discussions related to the LGBTQ community in the New York Times using computational linguistics methods. The authors applied supervised learning methods on a novel immigration-related tweet data set to detect the frames.

This study also analyzed how users' region and ideology correlate with the framing choices and how other users react to their posts. While these lines of research focus on a broad range of vital issues that include mass shooting [Demszky et al., 2019], abortion [Roy and Goldwasser, 2020], immigration policy [Mendelsohn et al., 2021] and dehumanization [Mendelsohn et al., 2020], an extensive analysis of the media stances on the issue of police reform does not exist.

Unlike prominent supervised and unsupervised approaches to mine stances [Darwish et al., 2020], in line with a recently proposed approach [Hossain, 2021], we treat the problem of stance mining as a natural language inference (NLI) task. NLI to mine stances has been deployed in another recent work which is similar to our work both in terms of method and research focus [Halterman et al., 2021]. Halterman et al. [2021] explored the role of police in the Gujrat riot (2002, India) by analyzing 1, 257 news articles using BERT-based language models. Our work contrasts with Halterman et al. [2021] in three key ways. First, we analyze media portrayals of police in multiple seminal events over longitudinal corpora spanning thirteen months. Second, instead of focusing on a single news source, we consider multiple news outlets and conducts a contrastive analysis. Finally, we present a label-efficient active sampling method that combines multiple active learning strategies and a novel sampling technique that exploits logical consistency. The susceptibility of textual entailment systems to negation is well-documented in Kang et al. [2018]. Our work leverages this knowledge to set up an effective active learning pipeline detecting challenging premises that produce logically inconsistent entailment inferences.

4 Technical Background andMethods

4.1 Text Entailment

Our proposed approach casts the problem of mining stance as a natural language inference (NLI) task [Dagan et al., 2005]. Given a premise P and a hypothesis H, the NLI task involves predicting entailment, contradiction, or semantic irrelevance. Textual entailment is much more relaxed than pure logical entailment and can be viewed as a human reading P would infer most likely H is true. For example, the hypothesis some men are playing a sport is entailed by the premise a soccer game with multiple males playing 3 .

Following Halterman et al. [2021], we cast the task of estimating the aggregate stance on policing as an NLI task. We consider two semantically equivalent hypotheses:

1. Police protect us (denoted by Hprotect)
2. Police make us safe (denoted by Hsafe)

and estimate the endorsement or support for police through entailment ratio described next.

Entailment ratio. Let NLI(P,H) takes a premise P and a hypothesis H as inputs and outputs o ∈ {entailment, contradiction, neutral}. For a corpus Di and a hypothesis H, we compute the entailment ratio (denoted by ent(Di, H)) as the fraction of the individual sentences present in Di that entails H:

3 This example is taken from Bowman et al. [2015].

ent(Di,H) = ΣP∈Di I(NLI(P,H)=entailment) |Di | where I is the indicator function. The larger the value of ent(Di,H) is the greater is the support for H in the corpus.

While state-of-the-art, off-the-shelf NLI systems can perform well on a broad range of examples, relying on NLP systems to draw conclusions on a subject of high social impact needs careful consideration [Hovy and Spruit, 2016; Olteanu et al., 2019]. In our work, we are interested in estimating the support for police in each of the corpora, a domain considerably different from the data on which our NLI system is trained on [Bowman et al., 2015]. We thus finetune the existing NLI system on domain-specific examples. In order to make the learning process label efficient, we use active learning [Settles, 2009], a well-known supervised machine learning technique described next.

4.2 Active Learning to Strengthen the NLI Model

Active Learning is a powerful and well-established form of supervised machine learning technique [Settles, 2009]. It is characterized by the interaction between the learner, aka the classifier, and the teacher (oracle or labeler or annotator) during the learning process. Pool-based active learning is a popular variant; in this setting, the learner is initially trained on a small seed set of labeled examples and has access to a large collection of unlabeled samples. At each iteration, the learner employs a sampling strategy to select an unlabeled sample and requests the supervisor to label it (in agreement with the target concept). The dataset is augmented with the newly acquired label, and the classifier is retrained on the augmented dataset. The sequential label-requesting and re-training process continues until some halting condition is reached (e.g., annotation budget is expended or the classifier has reached some target performance). At this point, the algorithm outputs a classifier, and the objective for this classifier is to closely approximate the (unknown) target concept in the future. The key goal of active learning is to reach a strong performance at the cost of fewer labels through an active participation of the learner. Since training and inference on a very large pool of samples can be computationally prohibitive, lines of work have examined the trade-offs of batch active learning [Yang and Carbonell, 2013]. In this paper, we follow the pool-based batch active learning pipeline where instead of requesting one label at a time, the learner requests labels for a batch of unlabeled samples.

Our active learning pipeline draws inspiration from Palakodety et al. [2020] and uses multiple sampling strategies to construct an evenly balanced train set with diverse and challenging examples. Since our paper's main thrust is analyzing a research question of high social impact, in what follows, we present a succinct description and rationale behind our sampling strategies.

Random sampling. In order to capture a diverse set of examples, we include 900 randomly sampled instances giving equal weightage to each of the four temporal slices and three news networks. These 900 instances form the seed set of our active learning pipeline. In this phase, we obtain 160 entailments, 202 contradictions and 538 neutrals confirming that neutral is the majority class.

Certainty sampling. Minority class certainty sampling is useful in rectifying high-confidence misclassifications involving short documents [Sindhwani et al., 2009]. We conduct minority class certainty sampling and added 200 instances that the model predicts entailment or contradiction with highest confidence.

Inconsistency sampling. This novel task-specific sampling strategy is guided by the intuition that instances where the model's predictions are logically inconsistent could be potentially challenging. We first note that Hsafe is a semantic variation of Hprotect and ∀p, NLI(p, Hsafe) = NLI(p, Hprotect). We further note that the same premise cannot both entail or contradict the hypothesis and its negation. Let H′ denote the negation of a hypothesis. ∀p such that NLI(p,H) ∈ {entailment, contradiction}, NLI(p,H) ̸= NLI(p,H′ ). We leverage these two observations to construct a novel active learning sampling strategy dubbed inconsistency sampling. We include 336 instances on which the model exhibits these two types of logical inconsistencies. To ensure that the data is balanced across different time slices and networks, while conducting sampling, we equally weight each network (CNN, Fox news, and MSNBC), each hypothesis (Hprotect and Hsafe), and each of the four temporal slices (Tpre-Floyd, TFloyd, Tcapitol, and Ttrial), each of the two types of logical inconsistencies and sampled from every combination of a network, hypothesis, logical inconsistency, and temporal slice.

Uncertainty sampling. Uncertainty sampling is one of the most well-known sampling strategies used in active learning [Settles, 2009]. Since we have multiple label categories in our prediction task, we follow the uncertainty sampling variant, margin sampling, proposed by Scheffer et al. [2001].

To summarize, our active learning pipeline consists of the following steps:

1. Construct an initial seed set of 900 instances (Dseed : 158 entailment, 204 contradiction and 538 neutral instances) using random sampling.
2. Conduct minority class certainty sampling and add 200 samples (Dcertainty : 91 entailment, 86 contradiction and 23 neutral instances).
3. Conduct inconsistency sampling and add 336 samples (Dinconsistency : 86 entailment, 123 contradiction and 127 neutral instances).
4. Finally, conduct uncertainty sampling (margin sampling) and add 540 samples (Duncertainty : 189 entailment, 199 contradiction and 152 neutral instances).

Overall, we obtain 1,976 instances (526 entailment, 610 contradiction, and 840 neutral instances). All annotations were conducted by two annotators in two phases with the two annotators independently annotating in the first phase followed by an adjudication step to resolve disagreements in the second phase. Before the adjudication process, Cohen's κ value was 0.82 indicating high inter-rater agreement.

We evaluate the performance of our trained models on an unseen evaluation set of randomly sampled 300 instances (Dtest: 69 entailment, 67 contradiction and 164 neutral instances) giving equal weightage to each of the four temporal slices and three news networks. We consider the base entailment model (Mbase), an NLI model published by Allen NLP 4 , as our baseline. We in fact fine-tune this model using our data obtained through our active learning pipeline (the project page contains further details). As show in Table 4, (1) our model performs reliably on unseen data; and (2) our method considerably outperforms the baseline underscoring the need for training on domain-specific examples.

5 Results and Discussions

Figure 2 summarizes the findings of our entailment analysis of the hypothesis Hprotect. Results for the hypothesis Hsafe are very similar. Each bar measures the proportion of sentences classified as endorsing the hypothesis. In the year prior to George Floyd's murder the entailment ratios for the three networks were very similar at about 0.2. Thereafter, they dramatically diverged. For period from Floyd's murder to the end of 2020 which included peaceful protests against police brutality but also riots in many cities the entailment ratios of CNN and MSNBC declined by 36.5% and 45.2%, respectively, whereas that for Fox it increased by 21.3%. The large reductions in agreement to the hypothesis "police protect us" for the Democrat favored outlets CNN and MSNBC and the large increase in support for the premise for the Republican favor outlet Fox is consistent with news commentaries on partisan difference in reaction to the Floyd murder and ensuing events during the period. For example, a June 1, 2020 analysis appearing in the Washington Post 5 observed: "There are two things about the protests against police brutality ripping through the nation that Democratic and Republican lawmakers generally agree on:

4 https://github.com/allenai/allennlp-models/blob/main/allennl p models/modelcards/pair-classification-roberta-snli.json

1. George Floyd should not have died the way he did, after a white officer knelt on his neck.
2. Looting, vandalism and violence that have dominated some protests after dark are bad.

But on social media, in statements and in interviews, there's a clear partisan difference between which of these two points members of Congress choose to emphasize, with Republicans choosing to focus on the second while Democrats the first."

The period January 1, 2021 to March 7, 2021 spans the Capitol Riot, in which police were both the principal victims and heroes, up to the conviction of Derek Chauvin. During this period the entailment ratio for Fox is unchanged whereas those for CNN and especially MSNBC increase dramatically by 82.4% and 145.9% respectively. The change differences for Tcapitol are consistent with the partisan differences which persist to this day in the interpretation of the events of January 6 with the Republican National Committee now characterizing the events of January 6 as an exercise of "legitimate political discourse." The fourth period from March 8, 2021 to July 24, 2021 covers the conviction and sentencing of Derek Chauvin. This period was associated with a 33.1% increase in the entailment ratio for Fox and 34.8% and 43.1% decreases for CNN and MSNBC, respectively, again not surprising given the political leanings of each network's viewers.

Figure 3 reports for each network by period the fraction of that network's total sentences with the police token. The results reinforce the conclusions of the entailment analysis. In the pre-Floyd period the police token appears proportionally far more often on Fox than CNN and MSNBC. In the second period the proportion of total mentions by network increases dramatically for both MSNBC and Fox even as their entailment ratios move in oppose directions. During Tcapitol mentions of the police token drop substantially for all three networks (this is also the shortest time slice in our analysis) but on CNN and MSNBC the entailment moves up from their low levels during TFloyd. Finally, like with period TFloyd, mentions of the police token rise substantially for all three networks during the Chauvin trial period even as the entailment ratio declines for MSNBC and increases for Fox.

5 https://www.washingtonpost.com/politics/2020/06/01/differe nce-between-democratic-republican-reactions-protests-elevate-geo rge-floyd-or-antifa/

Much has been written about the partisan divide in cable news coverage. To our knowledge our analysis is the first to show that across cable networks coverage of politically salient events responds quickly and dramatically to the partisan preferences of their viewership. This finding, we believe, is disturbing because it speaks to the siloed nature of political coverage in contemporary America. A necessary condition for reaching compromise in a democratic society is having an understanding of how individuals with opposing views perceive the world even as one disagrees that world view. That network reporting to their already partisan siloed viewership responds so quickly to comport with their viewership's world view is not an encouraging sign that partisan divides can be bridged anytime soon.

Ethical Statement

In this paper, we consider public-domain data available on the official web sites of three major US cable news networks. We thus do not foresee major ethical concern over choice of data source. In our work, we fine-tune large-scale pre-trained language models (BERT [Devlin et al., 2019] and RoBERTa [Liu et al., 2019]) for training purpose. While our entailment results are obtained after fine-tuning the large language model, recent works have indicated that these models have a wide range of biases that reflect the texts on which they were originally trained, and which may percolate to downstream tasks [Bender et al., 2021]. Finally, in this paper, we present a new approach to analyze news media content that can help us identify differences in media portrayals of major policing event in the US. While we do not discuss how to react to these discoveries, our analyses can inform the society better about these differences using our quantifiable approach.

Acknowledgments

The authors like to thank Sharon Lynch for her valuable input to the paper.