DISPUTool -- A tool for the Argumentative Analysis of Political Debates

1 Introduction

Digital Humanities (DH) is the research field concerned with the application of computational methods to traditional humanities disciplines such as literature, history, and philosophy. This field is receiving a growing attention from the Artificial Intelligence (AI) community, as many recent advances in AI can be thought to ease the work of humanities scholars.

A natural example is represented by political speeches and debates, where huge amount of textual data has to be analyzed to set up or verify the hypothesis of historians and social scientists. Political debates, in particular, are public interviews where the candidates of political elections are requested to confront each other about topics like unemployment, taxes, and foreign policy. They are particularly important during the presidential elections in the US, where it is customary for the two candidates of the two largest parties, i.e., the Democratic Party and the Republican Party, to engage in a debate around the most controversial issues of the time. Even though these debates are not constitutionally mandated, they are considered as a de facto election process.

∗ Shohreh Haddadan hereby acknowledges that this research is supported by the Luxembourg National Research Fund (FNR) (10929115). Elena Cabrio and Serena Villata have been partially supported by EU H2020 research and innovation programme under the Marie Sklodowska-Curie grant agreement No. 690974 for the project MIREL: MIning and REasoning with Legal texts.

Given the huge amount of data available in each election campaign, the work of historians and social scientists on this data requires a big effort. Natural Language Processing (NLP) methods and, in particular, argument(ation) mining methods [Lippi and Torroni, 2016b; Cabrio and Villata, 2018] can be successfully employed to automatically process this data and assist humanities scholars in data exploration and argumentative analysis. In this paper, we present DISPUTool, a tool conceived to support humanities scholars in the exploration and evaluation of textual political debates.

DISPUTool relies on argumentation mining methods to automatically identify argumentative components (i.e., premises and claims) from the textual transcripts of political debates in English. It allows to highlight the premises and claims proposed in the debate by the different candidates and to explore these debates, showing the main named entities mentioned in the debates by the candidates based on the year of the debate. Two tasks are crucial in argumentation mining: (1) Argument component detection: the identification of arguments within the input natural language text. This step may be further split in two different stages such as the detection of argument components (e.g., claim, premises) and the further identification of their textual boundaries. Approaches addressing this task adopt different methods like Support Vector Machines (SVM), Na¨ıve Bayes classifiers, Logistic Regression, and Neural Networks; (2) Relation prediction: the prediction of the relations holding between the argumentative components identified in the first stage. The predicted standard relations between the argument components are attacks and supports. Different methods have been employed for this task, from standard SVMs to Textual Entailment. In DISPUTool, we focus on the argument component detection task, while we leave as future work the relation prediction one.

Despite the plethora of existing approaches for argumentation mining, only few of them tackle the issue of mining argumentative structures from political debates [Menini et al., 2018; Duthie et al., 2016; Duthie and Budzynska, 2018; Lippi and Torroni, 2016a; Visser et al., 2019]. However, to the best of our knowledge, none of them take on the design of a tool like DISPUTool where the identification of premises and claims is addressed together with Named Entity Recognition (NER), supporting intelligent data exploration on political debate textual transcripts.

2 Argumentative Analysis and Data Exploration of US Political Debates

In this section, we first highlight the main facilities of DISPUTool, and second, we describe the dataset we annotated to train our argumentative component classifiers, the experimental setting we set up for training the system, and the results we obtained.

2.1 DISPUTool Main Facilities

DISPUTool is a tool designed to explore a dataset of presidential debates in the US for the period 1960-2016. The main feature DISPUTool offers is to automatically analyze from the argumentative point of view such debates, i.e., the detection of argumentative components (premises and claims) in the textual transcripts of these political debates. More precisely, DISPUTool provides the following facilities:

Argumentative analysis of the US presidential debates:

The user can explore a corpus consisting of 39 presidential debates held in the US annotated with argumentative components, i.e., premises and claims. The user can select the debate she is interested in, and after the selection, she is provided with the list of premises and claims proposed in the debate together with the candidate who put forward the argument component and the date in which the debate held. In addition, the user can visualize the whole debate transcript, where the premises and the claims can be highlighted with different colors, depending on the goal of the user.

Analyze your own debate: If the user is not interested in the debates of the corpus, she is presented with the opportunity to paste in a free text window the text of the debate she aims at analyzing, from the argumentative point of view. The result of the analysis is the pasted text where the claims are highlighted with the green color, and the premises with the blue color.

Explore the US presidential debates: the user can also explore the corpus based on the named entities we identified in the debates using the Stanford Named Entity Recognizer. 1 The user can filter the data based on the type of named entity (i.e., person, location, nationality, organization and religion), on the year in which the debate held (years from 1960 to 2016 are listed), and on the speaker (all the candidates of the debates are listed).

A demo video of DISPUTool is available at https://github.com/ElecDeb60To16/DISPUTool.

2.2 Experimental Setting and Results

The corpus in DISPUTool is a collection of transcripts of the televised presidential candidate debates in United States from 1960 (Kennedy vs Nixon) until 2016 (Clinton vs Trump) covering 56 years of debates. The data was obtained from the website of the Commission on Presidential Debates (CPD) 2 . The dataset is composed of 6601 speech turns, 34013 sentences and 676227 tokens. The dataset [Haddadan et al.,

1 https://nlp.stanford.edu/software/CRF-NER.html

2 www.debates.org

2019] has been annotated with claims (i.e., the conclusion of the argument) and premises (i.e., the reasons provided to justify the claim of the argument) with three annotators according to precise annotation guidelines. In total, we provided 29811 annotations divided into 16250 claims and 13561 premises. The USElecDeb60To16 v.01 dataset and the annotation guidelines are available at https://github.com/ ElecDeb60To16/Dataset.

We have devised a pipeline structure for identifying the argument components [Haddadan et al., 2019], i.e., we first distinguish the argumentative sentences from the nonargumentative parts so that the boundaries of the arguments are identified (Task 1), and then we classify whether the argument component is a premise or a claim (Task 2). We have trained three classifiers for each step of the pipeline. The first classifier is a SVM based classifier using structural (e.g., length of the sentences), semantic (e.g., sentiment polarity, named entities) and linguistic features (e.g., words, part of speech). The second classifier is a bidirectional Long Short Term Memory (LSTM) neural network using pretrained word-embeddings as features. The third classifier is a Feed-Forward neural network using the same features as the SVM classifier.

The results (precision p, recall r and F-Score F1) of the classification on Task 1 with these classifiers are as follows: SVM (Rbf Kernel - All features): p 0.851, r 0.853 F1 0.823, LSTM network (word-embedding features): p 0.841, r 0.854, F1 0.843, Feed Forward Network (All features): p 0.805, r 0.796, F1 0.800. The results of the classification on Task 2 are as follows: SVM (Rbf Kernel - All features): p 0.678, r 0.662, F1 0.651, LSTM network (word-embeddings): p 0.673, r 0.673, F1 0.673, Feed Forward Network (All features): p 0.641, r 0.641, F1 0.640. We have used the best performing model (LSTM with word-embedding features) in DISPUTool for the identification of argument components.

3 Concluding Remarks

In this paper, we presented DISPUTool, a tool to support humanities scholars in addressing an argumentative analysis of political debates. The tool automatically identifies premises and claims put forward by the candidates during political debates, and highlights the main named entities discussed in these debates. To the best of our knowledge, this is the first tool tackling this challenging goal in a fully automated way by employing Natural Language Processing and Machine Learning methods. Alas, several future work directions still have to be considered. First of all, we are annotating the US election corpus with relations between the argumentative components (i.e., support and attack), and we will train a classifier for this task as well, so that DISPUTool can be empowered with this further feature. Second, we will provide a graph-based visualization of the argumentative structure of each candidate viewpoint and the one combining the opinions of the different candidates about a specific topic. Third, we will include in DISPUTool the possibility to highlight contradictions that can possibly hold between the viewpoints of the same candidate or candidates from the same political party on a particular topic discussed in the debates.