Evaluating GPT-3 Generated Explanations for Hateful Content Moderation

1 Introduction

Over the last few years, researchers have proposed many machine learning and deep learning methods to detect online hate speech, which is content that expresses hate or encourages violence towards a person or group based on race, religion, gender, or other identity characteristics [Fortuna and Nunes, 2018; Schmidt and Wiegand, 2017]. While many existing methods have achieved reasonably good performance, they have primarily focused on predicting if a given content is hateful with few explanations. In the content moderation process, explanations for flagging are crucial for both moderators and users to understand why a piece of content was deemed hateful. By providing explanations for the predictions, explainable AI can improve the trustworthiness of the hate speech detection methods, increasing the overall accountability of the moderation process [Vaccaro et al., 2020; Haimson et al., 2021].

Previous research has explored various forms of explanations and constructed datasets such as HateXplain [Mathew et al., 2021], implicit hate speech corpus [ElSherief et al., 2021], and social bias frame [Sap et al., 2020] to support hate speech explanations. The availability of ground truth hate speech explanations can be used to fine-tune or prompt language models such as GPT-3 for free-text explanation generation [Radford et al., 2019]. For instance, [Huang et al., 2022] adopted the Chain-of-thought [Wei et al., 2022] framework to prompt GPT-2 to generate explanations for hate speech. However, while previous works have performed automatic evaluations by comparing generated explanations to ground truth [ElSherief et al., 2021; Sap et al., 2020; Huang et al., 2022], the human perception and implications of these generated explanations remain unclear. This paper aims to evaluate the effectiveness and limitations of the stateof-the-art language model GPT-3 in generating explanations for hate speech. We propose an analytical framework and conduct a comprehensive survey to assess the subjective and nuanced nature of hate speech explanations and answer the following research questions.

1. RQ1. To what extent do the explanations generated by GPT-3 for hateful content demonstrate qualities of being fluent, informative, persuasive, and logically sound?
2. RQ2. How persuasive are the GPT-3 explanations, and does the effectiveness of persuasion vary with different prompting strategies?
3. RQ3. Does the use of GPT-3 explanations lead to incorrect decisions in hateful content moderation?

The study utilizes the tweets annotated in the HateXplain dataset [Mathew et al., 2021]. The study aims to prompt GPT3 to explain hateful and non-hateful tweets and assess the generated explanations' quality through human evaluations. The evaluation metrics include fluency, informativeness, soundness, and persuasiveness. We conducted a survey to assess Table 1: Sample of GPT3 generated explanations and evaluators' 5-point Likert scale rating for the evaluated tweet. The scores are broken down into five categories: F for fluency, I for informativeness, S for soundness, P for persuasiveness, and H for hatefulness. Three annotators assigned each category's ratings. Scores from three annotators were averaged except for hatefulness ratings.

the quality of explanations generated by GPT-3 and the hatefulness of tweets after viewing the explanations. The study addresses three key research questions, and we summarize the key findings as follows:

• Human evaluators have assessed that the explanations generated by GPT-3 are fluent, informative, persuasive, and logically sound.
• Different prompting strategies illicit varying persuasive effects. When prompted to explain why a given tweet is hateful, GPT-3 generated a more persuasive explanation than simply asking it to provide contextual information on the tweet. The length of the generated explanation also affects its persuasiveness.
• The potential for GPT-3 generated explanations to mislead human moderators when evaluating hateful content is a matter of concern. Our study has revealed that the explanations produced by GPT-3 can cause human evaluators to misclassify roughly 20% of tweets. Such misclassifications can wrongly label non-hateful tweets as hateful or vice versa. This is in contrast to a baseline scenario where human evaluators made assessments without any explanation.
• We observed that presenting both hateful and nonhateful explanations generated by GPT-3 could mitigate the risk of misleading content moderators.

2 RelatedWorks

Content moderation and hate speech. In recent years, major social media platforms such as Facebook, YouTube, and Twitter have integrated AI into their operations for content moderation. The rise of user-generated content has made it increasingly necessary for these platforms to monitor and remove harmful or illegal content [Djuric et al., 2015; Badjatiya et al., 2017; Watanabe et al., 2018; Awal et al., 2021; Awal et al., 2023; Meng et al., 2022; Lin et al., 2021]. As a result, governments have imposed strict regulations on social media platforms, requiring them to remove hateful content quickly. In response, many platforms have implemented automated systems, such as algorithms and AI, to proactively detect and remove such content at scale [Lampe and Resnick, 2004]. However, these systems have been criticized for labour concerns, lack of transparency, perpetuating biases, and potential harm to marginalized communities [Gillespie, 2018; Haimson et al., 2021; Steiger et al., 2021; Suzor et al., 2019; Cao and Lee, 2020]. To address these concerns, there is a growing emphasis on developing trustworthy and explainable AI systems for hate speech detection and moderation.

Explainable hate speech detection. Hate speech detection aims to identify and prevent online hate. Although several studies have been conducted on this topic [Davidson et al., 2017; Cao et al., 2020], hate speech models like the Google Jigsaw API are vulnerable to racial biases and counterfactual and adversarial attacks [Sap et al., 2019; Park et al., 2018]. To address these issues, explainable hate speech detection approaches have received attention, wherein model predictions are described through natural language explanations [Camburu et al., 2018; Mathew et al., 2021]. Recent studies, such as [Sap et al., 2020] and [ElSherief et al., 2021], have examined stereotypes in social media and the implicit nature of hate speech, respectively, providing insight into the underlying causes of hate speech and aiding in the development of model explanations. Furthermore, the generation of counter-narratives to combat hate speech has been explored [Tekiroglu et al., 2020; Ashida and Komachi, 2022].

Human-generated explanations are limited and challenging to scale for real-world scenarios. To overcome this limitation, pre-trained language models are being utilized to automate the explanation generation process [Sap et al., 2020; Huang et al., 2022]. Furthermore, generative explanations can help address user appeals against platform flagging [Vaccaro et al., 2020]. However, using GPT-3 based API for content moderation by companies like Cohere and OpenAI raises concerns about bias and spreading misinformation [Markov et al., 2022]. Additionally, the quality of generated explanations requires further evaluation, as current studies primarily rely on automated quantitative analysis, such as the BLEU score. Therefore, our research aims to assess the credibility of generated explanations using a qualitative assessment conducted by human evaluators.

Prompting. Prompting is a technique that enables LLMs, such as GPT-3, to adapt to specific tasks by incorporating task-related instructions or questions into input text [Brown et al., 2020; Sanh et al., 2021; Wei et al., 2022; Kojima et al., 2022]. The format and order of the prompt and the demonstration examples, can affect the performance of LLMs [Zhang et al., 2022; Zhao et al., 2021]. Complex reasoning tasks can be improved by inducing a sequence of intermediate steps with few human-crafted demonstration examples [Wei et al., 2022]. Adding the instruction "Let's think step by step" before each response could have the same effect as [Wei et al., 2022] in zero-shot learning [Kojima et al., 2022]. Prompting has shown promising results in various natural language processing (NLP) tasks, including teaching machines to generate explanations with demonstrations, which can be even more suitable than human-written explanations [Wiegreffe et al., 2022; Petroni et al., 2019].

Previous research has investigated using GPT-3 for addressing online hatred and abuse, including hate speech detection [Chiu et al., 2021] and toxicity detection [Wang, 2022]. GPT-3 has the potential to generate explanations for its predictions, such as why it classified certain texts as hate speech. However, the generated content from GPT-3 may not always be accurate or factual, presenting potential risks in applications such as addressing online hatred and abuse. Generative models may not always critically evaluate their predictions, meaning they can make predictions but not provide reasoning behind them [Saunders et al., 2022]. GPT-3's ability to generate human-like text can be misused to create convincing disinformation, such as fake news and propaganda. Its writing has been shown to significantly impact readers' perspectives on international affairs [Buchanan et al., 2021]. It is crucial to weigh the potential benefits of using GPT-3 against the risks associated with its ability to generate convincing disinformation and influence people's perspectives, as well as its limitations in providing reasoning for its predictions.

3 Methodology and Experimental Setup

This section outlines our methodology for determining the quality of explanations generated for hate speech using a representative dataset. The simplified process is as shown in Figure 1. The overall process includes sampling examples of hate speech, utilizing various prompting strategies to generate explanations, and designing a survey to evaluate the quality of the generated explanations. The survey is administered to gain deeper insights into the nature of hate speech and how to combat it effectively. The following sections detail our methods and techniques used at each stage of the process.

3.1 Dataset

The HateXplain dataset [Mathew et al., 2021] comprises a set of hate speech instances acquired from Twitter and Gab. Each record in the dataset was annotated by three persons who assigned one of the following three labels: "offensive," "normal," or "hateful." The final label for each record was determined via majority voting. Our experiment only considered Twitter tweets labelled "hateful" or "normal." To ensure a comprehensive analysis of hate speech, we incorporated all annotations associated with each tweet. Consequently, the tweets were categorized into four groups: [nnn, nnh, nhh, hhh], where "h" represents "hateful" and "n" represents "normal." A total of 25 tweets were randomly selected from each group, resulting in a 100-tweet evaluation set.

3.2 Prompting GPT-3 for Explanation Generation

We fed the GPT-3 model with the evaluated tweet, followed by three prompting strategies to generate the explanations:

• Why is ...? (WHY) We prompt the model with "Please explain why this tweet is (hateful/non-hateful)". The goal is to prompt GPT-3 to generate a hateful or nonhateful explanation for the tweet.
• Chain-of-Thought (COT): We prompt the model with the same question asked in the WHY strategy and prefix the answer "Let's think step-by-step.". The goal is to prompt GPT-3 to generate a more detailed hateful or non-hateful explanation with reasoning steps.
• What is the context ...? (CONTEXT): We prompt the model with "What is the context of the tweet?". The goal is to encourage the GPT-3 model to generate an informative response that captures the relevant contextual details associated with the tweet without explicitly telling the model if the tweet is hateful or non-hateful.

We prompted GPT-3 with four demonstration examples for each strategy before asking about the specific tweet. These examples consist of tweets extracted from the HateXplain dataset, accompanied by explanations generated through zero-shot prompting using GPT-3 and carefully chosen for their fluency, soundness, informativeness, and persuasiveness. The Table 2 presents the overall prompt format.

Using the three prompting techniques, eight distinct combinations of tweets and explanations were generated for survey evaluation. These combinations involved pairing a tweet with either no explanation, a hateful explanation, a nonhateful explanation, a contextual explanation, or both hateful and non-hateful explanations side-by-side.

3.3 EvaluationMetrics

An extensive human evaluation was conducted to evaluate the hatefulness of the tweets and the quality of the explanations generated by GPT-3 based on four criteria: fluency, informativeness, persuasiveness, and soundness. A 5-point Likert scale was employed, where 1 represented the poorest quality and 5 the best. The definition of hatefulness and the quality criteria are as follows.

• Hatefulness. A hateful tweet is defined as: "Any speech that attacks a person or group on the basis of attributes such as race, religion, ethnicity, nationality, gender, disability, sexual orientation, or other identity factors." The rating scale spanned from 1 (non-hateful) to 5 (hateful), allowing

# Demonstration samples 1...N

Tweet: <user><user><user><user><user>what else we can expect from jihadi terrorist country

Question: Please explain why this tweet is hateful? Answer: [Let's think step-by-step.] The tweet is hateful because it mocks Pakistan's status as a haven for terrorists.

###

# Actual tweet in prompt

Tweet: <ser> so did the nazis,

Question: Please explain why this tweet is hateful?

Answer: [Let's think step-by-step.] This tweet is hateful because it trivializes the Holocaust and the suffering of the Jewish people. It is also offensive and insensitive.

Table 2: An illustration of a prompt utilizing N training examples is presented, consisting of the tweet (in yellow), the question (in green), and the explanations provided (in purple) that are utilized as input to GPT-3.

annotators to express varying degrees of hate speech [Poletto et al., 2019]. This approach offers a more detailed and flexible annotation process.

• Fluency evaluates whether the explanation follows proper grammar and structural rules, with a rating scale ranging from 1 (poor) to 5 (excellent).
• Informativeness assesses whether the explanation provides new information, such as explaining the background and additional context, with a rating scale ranging from 1 (not informative) to 5 (very informative).
• Persuasiveness evaluates whether the explanation seems convincing, with a rating scale ranging from 1 (not persuasive) to 5 (very persuasive).
• Soundness describes whether the explanation seems valid and logical, with a rating scale ranging from 1 (not sound) to 5 (very sound).

3.4 Human Evaluation Setting

Human evaluations were carried out on both Amazon Mechanical Turk and Clickworker platforms. As the HateXplain dataset primarily comprises Twitter content generated in a American context, we recruited human evaluators residing in the United States. In total, we recruited 2,400 participants for the human evaluation. They were asked to evaluate the level of hatefulness in tweets and the quality of explanations generated by the GPT-3 model.

A three-round survey was executed to evaluate the explanations generated by GPT-3. Each round involved distinct participants and a variable number of questions pertaining to the same set of 100 sampled tweets. To ensure the validity of the survey and the respondent attentiveness, a basic math question is included in each round, e.g. "What is 8+3?".

Round 1. In the survey's first round, participants assessed tweet hatefulness without explanations. This controlled baseline aimed to determine if providing explanations impacts hatefulness perception in those tweets.

Round 2. In the second round of the survey, tweetexplanation pairs were presented, with each tweet paired with either a hateful or a non-hateful explanation. The explanations were generated using the prompting strategies WHY and COT. Following this, participants were asked to evaluate the level of hatefulness in the tweets after reading the provided explanation and respond to four quality questions assessing the fluency, informativeness, persuasiveness, and soundness of the explanation. The primary objective of this round was to examine whether GPT-3 can generate high-quality and persuasive explanations (for both hate and non-hate labels) capable of influencing individuals' perceptions.

Round 3. The third round of the survey adopts two distinct formats. The first format combines hateful and non-hateful explanations generated by the WHY and COT prompting strategies. Participants were asked to respond to four quality questions of both explanations presented side-by-side. In the second format, the CONTEXT prompting strategy was utilized to elicit background information about the given tweet. Based on the contextual information, participants had to answer four quality questions assessing the explanations' quality. The primary objective of this format is to evaluate whether language models can generate high-quality and objective explanations (contextual) that reveal the tweet's implicit context without impacting readers' perceptions.

Participants who did not receive any explanations (i.e., Round 1) were given a survey consisting of two questions and were allotted 15 minutes to complete it. Conversely, participants who were presented with one explanation (i.e., Round 2) were given a survey consisting of five questions and given 30 minutes to complete it, while those who received two explanations (i.e., Round 3) were provided with a survey consisting of 10 questions and given 45 minutes to complete it. Three survey responses were collected for each data point, and only responses from evaluators who correctly answered the basic arithmetic questions were recorded in the analysis.

4 Experimental Results and Analysis

This section presents the human evaluation results and discusses our key findings in respect to the three research questions on hate speech explanation generated by GPT-3.

4.1 RQ1. Quality of Generated Explanations

Table 3 summarizes the quality assessment of the GPT-3 generated explanations using various prompting strategies. Specifically, we report the average Fluency, Informativeness, Persuasiveness, and Soundness scores provided by the human evaluators. These were rated on a scale of 1 to 5, with 5 being the best score.

We observe that within the same prompting strategy, the scores across four quality metrics for hateful explanations were higher than that for non-hateful explanations. This is because the HateXplain dataset used in our study focuses on hate speech and the tweets collected were more offensive in nature. Some tweets contain slurs such as "niggas", "faggots" or "fuck", but are marked as non-hateful because of its context. Consequently, GPT-3 faces a greater challenge in generating reasonable non-hateful explanations for such tweets, ultimately impacting human evaluators' assessments. We compared two strategies, WHY and COT, for generating nonhateful explanations and found that the latter significantly improved the quality score (with an average improvement of approximately +0.42 across all quality assessment metrics). The superior performance of the COT approach in generating non-hateful explanations suggests that breaking down the tweet analysis into steps to reach a conclusion is beneficial in generating non-hateful explanations, particularly in challenging contexts such as those presented by the HateXplain dataset. Conversely, we observe that the hateful explanations generated using WHY strategy scored better than those generated using COT, suggesting that the human evaluators prefer shorter and more direct hateful explanations over long and elaborate ones when evaluating generally offensive tweets.

Our findings indicate that the quality scores for the sideby-side hateful and non-hateful explanations (both) generated using WHY and COT were moderately effective. In contrast, CONTEXT was the least effective prompting strategy for generating high-quality explanations. Closer examination of the CONTEXT explanations revealed that many of them simply summarized the tweet content without providing significantly new information. In Section 4.3, we will delve deeper into the effects of applying the WHY-Both, COT-Both, and CONTEXT strategies to mitigate risks associated with using LLMs for moderating hateful content.

Overall, our quality assessment analysis revealed that the explanations generated by GPT-3 scored well in terms of fluency and informativeness, but slightly worse in terms of soundness and persuasiveness. This finding is consistent with previous research [Wiegreffe et al., 2022; Dou et al., 2022], which suggests that generating a sound and persuasive explanation necessitates a deeper understanding of the tweets, making it a more challenging task than achieving fluency and informativeness. Furthermore, we observed that the GPT3 model performed inadequately in generating explanations with a prompting label that contradicts the true label in the dataset. Consequently, hateful explanations received higher quality scores for hateful tweets than non-hateful tweets, while non-hateful explanations received higher scores for non-hateful tweets than hateful tweets.

4.2 RQ2. Persuasiveness of Generated Explanations

The earlier section has established that GPT-3 is able to generate high quality explanations for hateful tweets. However, little is known about the the influence of these generated explanations on human perception. This section will fill this gap by analyzing the persuasiveness of the generated explanations and their effects on the the tweets' hatefulness scores rated by the human evaluators.

Table 4 presents a summary of the average hatefulness scores of tweets rated by human evaluators when presented with the accompanying explanations generated by GPT-3 using various prompting strategies. We observe that the WHY-non-hateful explanations have significantly decreased the hatefulness scores of both hateful and non-hateful tweets. Conversely, we observe that the WHY-hateful explanations have significantly increased the hatefulness scores of nonhateful tweets, but no such effects are observed on the hateful tweets. One possible explanation for this finding is that the human evaluators had already rated the hateful tweets as hateful, and the generated explanation did not alter their assessment of these tweets.

Similar observations were made on the explanations generated using the COT strategy. For instance, the COT-nonhateful explanations have significantly decreased the hatefulness scores of non-hateful tweets, and the COT-hateful explanations have significantly increased the hatefulness scores of hateful tweets. Interestingly, we observe that the COT-nonhateful explanations did not significantly change the hatefulness scores of hateful tweets. This finding may be attributed to the elaborated explanations generated by the COT strategy, which may have required the GPT-3 model to fabricate convoluted explanations in an attempt to convince human evaluators that a hateful tweet is actually non-hateful. However, as discussed in the previous section, such explanations may be deemed as unsound, thereby failing to alter the human evaluators' ratings of the tweet. Similar results were observed for the COT-hateful explanations on non-hateful tweets.

When presented with both hateful and non-hateful explanation side-by-side using the WHY-both and COT-both strategies, we observe that hatefulness scores have little or no apparent change compared to not providing any explanations. This suggests that when presented with a balanced explanation, i.e., both hateful and non-hateful, the human evaluators are less likely to be persuaded by either explanation and alter their opinions on a given tweet. Similarly, when presented with the CONTEXT explanations for hateful tweets, the change in hatefulness scores is insignificant. Interestingly, we noted that providing the CONTEXT explanations reduced the hatefulness scores for non-hateful tweet significantly.

4.3 RQ3. Risks of GPT-3 Generated Explanations for Hateful Content Moderation

In this analysis, we aim to examine the impact of explanations generated by GPT-3 on human decision-making in hateful content moderation. To achieve this, we first classify each annotation as either non-hateful or hateful, based on their rated hatefulness scores of 1-2 and 4-5, respectively. Each tweet has three annotations from three different human evaluators, and the majority vote is used as the annotated label for the tweets. We identified misclassifications by comparing the labels assigned by human evaluators with and without seeing explanations. Specifically, we studied cases where human evaluators labeled tweets different from round 1, where there are no explanations, after reading GPT-3 generated explanations using various prompting strategies. Figure 2 displays the distributions of tweets that were misclassified by human evaluators after reading explanations generated by different prompting strategies.

Misleading Human Evaluators. Figure 2 shows the distribution of tweets misclassified by the human evaluators after reading the prompting strategies' explanation. From Figure 2, we observe that exposing the evaluators to WHYhateful explanations increases the misclassification of nonhateful tweets, as they are persuaded to label them as hateful. Similarly, exposing the evaluators to WHY-non-hateful explanations also leads to an increase in misclassifying hateful tweets as non-hateful. The COT-hateful and COT-nonhateful explanations also misled the evaluators, resulting in misclassification of both non-hateful and hateful tweets. Surprisingly, even CONTEXT explanations have been found to mislead evaluators, resulting in misclassifications of both hateful and non-hateful tweets. Notably, a significant number of hateful tweets were mistakenly rated as non-hateful when presented with CONTEXT explanations.

Our hypothesis was that presenting both hateful and nonhateful explanations together would provide human evaluators with balanced information, aiding them in making better decisions regarding moderating hateful content. However, our observations show that even with WHY-both explanations, there is still a significant number of misclassifications. Conversely, our findings indicate that the COT-both approach is a more promising method, resulting in fewer misclassifications for both hateful and non-hateful tweets.

Effects of Explanation Length. Despite the ability of COT-both to provide detailed and balanced explanations to human evaluators, we found that the generated explanations are significantly longer than those generated by other methods. For example, COT-both explanations have an average length of 173 words, compared to CONTEXT and WHY-both explanations with an average length of 28 and 75 words, respectively. Additionally, our research revealed that when human evaluators used a mobile device to respond to the survey, some COT-both explanations extended beyond a single page, requiring the evaluator to scroll to view entire content. To address this issue, we introduced a "scrolled" variable in our survey to verify whether evaluators had scrolled to read the full COT-both explanations when rating the hatefulness of the tweets. We next analyzed the scrolling behavior of the 300 evaluators who assessed tweets paired with COT-both explanations. We discovered that 69 out of the 300 evaluators did not scroll to read the complete COT-both explanations while rating the hatefulness of the tweets. This raises concerns about the reliability of these evaluations, and we therefore excluded these 69 ratings from our analysis. The resulting misclassification of tweets is reported in Figure 2 as COT-both Scrolled. The figure shows that the number of misclassifications decreased even further after removing the evaluations of human raters who did not scroll to read the full COT-both explanations. Misclassifications may occur when individuals focus solely on the first hateful explanations. This selective exposure raises the risk of mistakenly classifying non-hate tweets as hate and the drawback of lengthy explanations is also evident. Hence, future research will focus on exploring techniques that generate shorter and more concise explanations for effective moderation of hateful content.

5 Discussion

To the best of our knowledge, this study represents the first evaluation of the quality and ethical implications of PLMgenerated explanations for moderating hateful content on a large scale. Our findings demonstrate that GPT-3 can produce fluent, informative, persuasive, and logically sound explanations when properly prompted. These results suggest that GPT-3 may be a valuable tool for combating online hate speech and content moderation. However, we discovered that the persuasiveness of the explanations varied depending on the prompting strategy and the length of the explanations. In particular, we found that biased explanations, regardless of whether they were hateful or non-hateful, had a strong persuasive effect on human annotators. Therefore, the use of GPT-3 to produce explanations for content moderation should be approached with caution. Moreover, our observations revealed that the explanations generated by GPT-3 had the potential to lead to incorrect labeling of tweets. This is a critical issue that must be addressed to ensure the effectiveness and ethicality of using GPT-3 to aid content moderation. We recommend presenting both hateful and non-hateful explanations side-by-side, which can significantly reduce the risk of misleading content moderators.

Limitations. The study analyzed the Hatexplain dataset, which specifically addresses hate speech on Twitter and Gab, which may limit the generalizability of the study to other platforms. Moreover, the Hatexplain dataset, like other hate speech detection datasets, is susceptible to subjective hate speech annotation, leading to varying definitions among annotators and potentially inaccurate annotations. Additionally, the study used non-expert human annotators to evaluate the GPT-3 model's explanations, with precautions taken for accuracy, but some anomalies may still exist. Future research could involve experienced human annotators to obtain higherquality annotations. The study solely concentrated on the GPT-3 model, necessitating further investigation into whether similar results can be achieved with other language models.

6 Conclusion

This study introduced an analytical framework for evaluating hate speech explanations and conducted a comprehensive survey on their effectiveness. The research identified the potential of GPT-3 in generating high-quality explanations for hate speech while also highlighting the limitations and risks of using PLM-generated explanations for content moderation. For future work, we aim to develop improved evaluation metrics and prompting strategies to enhance the quality and reduce bias in explanations. These efforts would help increase the fairness and effectiveness of content moderation and combat online hate speech.