Decoding the Underlying Meaning of Multimodal Hateful Memes

1 Introduction

Internet memes are viral content spread among online communities. While most memes are often humorous and benign, hateful memes, which attack a target group or individual based on characteristics such as race, gender, and religion, have become a growing concern. As part of its effort to moderate the spread of hateful memes, Facebook recently launched the "hateful meme challenge" [Kiela et al., 2020]. The challenge released a dataset with 10K+ hateful memes to encourage submissions of automated solutions to detect hateful memes. This led to the development of various multimodal deep learning approaches for hateful meme classifications [Yang et al., 2022; Lee et al., 2021]. Other studies have also contributed to the growing effort to curb hateful memes by collecting and releasing large hateful meme datasets to support the training and evaluation of hateful meme classification models [Suryawanshi et al., 2020; Gasparini et al., 2021; Pramanick et al., 2021a; Sharma et al., 2023; Sharma et al., 2022b].

However, existing studies have primarily focused on performing hateful meme classification (i.e., predicting if a given meme is hateful) with limited explanations for its prediction. Providing explanations for the detected hate meme is integral to the content moderation process. The content moderators and users may want to understand why a particular meme is flagged as hateful. Nevertheless, explaining hateful memes is challenging as it requires a combination of information from various modalities and specific socio-cultural knowledge [Kiela et al., 2021]. Consider the hateful meme in Figure 1. To explain the hateful meme, one will need the socio-cultural knowledge that the girl in the image is Anne Frank, and realize the textual reference refers to the gas poisoning of Jews during the Holocaust.

Recognizing the importance of providing contextual reasons for the predicted hateful memes, recent studies have performed fine-grained analysis to classify the type of attacks [Mathias et al., 2021] and infer the targets being attacked [Pramanick et al., 2021a; Sharma et al., 2022a]. However, such fine-grained analysis may still be inadequate for content moderators to understand hateful memes. These analyses often only predict general protected characteristics (e.g., race) but not the specific social target attacked (e.g., Jews). Furthermore, having informative reasons in natural sentences, such as the example provided in Figure 1, would make it easier for content moderators to comprehend the hateful memes.

In a recent study, Elsherief [2021] collected a large textual dataset to support implicit hate speech classification. The dataset contains hateful textual posts annotated with their corresponding implied statements, which could be seen as a form of explanation to aid content moderators in understanding the implicit hate speech. The availability of ground truth explanations also allows researchers to apply and explore training Pre-trained Language Models (PLMs) such as GPT-2 for explanation generation. Ideally, we would also like to generate the underlying reasons for why a flagged hateful meme is considered hateful. To the best of our knowledge, there is no current dataset to facilitate this exploration.

Detected Web Entities

• Madame Tussauds
• Berlin
• London
• Anne Frank
• Zentrum Wax sculpture
• Museum

Reason

Disrespects the jews by downplaying the burning of jews during the holocaust

Figure 1: Example of a hateful meme in HatReD.

Research Objectives. To address the research gaps, we propose a new conditional generation task that aims to generate the underlying reasons to explain hateful memes automatically. We constructed the Hateful meme Reasoning Dataset (HatReD), which is a new multimodal hateful memes dataset annotated with the underlying hateful contextual reasons, to support the proposed task. Specifically, we carefully design a framework to annotate Facebook's Fine-Grained Hateful Memes dataset [Mathias et al., 2021] with the underlying hateful reasons. We fine-tune PLMs on HatReD and conduct extensive experiments to evaluate the PLM's performance and limitations on the new generation task. Finally, we also demonstrate the usefulness of HatReD by evaluating the fine-tuned PLMs' ability to generate the explanations for hateful memes in an unseen misogynous meme dataset.

Contributions. We summarize our contributions as follows: (1) We construct HatReD, which is a multimodal hateful meme dataset annotated with underlying hateful contextual reasons. To the best of our knowledge, this is the first hateful meme dataset with written explanations. (2) We introduce a new conditional generation task that aims to generate underlying reasons to explain hateful memes automatically. We conduct extensive experiments to establish the task baselines using state-of-the-art PLMs. (3) We analyze the challenges of generating the generation new task and demonstrate the usefulness of HatReD in explaining memes in seen and unseen domains.

2 RelatedWorks

2.1 Hateful Meme Datasets

Hateful meme classification is an emerging research topic made popular by the availability of several hateful meme datasets. Table 1 summarizes the hateful meme datasets released over the last few years. All the datasets contain class labels that support the hateful meme classification task. For instance, the memes in Facebook Hateful Meme Challenge dataset are labeled "hateful" or "non-hateful" [Kiela et al., 2020]. Similarly, Suryawanshi [2020] collected a small dataset of politics-related memes from Tumblr and annotated the memes as "offensive" or "non-offensive". Besides the class labels that facilitate hateful meme classification, some datasets have also provided supplementary information on hateful memes. For example, Pramanick [2021a] collected a dataset containing COVID-19 related memes. The researchers annotated the harmfulness of the memes and the types of target (e.g., individual, organization, and community) attacked in the harmful memes. Mathias [2021] extended the Facebook Hateful Meme Challenge dataset by annotating the types of attack (e.g., Dehumanizing) and target type (e.g., race) attacked in the hateful memes. While the supplementary information could provide additional contexts to the hateful memes, it is still inadequate in informatively explaining the hateful memes.

Recent studies have attempted to identify and explain the subtle hateful connotations of hate speech. Sap [2020] developed the Social Bias Frame, a pragmatic framework that can capture knowledge regarding the biased implications of hate speech, such as its group reference and implied statement. Elsherief [2021] subsequently extended the Social Bias Frame to include implicit hate speech, which has a broader scope than social bias and stereotypes. Nevertheless, these existing studies have mainly focused on explaining text-based hate speeches. In this study, we aim to fill the research gap by proposing a new hateful meme dataset that includes informative reasons to explain the background contexts in hateful memes.

2.2 Hateful Meme Classification

Hateful meme classification is an emerging multimodal task that has gain popularity in recent years. Existing studies have explored classic two-stream models that combine the text and visual features to classify the hateful memes [Kiela et al., 2020; Suryawanshi et al., 2020], and fine-tuning large scale pre-trained multimodal models for the multimodal classification task [Lippe et al., 2020; Muennighoff, 2020; Pramanick et al., 2021b; Yang et al., 2022; Lee et al., 2021; Cao et al., 2022; Sharma et al., 2022b]. Nevertheless, most of the existing studies have focused on the hateful meme classification task without providing any explanation for the hateful memes. A recent study proposed a post-hoc explanation framework to examine the visual-text slur grounding learned by pre-trained multimodal models trained to perform the hateful meme classification task [Hee et al., 2022]. However, the framework still falls short in providing informative reasons to explain hateful memes. We postulate that the primary reason for underwhelming research studies explaining hateful memes is the lack of a dataset. Therefore, we propose a multimodal hateful meme dataset with informative reasons to encourage researchers to contribute solutions in this space. Specifically, this study will provide the benchmark dataset for the hateful meme explanation task and comprehensively evaluate state-of-the-art PLMs' capabilities to generate natural language reasons for hateful memes.

3 HatReDDataset

In this study, we propose HatReD 1 , a new multimodal hateful meme explanation dataset. Specifically, we recruited four native English speakers to annotate the underlying reasons for the hate speeches found in Facebook's Fine-Grained Hateful Memes dataset [Mathias et al., 2021]. To the best of our knowledge, this is the first multimodal hateful meme dataset annotated with hateful contextual reasons. In the subsequent sections, we will discuss the dataset construction process and provide a preliminary analysis of HatReD.

1 Note that researchers will have to agree with Facebook's data access agreement to download the memes

3.1 Dataset Construction

Fine-Grained Hateful Memes is a large-scale multimodal memes benchmark dataset that contains five standard kinds of incitement to hatred, including sexual, racial, religious, nationality and disability hatred. The selection of these curated memes conforms with the community standards on hate speech employed by Facebook 2 , which presents a pragmatic view of hate speech in memes.

Dataset and Annotation Preparation

The main challenge of explaining hateful memes is that the explanation often requires knowledge of relevant sociocultural backgrounds and societal prejudices. For example, recognizing that the presence of rainbow-striped flags may indicate a connection to LGBTQ movement, such as Pride Day. Therefore, to assist annotators in explaining hateful memes, we used the Google Web Detect API to extract web entities from the meme. The extracted web entities could provide the additional socio-cultural context for the images used in the memes. For instance, the API will return "Anne Frank" for the image used in Figure 1. The annotators are also encouraged to search about the unfamiliar extracted web entities and slurs in meme text on external knowledge bases such as Wikipedia and Hatebase 3 . For instance, annotators without prior knowledge of Anne Frank can search Wikipedia for information about her and events related to her, such as the Holocaust. Through this process, the annotators can deepen their knowledge of relevant cultural backgrounds and societal prejudices over multiple iterations of annotation and improve their annotation.

Reason Annotation

We trained the four annotators to produce high-quality reasons for each hateful meme. We present annotators with the meme, the social characteristics of the attacked target (e.g., nationality), the type of attack (e.g., contempt), and the extracted web entities. Annotators have to identify and explain the hate speech with three primary goals: (i) the annotated reasons should specify and cover all implied hate speeches in the meme, (ii) the annotated reasons should accurately express and reflect the underlying hate implication, and (iii) the annotated reasons should be fluent and grammatically correct. We also ensure the annotated reasons are consistent across the annotators by requesting the reasons to be written in one of the two following Hearst-like patterns: (i) <verb> <target> <predicate> or (ii) use of derogatory terms against <target> <predicate>, where <target> represents the attacked social target and <predicate> highlights the hateful implication.

2 https://www.facebook.com/communitystandards/hate_speech

3 https://hatebase.org/

Annotation Quality Control

We conducted four trial annotations to ensure that the annotators were competent and proficient for the task. In each trial, 20 unique hateful memes are sampled for each annotator, and the annotators are tasked to craft the hateful reasons. At the end of each trial, the annotators will assess the quality of hateful reasons written by other annotators. The annotated reasons are evaluated based on the following criteria:

• Fluency: Rate the structural and grammatical correctness of the reasons using a 5-point Likert scale. 1: unreadable reasons with too many grammatical errors, 5: well-written reasons with no grammatical errors.

• Relevance: Rate the relevance of the reasons using a 5point Likert scale. 1: reasons misrepresent the implied hate speech, 5: reasons accurately reflect the implied hate speech

At the end of each iteration, we present the evaluation ratings of their hateful reasons to the annotators and discuss how to improve the poorly rated hateful reasons. These discussions helped our annotators to improve the quality of their annotation.

3.2 Corpus Analysis

In total, HatReD dataset contains 3,304 annotated reasons for 3,228 hateful memes. Some memes may have multiple annotated reasons because they attack multiple social groups. The minimum explanation length is 5, the mean explanation length is 13.62, and the maximum is 31.

To examine the quality of the annotated reasons, we conducted human evaluation similar to our annotation trial on 1,200 hateful memes in our HatReD dataset. The annotators are tasked to evaluate the annotated reasons written by others. This translates into having three human evaluation results for each annotated reason. Table 2 shows the average fluency and relevance of the human evaluation on the hateful reasons for 1,200 memes. We observe a high average score of 4.97 and 4.81 for fluency and relevance, respectively, suggesting that the annotated reasons fluently capture the hateful speeches in the memes. Furthermore, we observe that the three evaluators have a unanimous agreement for fluency ratings in 93.9% of the evaluated annotated reasons, i.e., the evaluators rated the same score in 93.9% of the evaluated reasons. Similarly, the evaluators unanimously agree for their relevance ratings in 81.2% of the evaluated annotated reasons.

Table 3 illustrates the distribution of the social targets and hateful reasons found in the HatReD dataset. We observe significant variations in the number of social targets per social category. For example, there are 673 memes targeting the Sex social group, comprising three social targets (i.e., LGBT, Female, and Males). In contrast, the Nationality social group has 328 memes are attacking 34 unique social targets. Therefore, we expect more diverse and sparsely annotated reasons for hateful memes targeting Nationality social targets.

4 Hateful Memes Explanation

The availability of HatReD enables the exploration of a new task, hateful meme explanation. Specifically, we propose a natural language generation task where trained models generate the underlying reasons to explain hateful memes. Such generated reasons can help content moderators better understand the severity and nature of automatically-flagged hateful memes. Similar to [ElSherief et al., 2021], our work can alert the users when they intend to share a particular meme flagged as "hateful" and explain the underlying reasons. This enables users to recognize the severity of the hateful meme and possibly reconsider their decision to post the meme.

4.1 Task Definition.

We formulate the hateful meme explanation task as a conditional generation task dependent on meme content. Formally, given a dataset of paired hateful memes and reasons

{xi , ri }Ni=1 , the goal is to learn the generation of a fluent and relevant reason conditioned on the text information xiT and visual information xiV extracted from the hateful meme. We can refer to the reasons as a sequence of tokens ri = ri1 , · · · , riℓ , where we pad the tokens to a maximal length l. The training objective is defined as follows:

where θ denotes the model's trainable parameters.

4.2 Generative Models

A common model architecture used for conditional generation tasks is the encoder-decoder PLMs. Encoder-decoder PLM uses an encoder model to map the inputs to a sequence of continuous representations, which is then passed to the decoder to generate the output sequence. We train two types of PLMs in our experiments: (a) text-only PLMs that only accept text inputs; and (b) vision-language (VL) PLMs that accept text and visual inputs.

For data pre-processing, we obtain the text information xT by tokenizing the text that overlays on the meme image. The input differences in the two types of encoder-decoder PLMs require the visual information xV to be pre-processed differently. For text-only PLMs, we extracted the the meme's image caption using ClipCap [Mokady et al., 2021]. In addition, we applied Google Vision Web Entity Detection API and Fairface classifier [Kärkkäinen and Joo, 2019] to extract the meme's entities and demographic information, respectively. Finally, we concatenate the image caption, extracted entities, and demographic information to represent the visual information. For VL PLMs, we used Detectron2 [Wu et al., 2019] with bottom-up attention [Anderson et al., 2018] 4 to extract object regions and bounding boxes from the meme's image.

Model Training. We trained the PLMs on the annotated reasons in the HatReD dataset. As the hateful memes in HatReD may contain multiple reasons, we randomly sampled one reason for each hateful meme during training. The training objective is to minimize the cross-entropy loss:

Model Inference. Conditioned on the meme's text information xT and visual information xV , we generate three token sequences via two following decoding strategies: (i) greedy decoding, which generates a sequence by greedily selecting the most probable token at each time step; and (ii) beam search, which generates the most likely N token sequences at each time step and selecting the token sequence with the overall highest probability. We choose the token sequence with the highest overall score.

4 https://github.com/airsplay/py-bottom-up-attention

5 Experiments

5.1 Experiment Settings

Baselines. To understand the challenges of our proposed hateful meme explanation generation task, we fine-tune and evaluate encoder-decoder PLMs using the HatReD dataset. For text-only PLMs, we use T5 [Raffel et al., 2020], RoBERTa [Liu et al., 2019], and GPT2 [Radford et al., 2019]. As GPT2 is a decoder-only architecture, we adopt RoBERTa as its encoder. For VL PLMs, we benchmark [Cho et al., 2021] and VisualBERT [Li et al., 2019]. As VisualBERT is an encoder-only architecture, we utilized RoBERTa and GPT2 as its decoder in two different settings. The evaluation of these PLMs establishes the baselines for this new hateful meme explanation task.

Feature Space Alignment. To align feature spaces between encoders and decoders with different architectures, we place the models into a sequence-to-sequence model architecture with randomly initialized cross-attention layers added to each decoder block [Rothe et al., 2020]. The training error, back-propagated through the cross-attention layer, fine-tunes the weights and aligns the models' feature spaces.

Evaluation Metrics. We perform both automatic and human evaluations on the baselines. We adopt metrics commonly used in natural language generation tasks for automatic evaluation: (1) N-gram matching for word similarity; and (2) embedding-based metric for semantic similarity. For n-gram matching metrics, we compute the average BLEU [Papineni et al., 2002] and ROUGE-L [Lin, 2004]

scores. We also compute the harmonic mean of these two metrics. We compute the precision, recall, and F1 of the BERTScore [Zhang et al., 2019] for embedding-based metric. For human evaluation, we recruit human evaluators to assess the generated reasons on two aspects: fluency and relevance. The human evaluators are tasked to rate the generated reasons on the Likert scales described in Section 3.1. We also mitigate positional bias by presenting the generated reasons in a scrambled order.

5.2 Experiments on HatReD

We fine-tune the baselines over ten random seeds using the HatReD training set and evaluate the baselines' ability to generate fluent and relevant reasons for the hateful memes in the HatReD test set. Table 4 shows the distribution of the train-test split.

Table 5 shows the average automatic evaluation of the generated reasons for hateful memes in HatReD. We observed that T5 outperforms other PLMs across most evaluation metrics. Nevertheless, the best-performing model still performed badly with low N-gram matching scores (i.e., H. Mean = 0.256) and moderate BERTScore. The results suggest that the generated reasons differ substantially from the ground-truth reasons. However, the generated reasons could still convey similar meanings. Therefore, we perform human evaluations to assess the acceptability of the generated reasons.

Table 6 shows the human evaluation results of the generated reasons and human-written reasons for hateful memes in HatReD. The results indicate that while most PLMs can generate fluent reasons, the generated reasons scored poorly in terms of relevance. Specifically, T5, which has the highest average relevance score, still had a significantly lower score compared to human-written ground-truth reasons. The superior performance of T5, a text-only model, over multimodal models can be attributed to its larger model size and the abun-

Hateful Meme

(a)

(b)

dance of text data used for pretraining. T5 has approximately three times more parameters and is trained on 40 times more data than other models [Roberts et al., 2020]. Nevertheless, the baseline PLMs' performance suggests the difficulty of the hateful meme reason generation task. We hope that the availability of HatReD enables researchers to design novel and better reason generation models for the task.

5.3 Experiment on Unseen Dataset

To further evaluate the usefulness of HatReD in explaining hateful memes, we conduct a domain adaption experiment where the baselines are trained on HatReD and tested on an unseen dataset. Specifically, we applied two top-performance baselines, namely, VisualBERT-RoBERTa and T5, to generate the reasons for hateful memes in the Multimedia Automatic Misogyny Identification (MAMI) 5 datasets [Fersini et al., 2022]. The MAMI dataset contains hateful memes that discriminate against females. As there are no ground truth reasons for the MAMI dataset, we perform a human evaluation of the generated reasons for 50 randomly sampled MAMI hateful memes.

5 https://competitions.codalab.org/competitions/34175

Table 7 shows the human evaluation results on the generated reasons for the hateful memes in MAMI. Similarly, we observe that the baselines are able to generate fluent reasons. We also noted that the generated reasons have much lower relevance scores, which again suggests the difficulty of the hateful meme reason generation task. Nevertheless, we observe that 20% of the hateful misogynous memes are rated highly relevant (i.e., relevancy ≥ 4). The promising result demonstrates the possibility of performing domain adaption, where generative models are trained on HatReD to generate the reasons for hateful memes in unseen domains.

5.4 Case Studies

Besides performing quantitative evaluations on the hateful meme reason generation task, we also perform empirical analysis on the best performing PLM's (i.e. T5 and VisualBERT-RoBERTa) generated reasons for hateful memes in the HatReD and MAMI datasets. Specifically, for this analysis, we examine generated reasons with either high (i.e., ≥ 4) or low (< 2) relevance scores.

Table 8 presents examples of generated reasons for hateful memes in the HatReD. We found that both models generate reasons that accurately reflect the implied hate speech for the meme 8a. Notably, while both generated reasons reveal the intention to undermine and objectify women, only the reason generated by the text-only T5 model captures the intention to sexualize women. On the other hand, the Hateful Meme

(b)

(a)

Table 9: Hateful memes from MAMI datasets with reasons generated by VisualBERT-RoBERTa and T5 models. The highlighted green and red words in the generated explanations outline the correct implications of hate and hallucinations (i.e. misinformation) present in the hateful memes, respectively.

T5 and VisualBERT-RoBERTa models generate reasons that misidentify the social target as Muslims and Immigrants for the meme 8b respectively. Examining the text-only T5 model that relies on extracted visual information, we observe that the image caption captures the essential visual information of a woman lying on a bed in meme 8a but fails to capture the crucial demographic information of the white man in meme 8b. The absence of demographic information in the image caption might be critical in associating the face of terrorism with white people, which led to the inaccurate identification of the social target in the generated reason for meme 8b. To examine this possibility, we make manual correction to the meme 8b's image caption and explore the input saliency for the generated social target. We found that manually correcting the image caption helped to generate a new reason that accurately identifies the targeted social target, shown in Figure 8. Additionally, in Figure 2, we explore and show the input saliency of the original and modified inputs via Integrated Gradients [Sundararajan et al., 2017]. The results demonstrate that the manual correction caused the model to focus more on the words "face," "terrorism," "senior," "white," and "man," which are critical word associations required to identify the social target. These observations suggest the significance of having reliable visual information extractors to capture accurate visual information. As for the VisualBERT-RoBERTa model, the generation error is likely due to the model's inability to associate information from different modalities or understand detailed visual information such as the fact that the senior man is a white person. Nonetheless, these hallucinations demonstrate the limitation of state-of-the-art generation models and the potential for future improvement.

Table 9 showcases examples of generated reasons for hateful memes in the MAMI dataset. We observed that the generated reasons are often accurate for memes containing antifeminism or patriarchal messages, as shown in meme 9a and 9b. This can be attributed to the high percentage of anti-feminism and patriarchy hateful memes in the training dataset, where approximately 26% of the HatReD's hateful memes in the female social category express anti-feminism and patriarchy messages. However, the generated reasons are found to be inaccurate when dealing with memes in new domains. For example, T5 model hallucinates that the meme 9c implies transgender women are sexual objects, despite no indication of this in the textual or visual modalities.

6 Conclusion

In this paper, we introduced HatReD, a new multimodal hateful memes dataset annotated with the underlying hateful contextual reasons. To the best of our knowledge, this is the first hateful meme dataset with written explanations. The availability of HatReD dataset opens the possibilities of training generative models to generate reasons for hateful memes, which can aid content moderators in understanding the severity and nature of flagged content. We defined a new conditional generation task to automatically explain hateful memes, and conducted extensive experiments using state-ofthe-art PLMs to establish task baselines. Nevertheless, the quantitative and qualitative evaluations highlighted the difficulty of the hateful meme reason generation task. We hope that HatReD and our benchmark study will encourage more researchers to develop better models to generate fluent and relevant reasons for hateful memes. For future works, we aim to expand HatReD further to cover more domains of hateful memes. We will also explore different strategies to improve the existing reason generation model, such as using retrieval augmentation to incorporate explicit knowledge or improving the utilization of implicit knowledge in PLMs.

Ethical Statement

Research indicates that annotating hateful or offensive content can have negative effects. To protect our annotators, we establish three guidelines: 1) ensuring their acknowledgment of viewing potentially hateful content, 2) limiting weekly annotations and encouraging a lighter daily workload, and 3) advising them to stop if they feel overwhelmed. Finally, we regularly check in with annotators to ensure their well-being.

Another consideration is the usage of Facebook's hateful memes; users will have to agree with Facebook's usage agreement to gain access to the memes. The usage of Facebook's hateful memes in this study is in accordance with its usage agreement. Respecting Facebook's licenses on the memes, the HatReD dataset only contains the annotated reasons for the Facebook memes, but not the hateful memes; users will have to download the memes from the Facebook Hateful Meme challenge separately.

One of HatReD's goals is to train AI systems to provide detailed warnings that explains the hateful nature of the meme content, raise users' awareness and discourages its dissemination. Nevertheless, we acknowledge the potential for malicious users to reverse-engineer and create memes that go undetected (or misunderstood) by the HatReD-trained AI systems. This is strongly discouraged. In our paper, HatReD is utilized as training signals for post-hoc explanation generation (i.e., after the meme is flagged as hateful), not hateful meme detection. Researchers and platform providers should be cautious about including HatReD as training signals for hateful meme detection.