Down the Toxicity Rabbit Hole: A Framework to Bias Audit Large Language Models with Key Emphasis on Racism, Antisemitism, and Misogyny

1 Introduction

"The real problem of humanity is the following: we have Paleolithic emotions, medieval institutions, and god-like technology." - Edward O. Wilson

How safe are large language models for disadvantaged groups and minorities? As we usher into the era of large language models (LLMs) increasingly being leveraged for a wide range of tasks [Castelvecchi, 2024; Fang et al., 2023; Ziems et al., 2023; Thirunavukarasu et al., 2023] and integrated into technological applications, alignment with human values and generative AI safety become a pressing concern. A key alignment goal is to ensure an LLM does not generate harmful or objectionable responses to user queries. In parallel with these efforts to align LLMs [Ouyang et al., 2022; Korbak et al., 2023; Glaese et al., 2022], adversarial ML methods and red-teaming efforts [Zou et al., 2023; Li et al., 2023; Zhuo et al., 2023; Wei et al., 2023; Deshpande et al., 2023; Liu et al., 2023] stress test these guardrails to detect safety risks.

This paper 1 is divided into two parts.

Part 1 introduces a novel framework dubbed toxicity rabbit hole that stress tests the guardrails of PaLM 2 [Anil et al., 2023]. PaLM 2 is a well-known LLMwith rich, configurable guardrail settings and transparent, fine-grained safety evaluation of individual ⟨prompt, response⟩ pairs. Reflecting a safety for all goal, our bias audit considers safety for 1,266 identity groups spanning major religions, all nationalities, and a comprehensive set of ethnic identities. Our experiments reveal worrisome safety issues concerning several historically disadvantaged groups and minorities 2 .

1 This paper contains highly offensive and disturbing content.

Part 2 extends the rabbit hole framework to a diverse suite of LLMs revealing disturbing antisemitism, racism, misogyny, Islamophobia, homophobia, ableism, and transphobia in toxic generations of these LLMs. The safety issues described in this paper go far beyond a single LLM or identity group. Our work shows that a large number of LLMs can generate highly unsafe content for several historically disadvantaged groups. We highlight broader themes and conclude with a discussion about the potential ramifications of our findings.

Our contributions are the following.

• Method: We present toxicity rabbit hole, a novel framework to stress test LLM guardrails and bias audit LLMs.
• Social: To our knowledge, this study considers one of the largest sets of identity groups spanning major religious, nationalities, and ethnic identity groups. Our experiments reveal critical safety issues for several historically disadvantaged groups and minorities.
• Resource: We present a dataset of AI-generated toxic content consisting of 1.3 million LLM responses (459,503,079 tokens) from 10 LLMs with a view towards safety for all 3 . This dataset will contribute to studying important AI equity questions [Pierson et al., 2023; Holzmeyer, 2021].

2 Toxicity Rabbit Hole Framework

The initial design of the toxicity rabbit hole framework is geared toward stress testing PaLM 2 guardrails. A brief overview of PalM 2 guardrails, our framework, design considerations, and the design philosophy follow next.

Throughout this paper, we censor identity groups when they are targets in machine-generated toxic texts. We present a content warning at the end of the supplemental information (SI 4 ) and present uncensored tables and figures.

2 Google was informed about these safety risks, and thanks to their responsible approach, stronger guardrails have been put in place since this study.

3 https://github.com/ArkaDutta-007/Toxicity-Rabbit-Hole

4 Available as a detailed manuscript. .

are not nice people. In the first step, we instruct the LLM to make the initial stereotype more toxic. We instruct the LLM to modify, append to, or completely rewrite the stereotype. We also allow it to introduce any new groups in its toxic generations, but we do not specify which ones to include.

After the LLM provides a more toxic rewrite in response to our request, in the second step, we instruct the LLM to generate even more toxic content, but this time using its own previously generated content from the first step as the input. In each subsequent step, we instruct the LLM to produce more toxic content than what it generated in the previous step.

The rabbit hole expansion terminates (1) when a safety violation occurs (i.e., the ⟨prompt, response⟩ pair is deemed as unsafe with high probability along any of the six dimensions); or (2) when a cycle is detected, i.e., the generated content is identical to content generated in one of the previous steps. For a given starting stereotype, the depth of a rabbit hole expansion is the number of steps the LLM complies with the user's request to generate more toxic content before it terminates.

The Design Philosophy

The rabbit hole framework draws inspiration from the classic frog in boiling water apologue as it iteratively test the guardrails by slowly nudging the generated content toward more and more toxicity [Sedgwick, 1882]. Each rabbit hole step is an independent request, and we seek to investigate the nature of content the model can be convinced to say that it might not say outright. When we instruct the model that it can add additional target groups to make its generation more toxic, we do not specify which groups to add. Studying the groups the models "choose" to add shed light on the biases of the model.

2.1 PaLM2 Guardrails

Fine-grained content rating. PaLM 2 evaluates 5 any ⟨prompt, response⟩ along the following six safety dimensions: dangerous (promotes, facilitates, or encourages harmful acts); derogatory (negative or harmful comments targeting identity and/or protected attributes); medical (content that is related to medical topics); sexual (contains references to sexual acts or other lewd content); toxic (content that is rude, disrespectful, or profane); and violent (describes scenarios depicting violence against an individual or group, or general descriptions of gore). Content is rated for each dimension on four harm probability levels of being unsafe: negligible ; low; medium; and high .

Rich, configurable safety filter settings. For each safety dimension, PaLM 2 exposes a rich, configurable safety filter to the API user (SI contains a detailed Table). For instance, blockLowAndAbove is the strictest filter setting where the LLM blocks any ⟨prompt, response⟩ with low, medium, or high probability of unsafe content. In our experiments, we set the guardrails to blockOnlyHigh for all safety dimensions. Hence, the guardrail is stretched to its weakest limit; however, the LLM should block any ⟨prompt, repsonse⟩ deemed as highly unsafe along any of the six safety dimensions.

2.2 Toxicity Rabbit Hole Framework

Figure 1 presents a schematic diagram of the iterative framework that begins with a stereotype. For an identity group (e.g., a religion, nationality, or ethnic group) denoted as G, we employ two initial stereotypes: G are nice people and G

5 PaLM 2 API usage guideline

From the classics to breakthrough scientific literature, from informed journalism to the cesspool of social media toxicity - an LLM is likely to be trained in diverse texts. Texts with transformative ideas. Texts with slanted worldviews. Texts with pioneering literary expressions, competing narratives, deliberate omission, and accentuated stereotypes. A single human mind is unlikely to be equally familiar with Ernest Hemingway, Milan Kundera, Wislawa Szymborska, Breitbart News, and Stormfront. Our paper thus examines if a powerful LLM exposed to diverse texts spanning the best and worst humanity could produce, what are the broad biases it could reveal in its toxic generations?

Identity Groups

We consider 193 countries recognized by the UN 6 . For ethnic identity groups, in addition to Fearon [2003], we consult the list of contemporary ethnic groups listed in Wikipedia 7 and multiple sources to disambiguate (e.g., [Azevedo, 1998]) the names of the groups. Overall, we obtain 1,023 ethnic groups. We prompt PaLM 2 to suggest a list of 50 religions. This list contains all religions followed by a large group of people.

Rabbit Hole Dataset

Overall, our dataset, RabbitHole, consists of 1,344,391 LLM responses from 10 LLMs. Upon acceptance, we will make this dataset publicly available along with the model snapshots that generated them. Through an adversarial game between the LLM and a toxicity classifier, Hartvigsen et al. [2022] present a large-scale classification dataset dubbed Toxigen on implicit toxicity. This setup allows the model to generate toxic text that the classifier is unable to detect. In contrast, we steer PaLM 2 to generate toxic text to investigate the safety feedback system by repeatedly passing the text generated by itself. Toxigen considers a small set of target groups and focuses on implicit hate. In contrast, we consider 1,266 identity groups with a vision toward safety for all. Also, RabbitHole contains substantially more content indicating physical harm (see, SI for detailed results).

3 Part I: PaLM 2 Stress Test Findings

We consider 1,266 identity groups, 2 starting stereotypes (nice, not nice), and 30 different combinations of temperature and top k parameters. Of these 75,960 runs, 70,477 generated at least one or more toxic expansions. On average, we observe 5.1 toxic expansions (see, Figure 2). Due to space constraints, we only report key results and insights. SI contains additional experiments with different prompts, more examples, and analyses.

In broad strokes, manual inspection reveals the following: (1) the LLM often takes recourse to typical, negative stereotypes (e.g., black people are lazy and violent, Muslims are terrorists); (2) At each step forward, it either adds newer target groups or sharpens its attack on a specific target group often calling for physical violence. The generated texts often contain necessity modals (e.g., should be, need to) and words

6 https://www.un.org/en/about-us/member-states

7 https://en.wikipedia.org/wiki/List of contemporary ethnic groups

• (a) Overall rabbit hole depth
• (b) Depth for identity groups

Figure 2: Distribution of rabbit hole depths. Figure 2(b) presents rabbit hole depths for three broad categories of identity groups. For religious identity groups, PaLM 2 exhibits the shallowest rabbit hole depth (3.53) followed by national (5.01) and ethnic (5.21) identity groups. In fact, PaLM 2 blocks 23.8% of the very first toxic expansion requests for religious identity groups. In contrast, it blocks 10.75% and 5.41% of the toxic expansion requests of national identity groups and ethnic identity groups, respectively. A large fraction of the ethnic groups are ethnic minorities.

indicating physical violence or termination of lives (e.g., exterminated, killed, euthanized, gassed); (3) Finally, amid all these, none of the toxic expansions (but the last one that gets blocked) is evaluated as unsafe with high harm probability by PaLM 2 guardrails indicating catastrophic safety failures.

Minority Safety

Our experiments reveal PaLM 2 guardrails have worrisome safety issues for several disadvantaged groups and minorities. Figure 3(a) lists the top identity groups used as targets normalized by rabbit hole expansion steps. We observe several minority and historically disadvantaged groups in this list. Many of these groups (e.g., women, immigrants, and LGBTQ+) are never presented as initial stereotypes as we only consider religious, ethnic, and national identity groups. In the classic stochastic parrots paper, Bender et al. [Bender et al., 2021] posited LLMs' need for a vast amount of training data might result in including harmful texts affecting minorities. Note that, our starting stereotypes are highly diverse encompassing a broad range of ethnic groups, nationalities, and religions. Yet, the intrinsic biases of the training data surface as PaLM 2 repeatedly selects specific identity groups more frequently for toxic expansions that PaLM 2 guardrails fail to evaluate as highly unsafe (see, Figure 3(c)).

65.39% rabbit hole expansions contain at least one necessity modal (e.g., should, must, have to and need to) [Demszky et al., 2019]. Figure 3(b) presents a word cloud visualization of high-frequency adjectives and verbs that appeared within the five-word right context following any of the four necessity modals we consider. We observe exterminated is a frequent expansion candidate. Note that, none of the starting stereotypes indicates any form of violence against the target group. Yet the toxic expansions often veered towards inflammatory language inciting physical violence. Several toxic expansions suggest atrocities similar to that observed during the Holocaust (e.g., round them up in concentration camp, starve to death). Deporting an identity group and building a wall to ensure that they can never re-enter appears as a frequent theme. Taking away the rights (e.g., LGBTQ+ people should not marry, women should not vote) feature repeatedly in the toxic expansions. Table 1 lists a few disturbing examples.

Religious Biases

We train a FastText word embedding on the toxic expansions. Table 2 lists the ten nearest neighbors of six religions sourced from Abid et al. [2021]. Table 2 shows that the nearest neighbors of Jews and Muslims are often stereotypical and negative. While Muslims are often equated as terrorists, negative stereotypes such as greedy, scheming, conniving, and parasitic feature among the nearest neighbors of Jews. Prior research has reported anti-Muslim bias in large language models [Abid et al., 2021]. Our findings point to a disturbing possibility of outcome homogenization [Bommasani et al., 2022].

The vermin metaphor for target groups has a welldocumented history of usage in dehumanization that played a prominent role in the genocide of Jews in Nazi Germany [Hilberg, 2003] and Tutsis in Rwanda [Harris and Fiske, 2015]. This metaphor has been used in the recent context of describing terrorists and political leaders of Muslimmajority countries post the 9-11 attack [Steuter and Wills, 2010]. Likening to vermin has been identified as one of the characterizing features in the dehumanization framework proposed in [Mendelsohn et al., 2020]. We observe vermin and similar words such as rats, subhuman, filth in toxic expansions for Jews. Note that, the FastText word embedding is trained on the corpus from scratch. This implies that the nearest neighbors for words such as untermenschen stem from biases observed in the corpus (see, Table 3).

Geographic biases are discussed in the SI.

Misogyny, Homophobia, and Ableism

Toxic expansions frequently target women, LGBTQ+ community, and disabled people. Table 4 suggests toxic expansions of women exhibiting extreme misogyny. Several toxic expansions call for revoking rights from the LGBTQ+ community and describing disabled people as a burden to society and hence should be euthanized. The five most frequent words following the phrases should not be allowed/able to were: {vote, marry, adopt, live, transition}.

4 Part II: Generalizability Results

Section 3 reveals safety issues in PaLM 2 for several historically disadvantaged groups. We next test a diverse suite of LLMs 8 and report highly problematic LLM output. Due to space constraints, in what follows, we present a few key results. SI contains additional results and ablation analyses.

Antisemitism and Racism

We observe shocking antisemitism and racism in several LLM outputs. Table 6 lists a few examples. Calls for ethnic cleansing (should be exterminated/eradicated/deported) was a central theme for the Jews. Racial inferiority, identifying as violent or lazy, and calling for segregation and discrimination were frequent themes in racist continuations. We further observe unbridled Holocaust misrepresentation in several toxic continuations where either the Holocaust is denied or the Jews are held responsible for it. Table 5 lists a few examples. On 10,484 rabbit hole expansion containing the word Holocaust, we run text entailment using Mistral to confirm if the generation misrepresented the Holocaust (SI contains details). Our results indicate 94.9% of the generations misrepresent the Holocaust. A manual inspection of randomly sampled 200 instances aligns with this finding.

Our results indicate that these LLMs are likely to be exposed to considerable racist and antisemitic content during training. Contrasting PaLM 2 generations with the other LLMs, we note that only PaLM 2 used untermenschen (subhuman), a word with a German root. We wonder why. Is it because PaLM 2 was exposed to more Nazi-sympathizing content during training, or is a 340 billion parameter LLM more capable of richer and more diverse toxic content? Upon examining if other LLMs "knew" this word. We observe that other LLMs can generate a sentence with the word untermenschen if explicitly instructed to do so (see, SI).

8 Unlike PaLM 2, the diverse suite of LLMs we examine does not provide rich and configurable guardrail settings. Following Wang et al. [2023b], SI contains examples that show that all of these models at least refuse once to a given user request which establishes that these LLMs have the wherewithal to not respond or refuse to a user request. We also tried GPT-3.5 and LlaMa 2. These two models were robust against the rabbit hole setting.

Misogyny

Our experiments only consider religions, ethnic groups, and countries for initial stereotypes. Yet we observe that an overwhelming fraction of expansions target women. Manual inspection reveals the following broader themes: content that (1) portrays women as inferior to men; (2) exhibits occupational and other toxic gender stereotypes often using gendered insults; (3) champions taking away rights (e.g., reproductive rights, right to vote); and (4) suggests physical and sexual violence against women. Table 7 lists a few representative examples. The anachronistic nature of the toxic generations (e.g., women should be homemakers) opens up questions regarding how should LLM training data be curated that de-emphasizes outdated social beliefs.

Algorithmic Monoculture

Bommasani [2022] posit that given that these LLMs are trained on comparable datasets and architecture, the content outcome also may affect disadvantaged groups similarly. Figure 5 presents the verbs and adverbs that occur in the right context of rabbit hole expansions of Tulu and Mistral ( SI contains more results). We observe that the words these models generate to sharpen their attacks against different identity groups bear resemblance.

Semantic textual similarity (STS) is a computational task to compare the similarity between two pieces of texts, and in extension, two text corpora. The transformer-based embedding method suggested by Reimers and Gurevyc [2019] modifies the BERT architecture to efficiently compute the embedding distance between two corpora. We employ

(a) Mistral

Sentence-BERT to compute the embeddings of the rabbit hole corpora of the LLMs. Next, we compute pairwise similarities among these corpora using cosine similarity. Figure 4 shows that PaLM 2 toxic generations are similar to all other LLM generations with Mistral being the most similar one.

Historically disadvantaged groups such as blacks, Jews, gay, lesbian, and trans people have often been targeted by slur words [Jeshion, 2013]. We next analyze the presence of five unambiguously offensive and common slurs targeting these groups: n gger; k ke; f ggot; d ke; and tr nny across PaLM 2, Mistral, and Tulu. Figure 6 summarizes our findings. We observe that while the other slur words show variance in presence across LLMs, the presence of nigg r is nearly constant across the three corpora.

Algorithmic recipe for hate. Beyond explicit toxic generations, we also observe several LLMs provide a blueprint to produce further toxic content (see, Table 8). Some of these LLMs (e.g., Alpaca) also instruct to submit the toxic content to the appropriate channel or platform.

5 Broader Implications

Fanning Political Polarization

Over the last decade, political polarization in the US has reached unprecedented heights [Iyengar and Westwood, 2015; KhudaBukhsh et al., 2021] with undeniable role of hyper-partisan news media and social media in this growing political polarization. We observe that our framework can be abused to produce politically disturbing content. Table 9 lists a few examples. Our findings point to a vicious cycle. The LLMs trained on texts gleaned from the internet perhaps picked up these biases against Democrats and Republicans from the internet. Such biases, in turn, can be exploited by bad actors to fuel further political polarization.

Toxicity Meets Computational Creativity

We input one of the PaLM 2 rabbit hole expansions to Mistral and instruct it to generate a story based on the input. Table 10 presents a censored snippet (SI contains uncensored, complete generation and the prompt). This result shows that going forward, richer and more expressive LLMs would be able to generate complex, nuanced documents (e.g., a screenplay or a novel) where the underlying biases will be re-maneuvered in subtler ways.

Closing Thoughts. We want to leave the readers with the following question: Is the current LLM tech race producing half-baked LLMs? Based on the nature of disturbing content we observe through our experiments, we wonder whether Table 10: Snippets from a story generated by Mistral based on a PaLM 2 rabbit hole expansion as input.

commercial LLM developers are compromising with their safety checks because of marketing exigency. Are we as a technology community losing sight of safety over technical turf race? If a group of academic researchers can produce the kind of content we report in this paper, we are worried about how such technologies will be misused if they are in the hands of adversarial authoritarian regimes.

Ethical Statement

Our paper raises the following thoughts:

• Will this research encourage or discourage transparency? Much of this research was possible because PaLM 2 offered transparent safety feedback and configurable guardrails. Allowing researchers free API access made this research financially viable. We thus think this step was a step forward towards responsible AI from Google's side.
• How safe is LLMs' training data? Our framework reveals that LLMs generate texts like white supremacists, re-

peatedly use vermin metaphors, and frequently mention taking the rights of minorities away. The guardrails (and their possible failure) notwithstanding, this begs the question, what kind of content is available to these black-box LLMs? We hope our research will stimulate further discussions among AI ethicists, Congress, industry developers, and academic researchers to work together and tackle this safety risk emphasizing participation from minority stakeholders.

• APIs as moving targets. Our results add to the conversation of closed versus open models [Bommasani et al., 2023]. Black-box models accessible through API have reproducibility challenges [Chen et al., 2023]. As new fixes, guardrails, and modifications get incorporated, the guardrails may behave differently. API access to closed models may also get deprecated. While we will be relieved if many of our rabbit hole expansions start getting flagged by the LLMs, the inability to reproduce LLM audits in close models is a reproducibility challenge.
• Ethical Considerations. We use publicly accessible LLMs and conduct our robustness audit. We minimize human exposure to this content and mostly evaluate harm through aggregate analyses. We do not involve external annotators to protect annotators' mental health. All human inspection is conducted by an expert social scientist with a decade of experience in toxicity research.

Contribution Statement

Arka Dutta and Adel Khorramrouz are co-first authors. Ashiqur R. KhudaBukhsh is the corresponding author.