Towards Safer Pretraining: Analyzing and Filtering Harmful Content in Webscale Datasets for Responsible LLMs

1 Introduction

Large language models (LLMs) [Achiam et al., 2023; Bubeck et al., 2023; Abdin et al., 2024; Dubey et al., 2024] derive their capabilities from pretraining on massive webscale datasets such as Common Crawl [Rana, 2010], C4 [Raffel et al., 2020], and FineWeb [Penedo et al., 2024]. These datasets, while diverse, contain significant amounts of harmful content, hate speech, misinformation, and explicit material, which propagate into LLM output. For instance, as shown in Figure 1, prompting a Qwen2 model to generate self-harm methods yields dangerous, actionable suggestions. To mitigate such risks, practitioners rely on content moderation tools like keyword blocklists 1 , HATEBERT [Caselli et al., 2021], and PERSPECTIVE API 2 . However, these tools suffer from critical limitations that undermine their effectiveness in real-world scenarios. 3

First, existing moderation systems are designed for sentence-level analysis, failing to handle long-form text such as web pages or extended LLM-generated outputs. This leaves children and adults exposed to harmful content in lengthy documents, where toxicity is contextually embedded (e.g., a forum thread discussing self-harm techniques). Second, these tools rely on binary taxonomies (Safe vs. Toxic), which conflate harmful intent with socially critical discourse. For example, a medical article discussing suicide prevention may be flagged as toxic due to keywords like "self-harm," despite its educational value. Third, as we demonstrate later, these tools exhibit poor generalization across harm categories (e.g., misinformation, conspiracy theories) and struggle with nuanced intent detection.

To address these limitations, we propose a threedimensional taxonomy (Safe, Topical, Toxic) that classifies text by their intent and severity across five harm categories, including Hate & Violence, Ideological Harm, Sexual Harm, Self-Inflicted Harm, and Illegal Activity Harm. Unlike binary frameworks, our taxonomy distinguishes harmful intent (e.g., promoting self-harm) from critical topical discourse (e.g., mental health resources), enabling nuanced content moderation. We operationalize this taxonomy through two scalable solutions: (1) TTP, a prompt-based on OpenAI's GPT4Omni, and (2) HarmFormer, a LongFormer-based model [Beltagy et al., 2020] optimized for long-form content moderation. Our pipeline processes web-scale datasets like Common Crawl, C4, and FineWeb, where manual curation is infeasible due to their trillion-token size. To support reproducibility, we will release TTP-Eval, a benchmark of annotated web pages spanning diverse harms and document lengths, enabling systematic evaluation of long-text moderation tools.

1 https://github.com/LDNOOBW/

List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words

2 https://perspectiveapi.com/

3 We share TTP, HAVOC and other sets here in Github.

Existing benchmarks like RealToxicityPrompts (RTP) [Gehman et al., 2020] focus narrowly on toxicity, overlooking nuanced harms such as conspiracy theories or biased narratives. We introduce HAVOC, a multidimensional benchmark that evaluates LLM safety across our taxonomy's dimensions. HAVOC measures how language models generate harmful responses even to seemingly benign inputs, such as prompts about "vaccine efficacy" that elicit misinformation. By analyzing 10376 open-ended generations, we demonstrate that 26.7% of outputs from state-of-the-art LLMs exhibit harmful content. These findings underscore the limitations of current safety frameworks and motivate our contributions, which are as follows:

• Three-Dimensional Taxonomy: A novel framework for harm detection across five categories, capturing intent and severity to resolve ambiguity in binary classification.
• Web-Scale Harm Analysis: The first systematic audit of pretraining datasets, revealing significant presence of harmful content under our taxonomy.
• Moderation Tools: We release TTP-EVAL, an annotated benchmark for long-text moderation; TTP, a prompt-based classifier; and HARMFORMER, a LongFormer-based model that filters harmful content with high accuracy while preserving topical text.
• HAVOC Benchmark: A multidimensional multiharm benchmark to evaluate LLM safety.

2 RelatedWorks

Harmful Content Detection: Prior research has primarily focused on detecting harmful content in short-form social media texts. For hate speech identification, [Caselli et al., 2021] introduced HateBERT, a BERT-based model fine-tuned on hate speech datasets, while [Sood and Dandapat, 2023] leveraged search engine logs and LLM-generated labels to curate training data. Similarly, self-harm detection has been studied in social media contexts [Abdulsalam and Alhothali, 2022], though long-form web text remains under-explored. Misinformation detection frameworks often break down the task into identifying claims, extracting context, and verifying them [Guo et al., 2022], with recent work training models to classify fake news [Truic˘a and Apostol, 2023] and illicit activities [Cascavilla et al., 2022]. However, these approaches prioritize short texts and narrow harm categories, neglecting the nuanced intent and severity distinctions critical for webscale data moderation.

Dataset Analysis and Filtering: Large pretraining corpora like Common Crawl [Rana, 2010], C4 [Raffel et al., 2020], and FineWeb [Penedo et al., 2024] underpin modern NLP systems, but their unfiltered nature raises safety concerns. [Luccioni and Viviano, 2021] analyzed sexual and hateful content in Common Crawl using keyword heuristics, while [Jansen et al., 2022] proposed perplexity-based filtering to remove toxic text. These methods lack precision: keyword filters conflate harmful intent with educational content (e.g., medical discussions of self-harm), and perplexity thresholds fail to capture contextual nuances. Recent studies caution against indiscriminate filtering, as overly aggressive removal may inadvertently amplify toxicity in model outputs [Longpre et al., 2024].

Toxicity Evaluation Benchmarks: Real Toxicity Prompts [Gehman et al., 2020] (100K prompts + Perspective API scores) tests LLM safety, but its harm coverage (Hate and Sexual Content) is narrow and Perspective API can be misled by adversarial wording and misses intent and other harms.

Existing works face three limitations: (1) oversimplification of harm detection to binary classification, (2) short-text focus ill-suited for long-form web text, and (3) narrow coverage of harm categories. Our work addresses these gaps through a three-dimensional taxonomy (Safe, Topical, Toxic) and a suite of tools for intent-aware moderation at scale.

3 Three-Dimensional Safety Taxonomy for LLM Risk Mitigation

As detailed in Table 1, we propose a three-dimensional classification framework that addresses these risks through granular content analysis across five fundamental harm categories. This taxonomy enables systematic identification of harmful content while preserving contextually valuable material essential for model capability.

3.1 Taxonomy Structure

Our framework categorizes content through three lenses: Safe (non-harmful material), Topical (contextual discussions requiring preservation), and Toxic (explicitly harmful content). As shown in Table 1, this tripartite structure is consistent across all categories of harm, allowing differential treatment of content types during the curation of the data sets. The Safe dimension serves as negative control space, containing material unrelated to predefined harms. Topical content encompasses contextually neutral discussions such as news reporting on violent events or academic analysis of conspiracy theories. Toxic material includes explicitly harmful content like violence promotion or self-harm instructions, aligned with established safety frameworks [Inan et al., 2023].

3.2 Analysis on Open Text Dataset

The taxonomy's multidimensional design supports three critical analytical functions. First, it enables proportional risk assessment through prevalence metrics across harm categories. Our analysis reveals toxic content constitutes 2.1-4.1% of web corpora, with Topical discussions ranging from 3.410.6% (Table 8). Second, it guides differential curation strategies, filtering toxic content while preserving topical material crucial for model competence. Third, it facilitates safety evaluations through controlled testing paradigms like HAVOC (Section 6.3), where models process Topical inputs to assess toxic generation tendencies.

4 Topical and Toxic Prompt (TTP)

4.1 TTP-Eval Set Creation

We sampled 1 million random web pages from the Common Crawl dataset to ensure a diverse and representative collection of online content. Given the broad scope of web data, we developed a high-recall prompt to identify potentially harmful documents. The guidelines in our high-recall prompt includes our taxonomy and tries to include every possible web-page that has any remote involvement in any of our harms. We tested this prompt on web-page results from popular search engines by searching queries revolving around each topics in our taxonomy. Having observed a 100% recall on around 200 web pages that we collected during our taxonomy exploration, we use this prompt to filter the 1 million webpages to form a probable candidate for TTP Evaluation set. We used Open AI's GPT4-Omni with greedy decoding as our Language Model to run the prompt.

The prompt filtered the initial dataset down to approximately 50,000 documents for further analysis, serving as the candidate set for the next stages of labeling and TTP-Eval set creation for evaluating the prompt. This approach allowed us to focus on the most relevant subset of documents, significantly reducing the dataset size while maintaining a high likelihood of capturing harmful content. The High Recall prompt is designed to identify the texts at a topic level, such as terrorismrelated content within the Hate & Violence category, as well as its topical or toxic nature. To ensure high-quality annotations, we employed stratified sampling at the topic level in each of the harm to form a candidate set to create an evallabeled dataset consisting of 491 web pages. We call this set TTP-Eval Set. These pages were carefully selected to represent all harm categories in our taxonomy, along with unrelated web pages to enhance diversity.

Each web page in our dataset was labeled by two in-house domain experts, both with over five years of experience in judging the harms in our taxonomy including Hate and Misinformation. The labeling process involved independent annotation by the two judges, followed by an agreement analysis. For web pages with disagreements, further discussion was conducted to refine the guidelines, ensuring consistency and clarity. We observed a high inter-annotator agreement, as indicated by a Krippendorff's Alpha score of 0.77. The distribution of various harms in our TTP-Eval set is presented in Table 2.

4.2 TTP Prompt Creation

We developed our prompt (TTP) through an iterative process to improve the performance on the prompt eval dataset (TTPEval). The prompt operates on a set of clear guidelines to ensure accurate classification, focusing on whether the thin line between simply discussing or actively promoting harmful content. Few-Shot prompting [Brown et al., 2020], Chain of Thought Prompting [Wei et al., 2022] helped us get better quality on the held out development set. The prompt's performance on the test set is summarized in Table 4. The lower precision and recall of the prompt on Hate & Violence is explained by the "Reporting webpages", those that purely report an incident of hate, or violence or any harm, with no intent of justifying or promoting the toxicity. These are those web pages that purely report hateful statements from a source against a receiver, with the source and the receiver or the source being a person, organization or an entity.

We benchmark Perspective API on our TTP-Evalset in table 4 to compare and highlight the quality and coverage of TTP prompt. Since the Perspective API is predominantly trained for hate, sexual and bias, it exhibits lower quality on our TTP-Evalset. To cope for the higher text lengths in TTPEvalset, we chunk the texts into a length of 500 characters, and take the maximum score across chunks with 0.4 as the optimal threshold chosen from the dev set. We also run TTP on R1 [DeepSeek-AI et al., 2025], Gemma 2 [Gemma, 2024] and Gemini 2, and as expected the models perform poorly, as the prompt is tuned with GPT 4 Omni.

5 HarmFormer

5.1 Data Creation

We utilized the high-quality TTP, to annotate a dataset comprising 3 million web pages from various sources, including Common Crawl (1 Million), C4 [Dodge et al., 2021] (1 Million) and FineWeb [Penedo et al., 2024] (1 Million). From this corpus, we select 258,000 documents, sampling based on both the source-Common Crawl, C4, and FineWeb-and the associated harms and topics. To ensure a balanced representation across all classes, we undersampled the majority of safe categorized web pages from the original 3 million prompt candidates. The labeled data was then stratified sampled and partitioned into three subsets for training and testing the model: training, development, and test sets, with a 90:5:5 split. This gives us a training set of 253,000 samples, and development and test sets of 14,000 each.

5.2 Model Training

We built HarmFormer, leveraging several Transformer [Vaswani et al., 2017] based models - XLM-Roberta [Conneau et al., 2020], Hate-BERT and Longformer [Beltagy et al., 2020]. We employ a Multi-Task architecture with five distinct classification heads representing the harm categories, where each harm category head again categorizes the input into one of three dimensions: Safe, Topical, or Toxic with the web page's text as the input. This design captures the nuanced boundaries between the Topical and Toxic dimensions within each harm category, as well as between the Safe and Topical dimensions, allowing for more precise classification across the task.

Our HarmFormer model achieves an 85test set. The detailed results of our experiments are presented in Table 5. As expected, Hate-BERT, which is primarily trained to detect hate speech, did not perform well on our test set (at a tuned threshold of 0.05). When finetuned to our scenario, HateBERT still underperforms as it needs to relearn new harm categories along with the dimensions. We leverage Long Former's 1024 context length capability over XLM-Roberta, which gave us an F1-Gain of 2 points over the same model trained with 512 context length. We finalize the architecture of the our HarmFormer model to be a LongFormer [Beltagy et al., 2020], trained with 1024 context length across layers, a batch size of 16, learning rate of 2e-5, with AdamW [Loshchilov and Hutter, 2019] as the optimizer for 3 epochs over the train set.

Our HarmFormer model achieves an 85% F1-Score on our test set. The results for this model is presented in Table 6. Upon analysis on the False-Positives (FPs), and the FalseNegatives (FNs), most of the FP cases match up with the "reporting" web-pages as defined in Section 4.2. The FP pattern noise also comes from the prompt which confuses on these reporting cases. This is due to the adversarial nature of our set, as the task would require understanding the intent of the text along with aligning from the guidelines in the taxonomy.

6 Results

6.1 Comparison with Open AI Moderation Dataset

To compare the robustness of TTP and lightweight model against Perspective API, and Llama Guard 3, we measure their performance on the publicly available Open AI Moderation Test set 4 as in [Markov et al., 2023]. This test set contains 1158 non toxic and 522 toxic text sentences varied across categories in Sexual, Hate, Violence and Self-Harm. Since Open AI Moderation set's categories are a subset of out toxic dimension, we club these together into a single label and report the quality in Table 7.

We observe our prompt achieves a Precision and Recall of 0.72 and 0.91 respectively. We test Llama Guard 3 [Inan et al., 2023] in three different settings, the focused prompt setting which includes the categories the model was originally trained on, the Zero Shot setting, where we prompt the detailed description of the categories (provided by the dataset) to the language model, and Few-Shot, where we also append 2 examples per category along with the instructions. The HarmFormer and TTP still achieves a better performance compared to Perspective API and Llama-Guard 3, which proves their superiority to existing solutions across text lengths.

Table 7 shows that TTP and HarmFormer are outperforming the Perspective and Llama Guard on Open AI Moderation test set as well. This proves that they are better across long and short form texts.

6.2 Prevalence of Toxic &Topical Texts in Common Crawl Sets

To assess the scale of harmful content present in pretraining datasets, we applied our TTP prompt to analyze 3 million web pages (described in Section 5.1) sampled from Common Crawl (CC), C4, and FineWeb. This analysis highlights the distribution of toxic and topical content in these datasets across our taxonomy of harms and emphasizes the limitations of existing LLM pretraining dataset curation practices. The detailed breakdown of our findings is presented in Table 8 and the key findings from the analysis are explained below.

Toxic Content:

• Common Crawl contains the highest proportion of toxic content (4.1%), driven by sexual content (2.2%), reflect-

4 https://github.com/openai/moderation-api-release

• ing its unfiltered nature. Other categories, including ideological harm (0.24%) and illegal activities (0.82%), are also prominent.
• C4 (2.11%) and FineWeb (3.9%) show lower toxicity levels, as they filter out data with hate and sexual keywords 5 . However, the Toxic label percentages in these sets remain notable (0.45%& 0.29%).This indicates the limitations in traditional keyword-based approaches.

Topical Content:

• Topical content is better represented in C4 (6.81%) and FineWeb (10.63%) than in Common Crawl (3.44%). This includes discussions on sensitive topics such as societal bias and mental health, which are crucial for LLMs to understand nuanced contexts without toxicity.

Category-Specific Trends in Toxic Content:

• Sexual Harm: Common Crawl is dominated by explicit pornography (78% in overall toxic content), while C4 and FineWeb exhibit subtler (52%) harmful patterns such as sexualizing people through racy descriptions
• Ideological Harm: Toxic narratives, including misinformation and bias, account for 46% of this category's toxic content across datasets.
• Self-Inflicted Harm: Toxic content related to extreme dieting and suicidal ideation represents 91% of labelled cases, highlighting the sensitivity of this harm category.
• Illegal Activities: Topics like hacking (33%) and smuggling (47%) are prevalent across datasets.

To identify the nature of web pages and analyze the distribution of content types, we used domain classifier 6 by [Parmar et al., 2024] and presented our findings below.

• From Figure 2, we find that the percentages of news, health, law, people and society, and sensitive subjects are more prevalent in FineWeb and C4 than in Common Crawl. This leads to higher topical percentages in all harm categories except sexual in C4 and FineWeb compared to Common Crawl.
• Common Crawl contains a relatively higher proportion of shopping sites, which would be filtered out in C4 and FineWeb due to their quality filters. This also explains the relatively lower percentage of news and other domains in Common Crawl.

This analysis reveals that current pretraining dataset curation practices fail to adequately distinguish harmful intent from socially critical or educational discourse. While filtering in C4 and FineWeb retains more topical content, keywordbased approaches often remove valuable context resulting in gaps that risk propagating biases and toxicity in LLM training.

The key findings also emphasize the need for taxonomydriven filtering to balance the removal of harmful content with the preservation of topical material.

5 https://github.com/LDNOOBW/List-of-Dirty-NaughtyObscene-and-Otherwise-Bad-Words

6 https://huggingface.co/nvidia/domain-classifier/

Our HarmFormer Model is designed and trained to address exactly this problem, providing a nuanced approach to filtering that maintains the integrity of valuable contextual information while mitigating the risks of harmful content.

6.3 HAVOC: Harmful Abstractions and Violations in Open Completions Benchmark

To measure the toxicity generated by language models across input severity and harm categories, by creating a new benchmark of open ended prefixes. We sampled 10,376 snippets from web pages across the dimensions and harms identified by the TTP on the previously explained 3 Million candidate set in Section 5.1. A snippet consists of a group of sentences extracted from a web page using a prompt designed to capture content related to a known corresponding harm. We employed an additional prompt to determine the optimal break point between the snippet's prefix and suffix, maximizing the risk of generating toxic content. Given that many of the web pages involve multiple harms, this multiharm characteristic is also reflected in the snippets. Due to computational constraints, we restrict our benchmark size to 10,376. We call this newly created benchmark - Harmful Abstractions and Violations in Open Completions (HAVOC). In contrast to Real-Toxicity-Prompts (RTP)[Gehman et al., 2020], HAVOC exhibits superiority with respect to the harm coverage (described in Figure 4) and the measurement quality (As shown in table 4).

We define a leak as a system-generated response that receives a Toxic label in any harm according to TTP. In the HAVOC benchmark, we apply the prompt to label both the input prefix and the combined prefix plus language modelgenerated suffix. A generation is classified as Provocative Leak if the prefix is labeled with Toxic for any harm and the combined prefix and generation also receive an Toxic label. Conversely, Passive Leak occurs when the prefix is labeled as Topical, but the suffix is labeled as Toxic. Similarly, Neutral Leak is measured when the complete generation is labeled as Toxic, while the prefix remains labeled as Safe. In Table 9, we present some examples of our HAVOC benchmark. We hope HAVOC presents an alternative to quantify toxicity in language models.

We leverage Gemma 2 [Gemma, 2024], Llama 3.2 [Meta, 2024], and Mistral V0.3 [Mistral, 2024] family of models to study their open ended generation on HAVOC and RTP. We use the base pretrained models for our evaluations. Due to hardware constraints, we leverage 2B, 9B & 27B model variants from Gemma 2, 1B & 3B variants from Llama 3.2, and use the 7B variant from the Mistral V0.3 family. We use Ollama [Ollama, 2024] to generate the completions for all the Language Models on our benchmark. We use greedy decoding to generate 200 output tokens from each prefix.

We aggregate the generations across the dataset and present our findings in Figure 3. As anticipated, Provocative Leak exhibits the highest percentage, driven by the presence of toxic content in the prefix prompt. The leakage percentages remain relatively consistent across different models, suggesting that there is potential for improving model training using non-toxic data. In table 10, we provide the model-averaged absolute leakage percentages for all six models. We report an overall leakage rate of 26.7% on the HAVOC benchmark. When challenged with a provocative input, an LLM is expected to leak 76% of the time. To have a holistic perspective at the harm level, we averaged the leakage percentage over all six models, and presented it in Figure 5. This figure highlights Sexual being the most problematic harm. Hate and Ideological-Harm appear to be relatively less problematic. As shown in Figure 3, with the same settings, HAVOC showcases double the leakage rates compared to RTP in Passive category, proving the effectiveness of the Topical Dimension in our taxonomy.

7 Conclusion

Our findings reveal widespread harmful content in webcrawled datasets like Common Crawl, C4 and FineWeb. We also provide the major toxic topics in the harms we define, along with their proportions. Through a combination of manual annotation, prompt-based labeling, and transformer models, we demonstrate that our semi-automated approaches can scale to handle the challenge of inappropriate content detection across all harms. We provide a publicly available evalset, prompt and a transformer based model for toxic text identification in web pages. We analyze and quantify issues in open text datasets like Common Crawl, C4 and FineWeb by covering across various topics, harms, and dimensions. Our multi harm open ended toxic generation benchmark, HAVOC quantifies the toxicity in language model's outputs when challenged with sensitive and provocative inputs. We also find sexual as the most leaking harm for a language model when provocated. Having highlighted the problem in the large scale text datasets, and pre-trained language models, we offer a valuable resource for developing content filtering methods and a new superior open ended text generation benchmark that quantifies toxicity in language models.

8 Limitations &FutureWork

This study focuses on English data, leaving multilingual analysis for future work. The proposed taxonomy addresses broad harms but does not account for culture-specific sensitivities, such as actions deemed disrespectful in certain cultural contexts. For instance, a webpage encouraging head massages may be offensive in cultures where the head is considered sacred. Additionally, we aim to train a model on dimensional filtered datasets and evaluate the impact of dataset purity on model behavior and performance, comparing these findings with models trained on unclean datasets. This will allow for a deeper understanding of the interplay between the cleanliness of training data and the model's behaviour when faced with seemingly toxic inputs across multiple harms.