Oasis: Data Curation and Assessment System for Pretraining of Large Language Models

1 Introduction

Building large language models (LLMs) for proficiency in versatility tasks has been spotlighted recently [OpenAI, 2023; Touvron et al., 2023; Anil et al., 2023]. The power of LLMs only emerges when their parameter size exceeds a certain threshold [Wei et al., 2022], propelling the models to evolve in parameter scale. Recent studies [Kaplan et al., 2020; Rae et al., 2021; Rosset, 2020] have demonstrated that larger models crave a massive, high-quality, and diverse pretraining corpus. The importance of data curation and assessment is increasingly evident.

∗Corresponding author

1 Project: https://github.com/tongzhou21/Oasis

2 Video: https://youtu.be/YLfMlnrUZPk

3 Corpus: https://huggingface.co/datasets/Oasis-Team/OasisCorpus

Data Curation: Some work details preprocessing pipelines for specific sources like Common Crawl [Wenzek et al., 2019; Abadji et al., 2022; Penedo et al., 2023] or Reddit [Gao et al., 2020]. However, these pipelines cannot be directly applied elsewhere because different curation pipelines should be built for various data sources by native speakers of target languages to ensure better quality control [Laurenc¸on et al., 2022]. Unfortunately, an open-source system for customized pretraining data curation is still absent in the community.

Data Assessment: The assessment of the pretraining corpus [Kreutzer et al., 2022; Dodge et al., 2021] aids in the development of LLMs in a data-centric fashion [Fries et al., 2022] more efficiently. It avoids optimizing the data curation by comparing the final model's performance after resourceconsuming training. Although there is no conclusion on quantifying the corpus's value, the consensus is that various aspects of pretraining data affect LLM performance, such as fluency, coherence, diversity, and bias [Longpre et al., 2023; Gunasekar et al., 2023]. However, there is still a lack of a holistic data assessment system for the progressive improvement of the data curation pipeline.

In this paper, we present a system for customized pretraining data curation and holistic corpus assessment called Oasis. The functionality of this system covers three types of filters used to curate high-quality corpora and two perspectives for the holistic assessment of these corpora.

Specifically, in the Customized Data Curation part, the first step in our pipeline is an Interactive Modular Rule Filter module, which enables users to construct the customized heuristic rule set with hit rate and bad cases as a reference. Then, we debias the neural filter for text quality estimation by paying attention to the process of constructing source-specific quality classification datasets for training, constituting a Debiased Neural Filter module. Finally, in the Adaptive Document Deduplication module, we optimize the widely used LSH deduplication method in memory requirement and exhibit the effect of different configurations for customized settings. In the Holistic Data Assessment part, we provide options to inspect the corpus in sentence fluency and document coherence by humans or GPT-4 in the Local Quality Evaluation module. The evaluated cases with quality labels could be further used to evolve the filtering pipeline. Additionally, the Global Distribution Assessment module displays the distribution information of the corpus in terms of diversity and richness by multiple heuristic metrics.

Aside from introducing Oasis, we demonstrate a complete case that utilizes this platform to build a high-quality and highdiversity Common Crawl corpus. Meanwhile, we holistic assess the corpus in the different development stages. The assessments also prove the effectiveness of the customized data curation process. In addition, we publicly release an 800GB English-Chinese bilingual corpus Oasis-Corpus cultivated from web pages by Oasis to promote LLM development.

2 SystemDesign and Algorithms

In this section, we will introduce the system design of Oasis and detail the internal algorithms that differ from previous paradigms.

2.1 Customized Data Curation

Interactive Modular Rule Filter. Building a rule filter for the pretraining corpus is a routine in state-of-the-art LLMs. A heuristic rule filter could preliminarily filter undesirable content efficiently. The heuristic ideas for building rules range from text length, punctuation, special tokens, blocklist, and language model perplexity. However, no rule sets can always be valid on various data sources and languages. Corpora from different sources could vary in quality, style, format, template, and meta information. Filter rules in the book field may emphasize removing structural information among high-quality content. On the contrary, when handling documents from the massive web, rules would pay more attention to inspecting the content quality. The essential processes in building and improving the rules involve manually concluding patterns to distinguish high- and low-quality texts and adjusting a single heuristic by examining the hit samples.

We design functions in the Interactive Modular Rule Filter module according to the above intuitions. A user builds a rule pipeline by interactively editing and connecting rule cells, referring to the patterns heuristic summarized from randomly displayed samples. A rule cell could be initiated with the predefined heuristic, and the user could also customize a heuristic function and add it to the predefined pool by typing Python code. Each rule cell's configuration, like thresholds and string patterns, can be freely adjusted according to the inspection of the hit rate and bad cases. After building a customized rule filter pipeline, Oasis can automatically generate a corresponding script according to settings and run the rule filter in the background.

DebiasedModel Filter. The original intention of the neural filter is to select high-quality content from massive web pages, similar to high-quality sources like Wikipedia. The model can filter out content with non-summarizable patterns in quality aspects. However, treating another well-known high-quality source as positive and current sources as negative samples could lead the model to bias toward the high-quality source, affecting the quantity and diversity of the filtered data. [Penedo et al., 2023] even abandoned this process due to scruples about the adverse effects of undesirable biases.

To address the bias issue, we propose a negative-centric dataset-building method for neural filter training. This method gathers the majority of positive samples from rule-filtered texts in the current source and obtains most negative samples through heuristic contamination of positive samples. The predefined text contamination rule focuses on coherence and readability, involving shuffling, replacing, inserting, and deleting at the word, span, and sentence levels. The perplexities from the statistical language model may detect these undesirable low-quality contents. However, the perplexity metric is susceptible to low-frequency special tokens and biased towards the training corpus (usually Wikipedia). We use perplexity solely to identify extremely low-quality content, which constitutes a part of the negative samples. These quality patterns are modeled using a neural filter with strong generalization capabilities, such as BERT. The finetuned BERT predicts scores for the text quality of every rule-filtered document. We then drop documents according to the quality score below the threshold.

The Debiased Model Filter module provides a management panel for the quality classification dataset. Users can adjust the composition of positive and negative samples, customize text contamination rules based on editing feedback, and set perplexity quantiles to identify extremely low-quality content through case inspection. Moreover, the dataset for neural classifier training could be further enhanced by incorporating evaluated texts from humans or GPT-4. After building a quality classification dataset, Oasis can generate corresponding scripts through parameter settings on the interface and run in the background with one click for neural filter training and the running process.

Adaptive Document Deduplication. Repetitive documents in the pretraining corpus would harm the LLM's generalization ability in various downstream tasks. Massive deduplication among documents has a theoretical time complexity of O(n2 ). The Locally Sensitive Hash algorithm approximates document similarity and reduces the time complexity, but it comes at the cost of increasing memory requirements to store hash collisions. Large-scale fuzzy deduplication becomes infeasible with limited resources.

To achieve this goal, we reduce the memory requirement of the LSH deduplication algorithm to adapt to customized hardware by adjusting r in the conditional probability formula. The system predicts the maximum r according to the user's configuration in corpus size and memory size. Since a smaller r will lead to a lower collision probability, the system also suggests the running times based on the Jaccard threshold and the expected duplication recall.

Although document-level deduplication could improve the diversity of the cultivated dataset, it could also significantly decrease the quantity. Our Adaptive Document Deduplication module also provides an interface to visualize the duplicated documents in a graph, offering options for users to make tradeoffs between the removal rate and quantity.

Holistic Data Assessment. Evaluating LLMs pre-trained on different curated corpora using downstream tasks' performance serves as an oracle for assessing the data value. This post-hoc method is resource-consuming and ineffective. It is urgent to establish a holistic data assessment system to quantify the data quality and support the optimization process of data curation. We achieve this goal through two views: local quality and global distribution, employing three evaluation methods: human assessment, heuristic metrics, and GPT-4.

Local Quality Evaluation. In this module, we focus on a document's fluency, readability, and coherence as assessed by humans or GPT-4. Due to the high consumption of the human inspection process, we only provide two quality options, High and Low, in the user-friendly human evaluation interface. It displays real-time statistics of manually labeled quality conditions. State-of-the-art LLMs like GPT-4 have demonstrated sufficient ability to score a document in multiple aspects, reflecting overall quality [Chen et al., 2023]. We provide predefined prompts for quality assessment, achieving more than 95% consistency with human opinions. The system also supports customized prompts for diverse demands. Moreover, the local quality evaluation samples can be incorporated into quality classification datasets to evolve the neural filter.

Global Distribution Assessment. Apart from the local document perspective, the global view of the corpus in statistical distribution can also reflect the broadly defined quality.

Oasis adopts six metrics to assess the corpus in heuristics from a randomly sampled subset of data: (1) Lexical Diversity Distribution [McCarthy and Jarvis, 2010]: We calculate each document's Measure of Textual Lexical Diversity (MTLD) score to reflect lexical diversity and plot the frequency histogram to obtain an overall perspective. (2) Task2Vec Diversity Coefficient [Lee et al., 2023]: The task2vec diversity coefficient is proven to have a high correlation with humans' intuitive diversity of the corpus. We sample batches of text and display the calculated overall score. (3) Semantic Diversity Distribution: We obtain all sampled documents' global semantic vectors using BERT and calculate the cosine similarity of each pair of documents to plot the frequency histogram. (4) Topic Diversity Distribution: We cluster the sampled documents by global vector and calculate the similarity of centroid vectors among clusters to reflect overall topic diversity. (5) Knowledge Density and Diversity: We inspect the knowledge view of the corpus by counting the different entities that occur. The density means the entities count normalized by word count, and diversity means the semantic similarity of all emerged entities. (6) Similarity to Wikipedia Distribution: [Jansen et al., 2022] shows that the Kenlmmodel's perplexity on the target source could reflect the approximation of the Kenlm model's training source. We train a Kenlm model on Wikipedia and plot the perplexity distribution to inspect the extent of corpus bias in Wikipedia.

These metrics can be displayed on a single page and overlay multiple corpora for convenient visual comparison.

2.2 Comparative Analysis

As shown in Table 1, the human-evaluated quality of the Chinese portion in the Oasis Corpus constructed by the Oasis system surpasses that of Wudao. Additionally, it exhibits a larger scale and greater knowledge diversity, demonstrating the advantage of Oasis, a comprehensive construction and evaluation system, over traditional data construction pipelines in pretraining data construction.

Compared to the corpora obtained by traditional positivecentric neural filters, the debias neural filter can produce comparable quality in human evaluation and a larger quantity. The perplexities in the Wikipedia source also indicate that our neural filter could alleviate the bias toward high-quality sources in the corpus, ensuring diversity.

3 Conclusion

We propose Oasis, a one-stop system for LLM's pretraining data curation and assessment. In customized data curation, users can tailor their pipeline according to specific corpus requirements and limited hardware resources in rule filter, neural filter, and document deduplication. In holistic data assessment, a corpus can be evaluated from two perspectives: local document and global distribution; and in three ways: human assessment, GPT-4 evaluation, and heuristic metrics. These two components collaborate to enhance the value of the LLM's pretraining corpus. The comparative analysis of the constructed corpora demonstrates the effectiveness of Oasis.

Acknowledgments

This work is supported by the National Key Research and Development Program of China (No. 2022ZD0160503), the National Natural Science Foundation of China (No. 62176257). This work is also supported by the Youth Innovation Promotion Association CAS.