Data Curation & Preprocessing

Part III built and trained the Transformer. But a model is only as good as the data it learns from. The architecture has been remarkably stable since 2017; the dramatic capability gains since then have come overwhelmingly from more and better data, plus scale. Yet data curation receives a fraction of the attention that architecture does. This chapter corrects that imbalance: the pipeline that turns the raw web into clean training tokens is the highest-leverage, least-glamorous work in building an LLM.

Chapter 16's Chinchilla recipe demands roughly 20 tokens per parameter, and over-training for inference efficiency pushes this far higher. A modern 70B model trains on 15 trillion tokens — the equivalent of tens of millions of books. There is no curated dataset of that size; it must be assembled and cleaned from the open web, and the cleaning is where most of the value is created.

Turning the raw web into training tokens is a multi-stage pipeline. Each stage discards a large fraction of the input; the survivors of all stages become training data. Here is the canonical flow:

Common Crawl is a non-profit that has crawled the web monthly since 2008, releasing the archives freely. It is the single largest source of pretraining data — nearly every major LLM is built on it. Understanding its formats and quirks is the practical starting point for data curation.

Serious pipelines prefer WARC files and run their own text extraction, because the pre-extracted WET text is noisy — it includes navigation, boilerplate, and poorly-formatted content. High-quality extractors like trafilatura or resiliparse pull the main article text and discard the chrome, which substantially improves the resulting data quality.

The web is multilingual, and most pipelines need to route documents by language — to filter to target languages, to apply language-specific quality rules, and to control the multilingual mixture. Language identification is a fast, early-stage filter that also doubles as a quality signal: text that no language detector can confidently classify is often garbage.

The web is extraordinarily repetitive: the same articles are syndicated across thousands of sites, boilerplate recurs everywhere, and popular text is quoted endlessly. Deduplication — removing exact and near-duplicate documents — is one of the most impactful curation steps. It typically removes 30–70% of the data, and the resulting model is BETTER, not just cheaper to train.

Lee et al. (2022) showed that deduplicating training data improves model quality and reduces memorization, with no downside. It is one of the rare 'free lunch' interventions: less data, less compute, AND a better model.

Fuzzy deduplication needs to find documents that are similar but not identical. The standard approach: represent each document as a set of shingles (overlapping n-grams), estimate the Jaccard similarity between documents using MinHash signatures, and use Locality-Sensitive Hashing (LSH) to find candidate pairs efficiently without comparing all N² pairs.

Most of the web is not useful training data: spam, auto-generated text, keyword-stuffed SEO pages, broken markup, and incoherent fragments. Quality filtering removes this noise. Two complementary approaches dominate: fast heuristic rules and slower model-based classifiers.

Heuristic filters are cheap rules that catch obvious garbage. Gopher (Rae et al., 2021) published an influential set of such rules, now widely adopted. They are fast enough to run on every document and catch a large fraction of low-quality content.

Heuristics catch obvious garbage but cannot judge subtle quality. Model-based filters train a classifier to distinguish 'high-quality' from 'low-quality' text. The classic approach (used by GPT-3 and LLaMA): train a classifier to recognize text resembling a high-quality reference corpus (e.g. Wikipedia, books, or curated web), and keep documents the classifier rates highly.

Beyond quality, some content should be removed for safety and legal reasons: toxic and abusive text, personally identifiable information (PII), and illegal material. This filtering reduces harmful model behaviour and memorization of private data, though it involves genuine trade-offs.

A subtle but critical safety filter is decontamination: removing copies of evaluation benchmarks from the training data. If test questions leak into training, evaluation scores are inflated and meaningless. Pipelines detect benchmark text via n-gram overlap and remove matching documents — but contamination is pervasive and hard to fully eliminate, which is why Chapter 21 treats evaluation contamination as a first-class concern.

Pretraining data is not monolithic — it spans web text, code, books, academic papers, Wikipedia, and more. The data mixture (how much of each domain, and how many times each is repeated) is a powerful lever on the model's capabilities. More code improves reasoning and coding; more books improve long-form coherence; more academic text improves knowledge.

Weights are usually expressed as sampling probabilities: when assembling a training batch, sample each domain with its weight. High-quality but small domains (Wikipedia, books) are often UP-weighted relative to their raw size — meaning the model sees them multiple times (multiple epochs) while seeing abundant web data once. This is where the epoch question becomes central.

Should the model see each token once, or repeat the data? Muennighoff et al. (2023) studied repeating data under the data wall and found that up to ~4 epochs of repetition is nearly as good as fresh data, but returns diminish sharply beyond that. So small high-quality domains can be safely repeated a few times; the abundant web is typically seen once or twice.

Chapter 16 introduced the data wall: the finite supply of high-quality human text, which Chinchilla-optimal scaling threatens to exhaust. One major response is synthetic data — text generated by existing models — which can be produced in unlimited quantities and tailored for quality and diversity.

The Phi model series (Gunasekar et al., 2023, 'Textbooks Are All You Need') demonstrated that a small model trained largely on synthetic 'textbook-quality' data could rival much larger models on reasoning and coding. This showed that data QUALITY and structure can substitute for raw scale — a direct lever against the data wall.

We now assemble the stages into a complete pipeline. In practice this runs as a distributed job (Spark, Ray, or custom MapReduce) over thousands of machines, because the data volume is far too large for a single node. The logical flow, however, is the sequence of filters we have built.

A model inherits everything about its training data: its knowledge, its blind spots, its biases, and its legal exposure. Responsible data curation includes documenting what went in and grappling with the ethical and legal questions that web-scale data raises.

Gebru et al. (2018) proposed datasheets for datasets: structured documentation of a dataset's composition, collection process, intended uses, and known limitations. For pretraining data this includes the sources, the filters applied, the languages covered, and the deduplication and decontamination steps. Documentation enables reproducibility, accountability, and informed downstream use.

Exercises 1–10 are pen-and-paper; 11–18 require code.

Further reading: “The RefinedWeb Dataset for Falcon LLM” (Penedo et al., 2023) and the FineWeb technical report (2024) for state-of-the-art open curation recipes. “The Pile” (Gao et al., 2020) and “Dolma” (Soldaini et al., 2024) for documented open datasets. “Deduplicating Training Data Makes Language Models Better” (Lee et al., 2022). “Documenting Large Webtext Corpora” (Dodge et al., 2021) on C4's filters and their biases. “Scaling Data-Constrained Language Models” (Muennighoff et al., 2023) on repeating data. “Datasheets for Datasets” (Gebru et al., 2018) on documentation.

Next → Chapter 18: Distributed Training

You now have a clean, curated dataset of trillions of tokens — far more than any single GPU could process. Chapter 18 confronts the engineering reality of frontier-scale training: how to spread one model and one dataset across thousands of GPUs. We will build up data parallelism, tensor parallelism, pipeline parallelism, and the ZeRO family of optimizer-sharding techniques, and see how they combine into the 3D-parallel strategies that train the largest models. The clean training loop of Chapter 15 becomes a distributed, fault-tolerant industrial system.