Scaling Laws

One of the most consequential discoveries in modern machine learning is also one of the simplest to state: as you scale up model size, data, and compute, the loss of a language model decreases in a smooth, predictable power law. This regularity is so reliable that you can forecast the performance of a model costing millions of dollars from a handful of small, cheap experiments. Scaling laws turned the question 'how big should the model be?' from guesswork into engineering.

A power law relates loss L to a resource X (parameters, data, or compute) as L ≈ (X₀/X)^α plus an irreducible floor. On a log-log plot, this is a straight line — and the empirical loss of language models falls almost exactly on such lines across many orders of magnitude. This is the central empirical fact of the chapter.

Before relating compute to loss, we need to measure compute. The fundamental unit is the floating-point operation (FLOP), and there is a beautifully simple rule for how many FLOPs it takes to train a Transformer: approximately 6 times the number of parameters, per token of training data.

The factor of 6 decomposes cleanly. Each parameter participates in roughly 2 FLOPs per token in the forward pass (one multiply, one add in the matrix multiplications). The backward pass costs about twice the forward pass — it computes gradients with respect to both inputs and weights — adding 4 FLOPs. Total: 2 (forward) + 4 (backward) = 6 FLOPs per parameter per token.

Kaplan et al. (2020) trained hundreds of models across orders of magnitude in size and data, and fit power laws to the results. Their findings established the quantitative foundation of large-model development and held up remarkably well — with one important correction we cover next.

Kaplan proposed a combined formula capturing how loss depends jointly on model size and data, with the loss limited by whichever is the binding constraint:

Kaplan's compute-allocation conclusion: given a 10× increase in compute, spend most of it on a bigger model and relatively little on more data. This recommendation — 'make the model much bigger, train on modestly more data' — guided the GPT-3 generation. It was reasonable given the data available, but it turned out to be suboptimal, as Chinchilla revealed.

Hoffmann et al. (2022) revisited the compute-allocation question with more careful methodology and reached a landmark conclusion: at a fixed compute budget, model size N and dataset size D should scale roughly in equal proportion. The implication was that the flagship models of the era were drastically under-trained — too big for the amount of data they saw.

This additive form is more interpretable than Kaplan's. There is an irreducible loss E — the entropy of language itself, which no model can beat. Above that, two terms decrease with model size and data size respectively. Minimizing the loss for a fixed compute budget C = 6ND becomes a clean constrained optimization.

Solving the optimization yields the compute-optimal recipe: for every parameter, train on roughly 20 tokens. A compute-optimal 10B-parameter model should see about 200B tokens; a 70B model about 1.4T tokens. This 20:1 token-to-parameter ratio is the single most cited number in modern LLM training.

The core practical use of scaling laws is allocation: given a fixed compute budget C, how do you split it between a bigger model (more N) and more training (more D)? The Chinchilla law turns this into a solvable optimization, and the answer is a specific N and D for every budget.

The key result: both the optimal model size and the optimal dataset size grow as roughly the square root of compute. Each time you get 10× more compute, you should make the model about 3.2× bigger AND train on about 3.2× more data — not pour it all into size as Kaplan suggested.

The most powerful practical application of scaling laws is forecasting. Before committing to a huge, expensive training run, you train a series of small models, fit a scaling law to their losses, and extrapolate to predict the loss of the large model. This de-risks enormous investments and is now standard practice at every frontier lab.

Scaling laws tell us that three things reliably move the loss: parameters, data, and compute. Equally important is what does NOT much affect the loss — because that tells you where not to spend your effort.

The classic scaling laws treat all tokens as equal, but data quality matters enormously. Better-filtered, deduplicated, higher-quality data shifts the entire scaling curve downward — you reach a lower loss at the same compute. The Phi model series demonstrated that carefully curated 'textbook-quality' data can let a small model punch far above its parameter count. Data curation (Chapter 17) is the lever that scaling laws assume away but practitioners obsess over.

Scaling laws predict loss smoothly. Yet some capabilities — multi-step arithmetic, instruction following, chain-of-thought reasoning — appear to switch on suddenly at a certain scale, absent below it and present above. These 'emergent abilities' (Wei et al., 2022) are among the most discussed and contested phenomena in the field.

The skeptical view (Schaeffer et al., 2023) is compelling: many 'emergent' jumps are artifacts of the metric. If you score a multi-step task with exact-match (all steps correct or zero credit), then a model whose per-step accuracy improves smoothly will show a sudden jump in exact-match once per-step accuracy crosses a threshold. Switch to a smooth, partial-credit metric and the emergence often dissolves into a smooth curve.

Scaling laws are power laws, and power laws have a sobering property: improvements shrink as you climb. Each halving of the loss-above-irreducible costs exponentially more compute. And there are hard limits — finite data, finite compute, the irreducible entropy floor — that bound how far pure scaling can go.

The most discussed limit is data. Chinchilla-optimal training of ever-larger models demands ever more high-quality tokens — but the supply of high-quality human text is finite. Estimates (Villalobos et al., 2022) suggest the stock of high-quality public text could be exhausted by training runs in the mid-to-late 2020s. This 'data wall' is driving intense interest in synthetic data, multi-epoch training, and data efficiency.

These limits are reshaping the field's direction. The frontier is shifting from pure scale toward better data (Chapter 17), test-time compute and reasoning (Chapter 24), sparse architectures like Mixture-of-Experts (Chapter 32), and post-training techniques (Part V) that extract more capability from a fixed base model. Scaling laws are not dead, but the era of 'just make it bigger' is giving way to a more multidimensional optimization.

Scaling laws are not just descriptive science — they are a decision-making tool. Here is how a practitioner actually uses them, from a fixed budget to a concrete training plan.

Exercises 1–10 are pen-and-paper or derivations; 11–18 require code.

Further reading: “Scaling Laws for Neural Language Models” (Kaplan et al., 2020) — the founding paper. “Training Compute-Optimal Large Language Models” (Hoffmann et al., 2022) — the Chinchilla paper. “Emergent Abilities of Large Language Models” (Wei et al., 2022) and “Are Emergent Abilities a Mirage?” (Schaeffer et al., 2023) for the emergence debate. “Will we run out of data?” (Villalobos et al., 2022) on the data wall. Sutton's “The Bitter Lesson” (2019) for the philosophical backdrop.

You have built the Transformer from first principles, learned to train it, and learned the scaling laws that tell you how big to build. But everything so far has lived on a single machine, on modest data. Part IV — Pretraining LLMs — confronts the realities of frontier-scale training: where the trillions of tokens come from and how they are curated (Chapter 17), how training is distributed across thousands of GPUs (Chapter 18), the architecture variants that make large models efficient (Chapter 19), the techniques that squeeze more out of every FLOP (Chapter 20), and how progress is measured during a months-long run (Chapter 21). The clean training loop of Chapter 15 becomes a distributed, fault-tolerant, data-hungry industrial process — and you now have the foundation to understand every part of it.