Skip to main content
ZICQ

Wiki Concepts

Scaling Laws

Concepts
Aliases: scaling law Chinchilla compute-optimal training scaling laws ·2026-10-10

Scaling Laws

Scaling laws describe empirical power-law relationships governing how test cross-entropy loss evolves as a function of model parameters $N$, training tokens $D$, and aggregate compute budget $C$ in deep autoregressive language models. These formulations transformed multi-million-dollar pre-training runs from black-box exploration into rigorous, predictable compute allocation engineering.


Formulations: Kaplan vs. Chinchilla

For standard autoregressive Transformer decoders, the floating-point operations required for forward and backward passes per token are approximated by:

$$C \approx 6 N D \quad (\text{FLOPs})$$

Historically, the optimal distribution of a fixed compute budget $C$ between model size and dataset size underwent a critical empirical revision:

                      Compute Budget C Scales
                                 │
           ┌─────────────────────┴─────────────────────┐
  Kaplan Scaling Laws (2020)                   Chinchilla Scaling Laws (2022)
           │                                                   │
  Allocate ~73% to parameters (N)              Scale N and D in equal proportion
  Allocate ~27% to dataset tokens (D)          Optimal Rule: ~20 tokens per parameter
           │                                                   │
[Led to under-trained GPT-3 / Gopher]         [Chinchilla 70B outperformed Gopher 280B]

1. Kaplan Scaling Laws (OpenAI, 2020)

Kaplan and colleagues fitted isolated power-law trends across sub-scale experiments:

$$L(N) = \left(\frac{N_c}{N}\right)^{\alpha_N}, \quad L(D) = \left(\frac{D_c}{D}\right)^{\alpha_D}, \quad L(C) = \left(\frac{C_c}{C}\right)^{\alpha_C}$$

Under a constrained compute envelope $C$, they deduced that parameter growth should outpace token growth: for every doubling of compute, allocate $\sim 73%$ of the expansion to parameter count $N$ and only $\sim 27%$ to data volume $D$.

This hypothesis governed the initial era of frontier LLMs, leaving models such as GPT-3 (175B parameters trained on only 300B tokens) severely under-trained relative to available parameters.

2. Hoffmann / Chinchilla Scaling Laws (DeepMind, 2022)

DeepMind identified a methodological issue in Kaplan's experimental setup: fixed cosine learning rate schedules across runs caused smaller models evaluated at large token counts to use sub-optimally decayed learning rates, severely underestimating the marginal loss reduction of additional tokens.

Hoffmann and colleagues formulated a unified objective with irreducible loss:

$$L(N, D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta}$$

Where:

  • $E$ represents the irreducible language entropy barrier;
  • $A, B$ are scale constants, with exponents $\alpha \approx 0.34, \beta \approx 0.28$.

Solving the constrained optimization $\min_{N, D} L(N, D) \quad \text{s.t.} \ 6ND = C$ via Lagrange multipliers yields the compute-optimal regime:

$$N_{\text{opt}} \propto C^{a}, \quad D_{\text{opt}} \propto C^{b} \quad (a \approx 0.5, \ b \approx 0.5)$$

Crucially: model size and token volume should scale in equal proportion. At the compute-optimal frontier, each model parameter must be paired with roughly 20 training tokens. Trained under this recipe, Chinchilla (70B parameters, 1.4T tokens) outperformed the 4x larger Gopher (280B parameters).


Industrial Paradigm Shift: Compute-Optimal vs. Over-Training

With the advent of high-performance open-weight models (led by Meta's LLaMA series), production architectures deliberately diverged from the Chinchilla ratio:

Model Architecture Parameter Count ($N$) Training Tokens ($D$) Token-to-Parameter Ratio Architectural Objective
GPT-3 175B 300B $1.7 : 1$ Under-trained (Kaplan Era)
Chinchilla 70B 1,400B (1.4T) $20 : 1$ Compute-Optimal (Chinchilla Era)
LLaMA-1 65B 1,400B (1.4T) $21.5 : 1$ Near-optimal threshold
Llama-3-8B 8B 15,000B (15T) $1875 : 1$ Inference-Optimal Over-training

Production Economics of Over-Training

The Chinchilla formulation presumes a model is trained once and evaluated without regard to downstream inference economics. In commercial deployments serving billions of user queries daily, cumulative inference serving costs outstrip initial training costs by orders of magnitude.

By aggressively "over-training" a smaller model (such as training an 8B model over 15T tokens), pre-training compute experiences diminishing loss returns, but downstream inference secures radical memory savings, higher serving concurrency, and lower dollar costs per generated token.


The Next Frontier: Inference-Time Compute Scaling

As the industry approaches the "data wall" (the depletion of high-quality public human web text), scaling research is expanding from pure pre-training compute to inference-time (test-time) compute scaling:

               ┌────────── Dual Scaling Frontiers ──────────┐
               │                                            │
[Pre-training Compute Scaling]             [Test-time Compute Scaling]
Scales parameters N and corpus tokens D    Scales reasoning tokens and search width
Bounded by FLOPs cost and data depletion   Amplifies problem solving via search and RLVR
               │                                            │
               └────────────────────────────────────────────┘

Pioneered by models like OpenAI o1 and DeepSeek-R1:

  1. Extended Chain-of-Thought (CoT): Allocating dynamic token budgets to hidden reasoning steps for math, code, and logic problems produces steep accuracy gains as a function of test-time compute.
  2. Search and Verification: Pairing Monte Carlo Tree Search (MCTS) or Process Supervision Models (PRM) with parallel rollouts enables real-time backtracking and self-correction, establishing a second growth curve for frontier reasoning capabilities.

Engineering Boundaries and Fallacies

  1. Loss Reductions vs. Downstream Benchmarks: Scaling laws forecast smooth convergence of autoregressive next-token cross-entropy loss, not discrete task benchmark scores (e.g., GSM8K, HumanEval). Metric non-linearities create sudden benchmark step changes that mimic emergent abilities.
  2. Data Quality Shifts Curve Intercepts: The scale factors $A$ and $B$ are highly sensitive to token distribution purity. Infusing high-density synthetic data, curated textbooks, and code lowers the loss curve, achieving target loss thresholds at lower token volumes.
  3. Architectural Biases: While power laws hold reliably for dense Transformer decoders, architectural innovations (e.g., Multi-head Latent Attention, State Space Models like Mamba, sparse MoE routing) alter the parameter-to-FLOP translation efficiency.

Sources and Key Literature