DeepMind's Chinchilla paper (Hoffmann et al., 2022) fit a loss surface over model size N (non-embedding parameters) and training tokens D from hundreds of real training runs — the same scaling-law logic behind deciding how big to make GPT-4-class or Gemini Ultra-class frontier models, and how to split a mixture-of-experts model's active parameters against its token budget.
Chinchilla loss surface:
L(N,D) = E + A/N^α + B/D^β
E=1.69, A=406.4, α=0.34, B=410.7, β=0.28
Compute-optimal split for a fixed FLOPs budget C = 6ND:
N_opt(C) ∝ C^(β/(α+β)) D_opt(C) ∝ C^(α/(α+β))
Too much N for your D → under-trained (wasted params).
Too much D for your N → over-trained (wasted tokens).
- Model size (N) — moves the marker along the parameter axis of the loss surface; height = how far below the worst-case loss you are.
- Training tokens (D) — moves the marker along the token axis; the surface saturates on both axes, matching diminishing returns from scale alone.
- Optimal frontier — the ridge line where, for each compute budget, loss is minimized; the purple line traces N_opt(C)/D_opt(C) across budgets.
- Snap to compute-optimal — keeps your current compute C = 6ND fixed and moves N,D onto the frontier at that exact budget, so you can compare "what you have" to "what Chinchilla would pick."