Model releases dominate AI headlines, but the trend actually driving progress is quieter and more predictable: scaling laws. Train models across a range of sizes and data amounts, and loss falls in a smooth, forecastable curve as a function of compute -- predictable enough that researchers can estimate a target model's eventual performance before spending the compute to train it at full scale.
The compute-optimal insight
For a fixed training-compute budget, there is a specific split between model size (parameters) and data size (training tokens) that minimises loss. DeepMind's Chinchilla research found this split sits around 20 training tokens per parameter for the model families studied -- and found that many earlier large models were meaningfully under-trained relative to their size, spending too much of their compute budget on parameters and too little on data.
💡 Key idea: a smaller model trained on proportionally more data can outperform a larger model trained on less data, for the exact same total training compute.
Why this matters for cost
Training compute (FLOPs) scales roughly as parameters times tokens; a common first-order estimate is 6 × parameters × tokens. Because practice scales both together to stay near the optimal ratio, total compute -- and dollar cost -- grows closer to the square of model size than linearly with it, which is the underlying reason frontier-scale training runs cost dramatically more than models even a few times smaller.
What scaling laws don't tell you
- They estimate training loss, not downstream task performance, product usefulness, or safety -- a lower loss number does not automatically translate into a better product.
- They say nothing about inference cost, which is paid per query rather than once, and which a larger model makes proportionally more expensive at serving time.
- They are fit to specific model families and training recipes -- architecture changes, better data quality, or new training techniques can shift the curve, not just move a point along it.
The practical takeaway
Before committing a training budget, the tokens-per-parameter ratio deserves as much attention as the raw parameter count -- an under-trained large model and a well-trained smaller model can cost the same to build and land in very different places on the loss curve. Compute-optimal is a starting point for planning a training run, not a guarantee of the best model for a given product need.
🧪 Try it yourself: the AI Scaling Trends Lab simulation lets you experiment with everything described above directly in your browser.