Large-model performance follows an empirical scaling law (Kaplan et al. 2020): loss falls as a power law in training compute, with no sign of an asymptote reached yet — model size, data and compute are exchangeable resources.
Loss: L(C) = L0 + (Cc / C)^0.05
Optimal N: N ≈ sqrt(C / (6 · r)) where r = tokens-per-parameter ratio
Chinchilla-optimal: r ≈ 20 (Hoffmann et al. 2022)
Overtrained: r ≈ 100 (small model, more tokens — cheaper inference)
L0 ≈ 1.5 nats — irreducible entropy of natural text
Cc — reference compute constant
- Compute slider — total training FLOPs, log scale from GPT-1-era (10^18) to speculative AGI-scale (10^26). Grows the visualized network: more layers, more neurons per layer.
- Data/parameter ratio — Chinchilla-optimal balances model size against training tokens for the lowest loss at a given compute budget; overtraining shrinks the model but trains it on far more tokens, trading a slightly higher loss for cheaper inference.
- Architecture — Transformer follows the pure scaling curve. Neuro-symbolic adds an explicit "world-model" core (the bright central cluster) — per Yann LeCun's critique, this buys a small loss discount by encoding structure directly rather than learning it purely from scale.
- Glowing markers along the network mark emergent capabilities — abilities (arithmetic, chain-of-thought, code generation, theory of mind, cross-domain transfer) that appear abruptly once compute crosses a threshold, rather than improving gradually.
This is illustrative, not a trained model: the network's shape encodes the scaling relationship, not real weights.