Presets
Generate
Sampling and model
Quality
Model checks
The science
Chain rule. Any joint distribution over a sequence factorises exactly, with no approximation:
p(x1,...,xn) = ∏t=1..n p(xt | x1,...,xt-1)
An autoregressive model only has to learn the next-token conditional. Taking logarithms turns the product into a sum, which is what the readout shows: log p(x) = ∑ ln p(xt | x<t). The bars in the first graph are the individual terms; their running total is the negative log-likelihood. Perplexity is exp(-log p(x) / n), the effective number of equally likely choices per token (a uniform guess over V words gives exactly V).
Causal mask. A decoder-only transformer computes attention scores for every pair of positions and then sets the entries with key position s > query position t to -infinity before the softmax. The weight matrix is therefore lower-triangular (the lit pattern on the wall / matrix): position t can look at itself and the past, never the future, so every position can be trained to predict its successor in parallel. Row t of the matrix is the context used to predict token t+1.
The model used here is a toy, not a transformer. It is trained in closed form by counting: for each lag j = 1..K and previous word a, Lj[a][b] = ln((Cj[a][b] + 0.1) / (∑bCj[a][b] + 0.1 V)) from a 32-sentence family-friendly corpus (36 tokens including the end token). The attention weights over the K visible positions are wj = softmax(-λ (j-1)) (a fixed recency bias, not learned), the logits are zb = ∑j wj Lj[xt-j][b], and p(b | context) = softmax(z)b. With K = 1 this is exactly an add-0.1 bigram model. Real transformers learn their attention scores and values by gradient descent on billions of tokens.
Sampling. Temperature rescales the logits, pT(b) = exp(zb/T) / ∑ exp(zc/T): T < 1 sharpens, T > 1 flattens (entropy rises with T, third graph). Top-k keeps the k likeliest tokens; top-p (nucleus) keeps the smallest set of top tokens whose renormalised mass reaches p; the survivors are renormalised and one is drawn with a seeded random number. Greedy decoding is top-k = 1. The log-likelihood always uses the unfiltered T = 1 model probabilities, not the sampling distribution.
Honest simplifications. The corpus is tiny, so the trained model scores a low perplexity only on its own text (it has memorised it; this is in-sample, not generalisation). The "attention" has no learned queries, keys or values. Probabilities, sampling frequencies and the matrix are exact for this model, but the bar heights in the 3D page are linear in p, so rare tokens look tiny, as they should.
Checks. (1) chain rule against an independently enumerated joint of 64 sequences, (2) the model joint over all 46,656 three-token sequences sums to 1, (3) 20,000 samples match exact probabilities within 4 sigma, (4) the mask is strictly lower-triangular and changing a future token leaves earlier predictions bit-identical, (5) softmax, temperature and nucleus properties, (6) perplexity of a uniform model equals V and K = 1 equals the bigram formula.