🎲 LLM Next-Token Sampling Lab

How a language model picks its next word: logits → softmax(z/T) → top-k / top-p / min-p → one random draw. Drag the bars, turn the knobs.

1. Logits zi (the model's raw scores)

Drag a bar up or down. Tap a word label to mark it as "already used" (repetition penalty applies to it).

2. Probabilities: before (grey) and after (green) truncation

Sampler settings

Order applied: repetition penalty, then temperature, top-k, top-p, min-p, then renormalise. T = 0 means greedy (argmax).
Numbers (table)

3. Experiments (use the settings above)

4. Toy sentence generator (made-up bigram table, NOT a real language model)

About 30 words with next-word probabilities written by this page purely for illustration. Real LLMs condition on the whole context and have 30,000+ tokens. The sampler settings above are applied at every step.
The science

Softmax with temperature. A model outputs logits zi, one per vocabulary token. Probabilities are pi = exp(zi/T) / Σj exp(zj/T). T = 1 is the raw softmax. T → 0 concentrates all mass on the largest logit (greedy / argmax). T → ∞ flattens towards the uniform distribution 1/V. The T-dependence is a Boltzmann distribution, as in statistical physics (T plays the role of kBT, logits are negative energies).

Top-k keeps only the k most probable tokens (Fan, Lewis and Dauphin, 2018). Top-p / nucleus keeps the smallest set whose cumulative probability is at least p (Holtzman et al., 2020, "The Curious Case of Neural Text Degeneration"). Min-p keeps tokens with pi ≥ m · pmax; it is a newer heuristic (proposed around 2023 to 2024), so treat claims about it being better as still being debated. After truncation the surviving probabilities are renormalised. Real libraries differ in the order of these steps; this lab uses the order shown under the sliders.

Repetition penalty (Keskar et al., 2019, CTRL): for tokens already used, a positive logit is divided by θ and a negative logit is multiplied by θ, so both become less likely.

Entropy and perplexity. H = −Σ pi log2 pi bits (Shannon, 1948). Perplexity = 2H is the "effective number of equally likely choices": 1 for greedy, V for uniform.

Gumbel-max trick. If gi = −ln(−ln Ui) with Ui uniform on (0,1), then argmaxi(zi/T + gi) is distributed exactly as softmax(z/T). So "sampling" is "argmax plus noise", which links greedy decoding and sampling (Gumbel, 1954; Maddison et al., 2014). The experiment tab compares both methods with a two-sample chi-square test.

Beam search is not sampling: it keeps the b highest-scoring partial sentences by summed log-probability. It is deterministic, and for open-ended text it tends to produce bland or repetitive output, which is part of what motivated sampling methods (Holtzman et al., 2020). Sampled text varies from run to run because each step is a random draw, and early differences compound.

Statistics. With N draws, each observed frequency has standard deviation √(p(1−p)/N), so the RMS error shrinks as 1/√N. The chi-square p-value is the chance of a fit at least this poor if the draws really follow the shown probabilities; categories with expected count below 5 are pooled. A p-value below 0.05 will still happen in about 1 run in 20 even for a correct sampler.

Simplifications. The 12-token vocabulary and the logits are hand-made. The toy sentence generator uses a made-up bigram table that only looks at the previous word. Neither is a real language model, and no real model's weights or outputs are used.