At every generation step a language model outputs one logit per vocabulary token, turned into a probability with a temperature-scaled softmax:
P(t) = exp(logit(t) / T) / Σ exp(logit(t') / T)
Grammar-constrained decoding (used by JSON-schema / function-calling "structured output" modes in real LLM APIs) runs a formal grammar as a finite-state acceptor alongside the model. At each step only tokens that keep the output a valid string in the grammar are allowed — every other token's probability is forced to zero and the rest are renormalized:
mask(t) = 1 if t ∈ valid-next-tokens(state), else 0
P'(t) = P(t)·mask(t) / Σ P(t')·mask(t')
Nucleus (top-p) sampling then keeps only the smallest set of highest-probability tokens whose cumulative mass reaches p, discarding an unlikely tail before the final draw.
- Sample next token — draws one token from the current (possibly masked) distribution and advances the grammar's finite-state machine, which here is producing
{"name":…,"age":…,"active":…}.
- Grammar constraint ON/OFF — with it off the model samples from its raw distribution and can emit a token the grammar forbids (e.g. a number where a key was required), producing invalid JSON — exactly the failure mode structured-output decoding exists to prevent.
- Temperature — low T sharpens the distribution toward the top logit; high T flattens it toward uniform, more entropy per bar.
- Top-p — truncates the long tail of implausible tokens before sampling, independent of the grammar mask.
The 20 bars are the full toy vocabulary arranged in a ring; bar height is that token's current probability, color shows whether the grammar currently allows it, and the ring pulses at the bar that gets sampled each step.