Grammar-Constrained Decoding Visualizer (2D)

Decoding Mode

Constrained decodingMask invalid tokens before sampling

Sampling temperature T 0.80

Auto-play speed 2.0/s

Grammar State

S0 — expecting {

(no tokens generated yet)

Valid tokens here

1 / 13

Tokens this run

Runs (ok / broken)

0 / 0

Valid-JSON rate

—

Next-token distribution

How it works

A language model scores every vocabulary token with a logit. Turning those logits into the next token normally uses temperature-scaled softmax:

p_i = exp(logit_i / T) / Σ_j exp(logit_j / T)

Grammar-constrained decoding (used by tools like Outlines, llama.cpp GBNF grammars and OpenAI/Anthropic structured-output modes) tracks a finite-state automaton over the target grammar — here, a 2-field JSON object {"name": …, "age": …}. At every step it computes the set of tokens the automaton currently accepts, and masks everything else to −∞ before the softmax:

logit_i' = logit_i          if i ∈ allowed(state)
logit_i' = −∞               otherwise
p_i = softmax(logit')_i

Because masked tokens get exactly zero probability, the sampled token is guaranteed to keep the automaton in a valid state — the output can never break, no matter how high the temperature is. Switch Constrained decoding off to sample straight from the unmasked distribution instead: sooner or later a high-probability but grammatically illegal token gets picked (e.g. sampling true where a : was required), and the run is marked broken — exactly what unconstrained JSON-mode prompting risks in production.

Top bar chart — live softmax probability of every vocabulary token at the current automaton state; green dots/bars mark grammar-valid tokens, the glowing bar is the one just sampled.
Bottom graph — the automaton's ten states; the glowing marker is the current state and travels along the accepted edge each step. Drag to pan, scroll or use the Zoom buttons to zoom.
Temperature — higher T flattens the distribution (more exploration); it changes *which* valid token gets picked, never *whether* the grammar holds when constrained decoding is on.

Grammar-valid token Sampled token Invalid pick (broken)

drag to pan · scroll to zoom