JSON mode works by intersecting the model's token probabilities with a formal grammar at every step, so only tokens that keep the output on a valid parse path can ever be sampled.
accept(token) iff token in grammar(FSM_state)
P(valid_json) -> 1 as strictness -> max
- Token candidates — raw next-token proposals from the model's logits before grammar masking.
- Schema slots — required keys/types in the target JSON schema (id, name, array, nested object...).
- Grammar strictness — how aggressively invalid tokens are masked out before sampling — a finite-state grammar constraint.
- Fill completion rate — how quickly each schema slot receives a syntactically valid value.
Constrained decoding (grammar-masked sampling) is what lets an LLM guarantee syntactically valid JSON every time, instead of just being asked nicely to output JSON.