JSON Output Validation and Repair Loop

Schema errors, error feedback and the chance that a retry succeeds.

Switch to 2D

Presets

Playback

Schema and error rates

How big the schema is and how often a fresh reply is wrong.

Repair loop

What the model does with the validator feedback, and how many tries it gets.

Cost of an attempt

Time and tokens; the price is illustrative.

Animation

Experiment

Simulate 20,000 replies against the matrix prediction.

Camera

Quality

Layers

Model checks

The science

One attempt

A reply has n schema fields plus the "is it well-formed JSON" gate. On a fresh reply each field is wrong with probability q and the JSON is broken with probability s, independently, so the first attempt passes with p0 = (1 - s)(1 - q)^n. A schema with 6 fields at 8 % each and 3 % broken JSON passes first time only about 59 % of the time: many small error rates multiply.

Blind retry

If the failed reply is simply discarded and the model is asked again, every attempt is an independent trial with the same p0: P(valid by attempt k) = 1 - (1 - p0)^k, the mean number of attempts is 1/p0, and with a budget of K attempts the mean used is (1 - (1 - p0)^K) / p0. This is a geometric distribution.

Repair with feedback

When the validator returns the error list, the model only has to fix what is wrong. The state is (g, j): g = JSON broken or not, j = number of wrong fields. On each retry a wrong item is fixed with probability r and a correct item is broken again with probability d (regression). From j wrong fields the next count is j - X + Y with X ~ Binomial(j, r) and Y ~ Binomial(n - j, d). The state "everything valid" is absorbing, so P(valid by attempt k) is the absorbed probability after k - 1 steps of the transition matrix, and the mean number of attempts is 1 + pi (I - Q)^-1 1 where Q is the matrix restricted to the invalid states and pi the start distribution. With d = 0 the fields are independent and P(k) = (1 - s(1-r)^(k-1)) (1 - q(1-r)^(k-1))^n exactly.

Budget, latency and cost

With a budget of K attempts the loop stops after min(A, K) attempts, so E[min(A,K)] = sum_{i=0}^{K-1} P(A > i). Latency is that number times the time of one attempt and the token count is that number times the tokens of one attempt; the dollar figure uses the price you set (illustrative, not a real price list). The chance that the reply is still invalid when the budget ends is 1 - P(valid by K): this is the residual failure rate your application must handle (fallback, human review or an error).

Checks shown in the page

(1) the blind-retry closed form against a simulation of 12,000 replies; (2) the matrix chain against the exact closed form when regression is switched off; (3) the matrix mean attempts against the simulation, and the fundamental-matrix mean against the tail sum sum_k P(A > k). The experiment button runs 20,000 replies and compares the measured pass rate with the chain.

Simplifications

Fields fail independently with the same rate (real errors are correlated: one misunderstood field often breaks several), every wrong item has the same repair probability, one number r stands for the quality of the error message, and the syntax gate is treated as one more item. The model ignores that retries make the prompt longer (cost per attempt is constant), that a valid-looking reply can still be semantically wrong, and that real systems may also switch to constrained decoding. All rates, token counts and prices are illustrative; they are not measurements of any real model.

What to try
  1. Watch the default loop: a card comes out of the model, the validator turns red or green, and a rejected card goes round the repair lane. Compare the live tally with the cyan probability curve.
  2. Press the vague feedback preset: the repair rate falls to 25 % and the orange blind-retry curve overtakes the cyan one, because a vague error message is worse than starting over. Then press precise error paths and see the mean attempts drop.
  3. Press the big schema preset and look at the "within the budget" graph: each extra field costs success probability.
  4. Tick blind retry: the replies are regenerated from scratch and the orange curve applies instead of the cyan one.
  5. Press the regression-prone preset and raise the regression rate: fixing one field starts to break another and the mean attempts grow.
  6. Press "Run 20,000-reply experiment" and compare the measured pass rate with the matrix prediction.
Loading Blender model...
Probability of a valid reply by attempt: repair loop, blind retry and simulation
Which attempt first passes (probability)
Mean attempts used against repair rate
Probability of a valid reply within the budget against schema size
Made with 🟠 Blender · Three.js