Every square in the grid is one held-out benchmark question. Each run resamples the whole grid from the current settings, exactly like re-scoring a hidden test set with a new random seed โ that resampling is what "seed variance" below is measuring.
Per-step correctness: p = 1 โ difficulty
Whole reasoning chain: P_chain = p ^ steps
With tool-assisted verifier (catches ~55% of chain errors):
P_verified = P_chain + (1 โ P_chain) ร 0.55
Self-consistency, k samples, majority vote:
P_majority(k) = ฮฃ (i โฅ โk/2โ) C(k,i) ยท P_verified^i ยท (1 โ P_verified)^(kโi)
Eval cost (relative units): cost = steps ร k ร (verifier ? 1.4 : 1)
- Task difficulty โ lowers per-step correctness p; harder hidden tests fail more often per reasoning step.
- CoT steps โ more steps means more chances for a single wrong step to break the whole chain (process-based reward models score exactly these individual steps).
- Tool-assisted verifier โ an external calculator/search/code-interpreter check that catches and fixes a fraction of wrong steps, at extra eval cost.
- Self-consistency samples (k) โ draws k independent reasoning attempts per question and keeps the majority answer; boosts accuracy but multiplies compute cost and is exactly what the "noise / seed variance" FAQ is trading off against eval budget.
The grid re-samples every few seconds to imitate re-running the benchmark with a fresh random seed โ watch how the accuracy estimate wobbles more with small k (few samples) and settles down as k grows, the same self-consistency effect used in real LLM reasoning evals.