Evaluating Reasoning in LLMs: Benchmark & Chain-of-Thought Verification Lab

Benchmark

Task difficulty 40%

Chain-of-thought steps 4

Evaluation setup

Self-consistency samples (k) 1

Benchmark accuracy

Eval cost (relative units)

Seed-to-seed variance

±0.0%

How it works

Every cube in the grid is one held-out benchmark question. Each run resamples the whole grid from the current settings, exactly like re-scoring a hidden test set with a new random seed — that resampling is what "seed variance" below is measuring.

Per-step correctness:      p = 1 − difficulty
Whole reasoning chain:      P_chain = p ^ steps
With tool-assisted verifier (catches ~55% of chain errors):
                             P_verified = P_chain + (1 − P_chain) × 0.55
Self-consistency, k samples, majority vote:
                             P_majority(k) = Σ (i ≥ ⌈k/2⌉) C(k,i) · P_verified^i · (1 − P_verified)^(k−i)
Eval cost (relative units): cost = steps × k × (verifier ? 1.4 : 1)

Task difficulty — lowers per-step correctness p; harder hidden tests fail more often per reasoning step.
CoT steps — more steps means more chances for a single wrong step to break the whole chain (process-based reward models score exactly these individual steps).
Tool-assisted verifier — an external calculator/search/code-interpreter check that catches and fixes a fraction of wrong steps, at extra eval cost.
Self-consistency samples (k) — draws k independent reasoning attempts per question and keeps the majority answer; boosts accuracy but multiplies compute cost and is exactly what the "noise / seed variance" FAQ is trading off against eval budget.

The grid re-samples every few seconds to imitate re-running the benchmark with a fresh random seed — watch how the accuracy estimate wobbles more with small k (few samples) and settles down as k grows, the same self-consistency effect used in real LLM reasoning evals.

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.