Home▸AI & Machine Learning▸Agent Benchmark Significance Lab 2D: Permutation Test & Bootstrap CI

Agent Benchmark Significance Lab (2D)

Run two tool-using LLM agents through a seeded task battery and see whether one really is better: a permutation test on the success-rate gap plus a bootstrap confidence interval, rendered as a 2D pass/fail grid, a draggable null-distribution histogram and a bootstrap CI strip.

AI & Machine Learning2DAdvanced60 FPS📱 Mobile-adapted⇄ 3D version

Details

CategoryAI & Machine Learning

RenderingCanvas 2D

DifficultyAdvanced

Target FPS60

ControlsSliders set the task-battery size, each agent's hidden true success rate, the number of permutation/bootstrap resample iterations, and the significance level α; a Run battery button draws fresh pass/fail trials and recomputes the test, New seed reshuffles the random draw, and the null-distribution histogram supports drag-to-pan and scroll-to-zoom (Reset view restores it).

ai agentsllm evaluationstatisticspermutation testbootstrapbenchmarking

Related simulations

Pass@k Benchmark Estimator: Sampling Grid & Score Curve (2D)A 2D evaluation-harness simulator: every attempt is drawn as a cell in a real per-task…

Pass@k Benchmark Estimator: Agent Evaluation Under Stochastic SamplingA 3D evaluation-harness simulator that samples multiple attempts per task, computes the…

Bootstrap Resampling — Confidence IntervalsResample one dataset thousands of times with replacement to build a sampling distribution… ⊞ All 1161 in AI & Machine Learning →

Two tool-using LLM agents each run a seeded battery of pass/fail tasks, and the question this simulator answers is the one every real evaluation harness has to answer: is the difference in their success rates real, or could it just as easily be noise? This 2D dashboard renders both agents' task outcomes as pass/fail grids, runs a permutation test that reshuffles which trials "belong" to which agent thousands of times to build an empirical null distribution (drag to pan, scroll to zoom), and a bootstrap resample to produce a 95% confidence interval on the success-rate gap — the same resampling methodology used to validate benchmark claims before trusting them.

⚙ Under the hood

Run two tool-using LLM agents through a seeded task battery and see whether one really is better: a permutation test on the success-rate gap plus a bootstrap confidence interval, rendered as a 2D pass/fail grid, a draggable/zoomable null-distribution histogram and a bootstrap CI strip.

ai agentsllm evaluationstatisticspermutation testbootstrapbenchmarking

2D · HTML5 Canvas 2D · 60 FPS target · runs fully client-side, no install