Home▸AI & Machine Learning▸Agent Benchmark Significance Lab

Agent Benchmark Significance Lab

Run two tool-using LLM agents through a seeded task battery and see whether one really is better: a permutation test on the success-rate gap plus a bootstrap confidence interval, rendered as 3D outcome grids and a sampling-distribution histogram.

AI & Machine Learning3DAdvanced60 FPS📱 Mobile-adapted⇄ 2D version
Agent Benchmark Significance Lab ↗ Open standalone

Two tool-using LLM agents each run a seeded battery of pass/fail tasks, and the question this simulator answers is the one every real evaluation harness has to answer: is the difference in their success rates real, or could it just as easily be noise? It renders both agents' task outcomes as a 3D grid of pass/fail blocks, runs a permutation test that reshuffles which trials "belong" to which agent thousands of times to build an empirical null distribution, and a bootstrap resample to produce a 95% confidence interval on the success-rate gap — the same resampling methodology used to validate benchmark claims before trusting them.

⚙ Under the hood

Run two tool-using LLM agents through a seeded task battery and test whether the gap in their success rates is real, using a permutation test and bootstrap confidence interval rendered in 3D.

ai agentsllm evaluationstatisticspermutation testbootstrapbenchmarking

3D · Three.js / WebGL renderer · 60 FPS target · runs fully client-side, no install

What did you find?

Add reproduction steps (optional)