Each of N tasks is scored pass/fail. Agent A and Agent B each get their own seeded task battery, drawn as Bernoulli trials from a hidden true success probability pA, pB (the sliders — a stand-in for "how good the agent really is").
observed rate: r = successes / N
observed gap: Δ = r_B − r_A
A single gap could be noise, so the significance test does not assume a formula — it resamples. A permutation test pools all 2N outcomes, reshuffles the A/B labels many times (the "Resample iterations" slider), and recomputes Δ each time. That builds an empirical null distribution — "what gaps would we see if the two agents were identical." The p-value is the fraction of shuffled gaps at least as extreme as the real one:
p = count(|Δ_shuffled| ≥ |Δ_observed|) / iterations
Separately, a bootstrap resamples each agent's own trials with replacement (same iteration count) to build a distribution of Δ itself, whose 2.5th/97.5th percentiles give the 95% confidence interval shown above and the strip at the bottom of the canvas. The middle panel's histogram is the permutation-null distribution of Δ; the gold marker is the real observed gap. Drag the histogram left/right to pan, scroll/pinch to zoom into its tails.
- Trials per agent — battery size N; small N makes even a real gap look noisy.
- True success rate sliders — the hidden ground truth each agent's trials are drawn from.
- Resample iterations — how many shuffles/bootstrap draws build the null and CI; too few makes the p-value itself noisy.
- Significance level α — the threshold the verdict compares the p-value against; lower α demands stronger evidence.
- Run battery — draws fresh Bernoulli outcomes, then runs the permutation test and bootstrap.
- New seed — reshuffles the random generator so repeated runs at the same pA/pB aren't identical.
Real-world relevance: this is exactly how "agent A beats agent B" claims get validated in LLM-agent evaluation harnesses — a raw score difference on a benchmark is meaningless without a resampling test showing it isn't sampling noise.