Multiple Comparisons: p-Hacking vs Bonferroni

Correction Mode

Tests per family (n) 20

This Family

Tests run

0 / 20

Flagged "significant"

Threshold per test

0.0500

True effect present?

None — pure noise

Across All Families Run

Families run

Families w/ ≥1 false pos.

Empirical false-positive rate

–

Theoretical target

–

Expected vs Actual FP Rate

Uncorrected (theory) Bonferroni (theory) Uncorrected (actual) Bonferroni (actual)

Why this happens

Every test in this simulator runs on freshly generated random noise with no real underlying effect — the null hypothesis is always literally true. A single test at α = 0.05 still has a 5% chance of a false positive purely by chance.

Test n independent hypotheses on the same data and the chance that at least one comes back "significant" by pure luck is:

P(≥1 false positive) = 1 − (1 − α)ⁿ

n = 1  → 5%
n = 20 → 1 − 0.95²⁰ ≈ 64%
n = 60 → 1 − 0.95⁶⁰ ≈ 95%

The Bonferroni correction divides α by the number of tests, so each individual test must clear a far stricter bar: α/n instead of α. That keeps the family-wise chance of any false positive back down near the original 5%, no matter how many tests you run.

threshold(per test) = α / n
P(≥1 false positive) ≈ 1 − (1 − α/n)ⁿ ≈ α

Run enough families at a given n and the empirical "families with ≥1 false positive" rate converges onto these curves — dramatically climbing when uncorrected, pinned near 5% when Bonferroni-corrected, on the exact same random data.

UNCORRECTED · flags noise as "significant" flag if p < 0.0500

Each tile = one independent hypothesis test on random noise. Red = flagged "significant" by chance. last p-value: –

⚠ Something went wrongThe simulation failed to initialize. Reload the page.