🔬 Scientific Method Lab: Hypothesis Test, Power and the Multiple-Comparison Trap

What to do

Question: does a new fertiliser make plants grow taller? The real effect is hidden. Choose sample size and significance level before the experiment, press Run experiment, read the Welch t-test, then Reveal the truth. Use Replicate x500 to see what happens when the same experiment is repeated, and raise "unrelated outcomes" to see how testing many things produces false positives.

Presets

Controls

Live readouts

Numerical checks

The science

Hypotheses. H0: the fertiliser has no effect (μT = μC). H1: the means differ. Plant heights are modelled as Normal(20 cm, σ) in both groups; the hidden world adds δ = dσ cm to the treated group (d = 0, 0.2, 0.5, 0.8: none, small, medium, large effect).

Test. Welch's t: t = (mT−mC)/√(sT²/n + sC²/n) with Welch–Satterthwaite degrees of freedom; two-sided p-value from the regularised incomplete beta function, p = Iν/(ν+t²)(ν/2, 1/2). The 95 % confidence interval is Δ ± tcrit·SE.

Error rates. Type I (false positive) rate = α when H0 is true. Power = 1 − β = chance to detect a real effect; the normal approximation is power ≈ Φ(d√(n/2) − z1−α/2) + Φ(−d√(n/2) − z1−α/2). Testing k unrelated outcomes gives P(at least one false positive) = 1 − (1−α)k.

Honest simplifications. Idealised Normal data, equal group sizes, independent outcomes, no randomisation or measurement problems, which are where real experiments usually fail. A "significant" p-value says the data are unlikely under H0, not that H1 is probably true, and p says nothing about effect size or importance.