Hypothesis Testing: p-values, Errors and Power

The question behind every experiment

You measure a difference — a new drug lowers blood pressure by 4mmHg, version B of a website converts 0.8% better than version A. Is that real, or could random sampling noise alone have produced a difference that size? Null hypothesis significance testing (NHST), formalised by Jerzy Neyman and Egon Pearson in the 1930s, is the standard machinery for answering that question. It underlies drug trials, particle-physics discoveries, and every A/B test you've ever run — and it is also, reliably, the part of statistics researchers get wrong most often.

Null vs alternative

Every test starts with two competing claims. The null hypothesis H₀ is the conservative default — usually "no difference," "no effect," "the new drug does nothing." The alternative hypothesis H₁ is what you're actually trying to find evidence for. You then pick a significance level α in advance — the maximum acceptable rate of false alarms, conventionally 0.05 in medicine and social science, but tightened to 0.0000003 (the famous "5σ") in particle physics, where a false "discovery" is expensive to retract. Everything downstream — the test statistic, the p-value, the decision — is built from data collected after α is fixed, not chosen to make the result look better afterward.

The t-test statistic

The workhorse test for comparing means is Student's t-test. For two independent samples with means x̄₁, x̄₂, it standardises the observed difference by its estimated standard error:

t = (x̄₁ − x̄₂) / SE_diff
SE_diff = √(s₁²/n₁ + s₂²/n₂)        // pooled or Welch variant

// t follows a Student's t-distribution with df degrees of freedom
// under H₀ (no true difference in means)
// p = P(|T| ≥ |t observed|)  — area in both tails beyond ±t

The larger |t| is, the further the observed difference sits from what pure sampling noise would typically produce under H₀ — and the smaller the resulting p-value. Other common tests follow the same logic with a different statistic and reference distribution: chi-square (χ² = Σ(O−E)²/E) for categorical counts, F for comparing more than two group means (ANOVA), Pearson's r for linear association.

What a p-value is — and is not

The single sentence that matters: the p-value is P(data at least this extreme | H₀ is true) — a statement about the data, conditioned on the null being true. It is not P(H₀ is true | data), which would require Bayes' theorem and a prior probability the classical framework never supplies. It is not "the probability the result is due to chance." And it is not effect size — a p-value tells you whether an effect is distinguishable from zero, never how large or important it is. With n = 1,000,000 a trivial, clinically meaningless difference will still produce p < 0.001, because a p-value shrinks with sample size even while the effect stays the same.

Two ways to be wrong

Every decision to reject or not reject H₀ can go wrong in one of two directions:

              H₀ true            H₀ false
Reject H₀     Type I error (α)   Correct rejection (power = 1−β)
Fail reject   Correct (1−α)      Type II error (β)

// Type I  = false positive: "found" an effect that isn't there
// Type II = false negative: missed a real effect
// α and β trade off for fixed n — tightening α raises β

Which error matters more depends on the stakes: in drug-safety screening, a missed side effect (Type II) can be far more costly than a false alarm; in exploratory particle physics, chasing every 2σ blip (Type I) would flood the literature with "discoveries" that vanish on replication — which is exactly why physics uses 5σ instead of the social-science default of 0.05.

Statistical power and sample size

Power is 1 − β: the probability of correctly detecting a real effect of a given size. It rises with a larger sample n, a larger true effect δ, a smaller measurement noise σ, and (perversely) a looser α. Convention targets 80% power. The required sample size for a two-sample, two-sided comparison follows directly from the normal approximation:

n ≈ 2(z_α/2 + z_β)² / d²      // per group, d = δ/σ = Cohen's d

// Example: medium effect d = 0.5, α = 0.05, power = 0.80
// z_α/2 = 1.96, z_β = 0.84
// n ≈ 2 × (1.96 + 0.84)² / 0.25 ≈ 63 subjects per group

Studies run with fewer subjects than this are underpowered: they are unlikely to detect the effect at all, and on the rare occasion they do reach significance, the estimated effect size tends to be inflated — the so-called "winner's curse" that plagues small, underpowered studies across every field. Reporting an effect size (Cohen's d, correlation r, odds ratio) alongside the p-value is now considered mandatory good practice for exactly this reason: statistical significance and practical importance are simply different questions.

Frequently asked questions

Try it live

Every distribution and rejection region above runs live in Hypothesis Testing — t-test & p-value. Change sample size, effect size and α, and watch power and the p-value update instantly, entirely in your browser.