This simulation visualises how a statistical hypothesis test decides between a null hypothesis H₀ and an alternative H₁. It draws random samples from a normal population, computes a test statistic, and converts it into a two-tailed p-value using the t-distribution (or the χ² distribution for the goodness-of-fit option). The upper chart shows the null and alternative sampling distributions with the rejection region shaded; the lower chart accumulates the empirical distribution of p-values across repeated trials.
You select the test (one-sample t, two-sample t, or χ² goodness-of-fit) and adjust sample size n, true effect size δ, significance level α, and population standard deviation σ. Each run randomly draws either a true null or a true effect, so the running tallies of Type I errors, Type II errors, and correct decisions reveal how power = 1−β responds to your choices — the same logic researchers use when planning clinical trials, A/B tests, and quality-control checks.
What is hypothesis testing?
Hypothesis testing is a procedure for deciding whether sample data are consistent with a default claim called the null hypothesis. You compute a test statistic, find how surprising it would be under the null, and reject the null if that surprise (the p-value) falls below a chosen threshold. This simulator lets you watch that decision unfold over many random samples.
What does the p-value mean here?
The p-value is the probability of observing a test statistic at least as extreme as the one obtained, assuming the null hypothesis is true. The simulator computes it two-tailed from the t-distribution, so a t-statistic far from zero gives a small p. If the p-value is below α, the decision flips from "Fail to Reject" to "Reject H₀".
What do the four sliders control?
Sample size n (5–200) sets how many observations are drawn per trial; effect size δ (0–3) is the true mean shift used when an effect is present; significance α (0.01–0.20) sets the rejection threshold and critical value; and standard deviation σ (0.5–3) sets the population spread. Larger n or δ, or smaller σ, increase statistical power.
For the one-sample test the statistic is t = x̄ ÷ √(s²/n), where x̄ is the sample mean and s² the sample variance, on n−1 degrees of freedom. The two-sample version uses the difference of means divided by the pooled standard error on 2n−2 degrees of freedom. The χ² option instead sums (observed−expected)²/expected over four categories.
A Type I error is rejecting a true null hypothesis — a false positive whose long-run rate equals α. A Type II error is failing to reject a false null — a missed real effect, with rate β. The counters in the panel tally both as you run trials, so you can confirm that the Type I rate hovers near your chosen α.
Power is the probability of correctly rejecting the null when a real effect of size δ exists, equal to 1−β. The simulator estimates it from the non-centrality parameter δ÷(σ/√n) relative to the critical value. Raising n or δ, or lowering σ, pushes the alternative distribution further into the rejection region and so increases the displayed power.
When the null hypothesis is true, p-values are uniformly distributed between 0 and 1, which is why the lower histogram flattens toward the dashed reference line for null trials. When a true effect is present, p-values pile up near zero. Because each run is a 50/50 mix of null and effect, you see a blend of both shapes.
The core mathematics is genuine: it samples from a normal distribution via the polar Box–Muller method and evaluates the t and χ² distribution functions using continued-fraction and series approximations of the incomplete beta and gamma functions. The power figure uses a normal approximation to the non-central t, so it is a close estimate rather than an exact tabulated value.
The one-sample t-test checks whether a single group's mean differs from a reference value. The two-sample t-test compares the means of two independent groups. The χ² goodness-of-fit test checks whether observed category counts match an expected distribution — here, four categories against a uniform expectation. Each uses different degrees of freedom and a different sampling distribution.
The critical value marks the boundary of the rejection region, and lowering α demands stronger evidence, pushing that boundary further into the tails. The simulator solves for it by bisection on the t (or χ²) distribution so that the tail area equals α. A smaller α reduces Type I errors but, all else equal, lowers power and raises Type II errors.
The same framework underpins clinical trials, A/B tests in software, agricultural field trials, and manufacturing quality control. Practitioners use power analysis — exactly the n, δ, σ, and α trade-off shown here — to choose a sample size large enough to detect a meaningful effect while keeping false-positive and false-negative rates acceptably low before any data are collected.