Student's t-Test: Deciding If a Difference Is Real

The question behind every t-test

Every t-test asks the same underlying question: is the difference I measured bigger than I would expect from random sampling noise alone? William Sealy Gosset, working as a statistician at Guinness in 1908 and publishing under the pseudonym "Student" because the brewery restricted employee publications, derived the exact sampling distribution for small samples where the population standard deviation is unknown and has to be estimated from the data itself — the situation almost every real experiment is actually in.

One-sample, two-sample, paired: three different questions

The one-sample t-test compares a sample mean against a fixed known value. The two-sample (independent) t-test compares the means of two separate, unrelated groups. The paired t-test compares two measurements taken on the same subjects (before/after, or matched pairs), which is really a one-sample test run on the differences — pairing removes the between-subject variability that would otherwise drown out a real effect, so it is more powerful whenever the pairing is genuine.

one-sample:   t = (x̄ − μ₀) / (s / √n)
two-sample:   t = (x̄₁ − x̄₂) / SE(x̄₁ − x̄₂)
paired:       t = d̄ / (s_d / √n)          d = paired differences

degrees of freedom (Welch, unequal variance):
df ≈ (s₁²/n₁ + s₂²/n₂)² / [ (s₁²/n₁)²/(n₁−1) + (s₂²/n₂)²/(n₂−1) ]

Why the t-distribution has fatter tails than the normal

When the sample size is large, the sample standard deviation s is a very reliable estimate of the true population deviation σ, and the t-distribution converges to the standard normal. When n is small, s itself is noisy — sometimes it happens to underestimate the true spread, which inflates the t-statistic and would make the normal distribution declare significance too often. The t-distribution's fatter tails exist precisely to correct for that extra uncertainty in estimating σ from a small sample; as degrees of freedom (roughly, sample size minus the number of means already estimated) increase, the tails shrink back toward the normal curve.

p-values and Cohen's d are answering different questions

The p-value is the probability, assuming there is truly no effect (the null hypothesis), of seeing a t-statistic at least as extreme as the one observed. A small p-value is evidence against the null, not a measure of how large or important the effect is — a tiny, practically meaningless difference can still produce a small p-value if the sample is large enough. Cohen's d, the standardized effect size (the mean difference divided by the pooled standard deviation), answers the separate question of how big the effect actually is in units of natural variability, independent of sample size. Reporting both is standard practice precisely because statistical significance and practical importance are not the same thing.

The assumptions worth knowing

The classical t-test assumes the underlying data (or the differences, for a paired test) are approximately normally distributed, though the test is fairly robust to mild violations once n is reasonably large, thanks again to the central limit theorem. The two-sample version further assumes either equal variances (the pooled/Student's version) or uses Welch's correction for unequal variances and unequal sample sizes, which is now the more commonly recommended default because it does not require an assumption that is often untrue and rarely tested.

Frequently asked questions

Try it live

Everything above runs in your browser — open Student's t-Test and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.