The Normal Distribution and the Central Limit Theorem

The shape of the normal distribution

The normal distribution (also called the Gaussian distribution) is a continuous, symmetric, bell-shaped probability distribution fully described by just two parameters: its mean μ, which sets the location of the peak, and its standard deviation σ, which sets how spread out it is. Its probability density function is

f(x) = 1 / (σ√(2π)) · e^{−(x−μ)²/(2σ²)}

The curve is highest at x = μ, falls off symmetrically in both directions, and never quite touches zero, though it becomes negligibly small a few standard deviations out. The area under the entire curve is exactly 1, consistent with the second Kolmogorov axiom, and the area under any portion of the curve gives the probability of a value falling in that range.

The 68-95-99.7 rule

For any normal distribution, regardless of its specific μ and σ, the proportion of values falling within a given number of standard deviations of the mean is fixed. Approximately 68.27% of values fall within μ ± 1σ, approximately 95.45% fall within μ ± 2σ, and approximately 99.73% fall within μ ± 3σ. This is often shortened to the "68-95-99.7 rule" and is one of the most useful mental shortcuts in applied statistics: it lets you eyeball how unusual an observation is just from knowing how many standard deviations it sits from the mean, without touching a formula. Note that the commonly used 95% confidence interval threshold of ±1.96σ is deliberately slightly wider than the rounded "2 sigma" rule of thumb, because 1.96 is the precise value that captures exactly 95.00% of the area, not 95.45%.

Statement of the Central Limit Theorem

The Central Limit Theorem (CLT) is arguably the single most important result in statistics, because it explains why the normal distribution appears so often even when the quantities being measured individually are not normally distributed at all. In its most common form: if X₁, X₂, ..., X_n are independent, identically distributed random variables with finite mean μ and finite variance σ², then as n grows large, the distribution of the sample mean X̄ = (X₁ + X₂ + ... + X_n) / n approaches a normal distribution with mean μ and variance σ²/n — that is, X̄ approximately ∼ N(μ, σ²/n) — irrespective of the shape of the original distribution the X_i came from.

That last clause is the entire point. The individual data could be drawn from a distribution that is heavily skewed, bimodal, bounded, or otherwise nothing like a bell curve — incomes in a population, waiting times at a bus stop, the outcome of a single die roll — and the CLT still guarantees that the average of many independent draws from that distribution settles into an approximately normal shape as the sample size grows.

Watching convergence: what changes as n grows

A histogram of many repeated sample means, built by simulation, makes the CLT concrete rather than abstract. Start with a strongly skewed population — for instance, values drawn from an exponential distribution, which has a sharp peak near zero and a long right tail. Draw a sample of size n from that population, compute its mean, and repeat this process thousands of times, plotting a histogram of all the resulting sample means. With n = 1, the histogram of "sample means" is just the original skewed exponential shape. With n = 5, the skew is visibly softened. By n = 30 or so, the histogram is already close to a smooth, symmetric bell curve centred on the population mean — even though not a single one of the original data points was ever close to normally distributed.

Two things change simultaneously as n increases: the shape becomes more normal, and the spread narrows, because the standard deviation of the sample mean (called the standard error) equals σ/√n. Quadrupling the sample size halves the standard error, since √4 = 2 — a direct, testable consequence of the CLT's variance formula, and the mathematical reason that larger surveys and larger clinical trials produce more precise estimates.

Why the CLT underlies so much of applied statistics

Because sample means (and, by very similar reasoning, sums, sample proportions, and many other aggregated statistics) tend toward normality regardless of the underlying population, an enormous amount of classical statistical machinery — Z-tests, t-tests, confidence intervals built from a Z or t critical value, control charts in manufacturing quality assurance — can be constructed around the normal distribution without first needing to know or verify the shape of the raw data's distribution. This is precisely why a pollster surveying 1,000 randomly selected voters can quote a margin of error using normal-distribution mathematics, even though any individual voter's opinion is a binary, distinctly non-normal outcome.

Common misconceptions about the CLT

The CLT is frequently misstated, so it is worth being precise about what it does not say. It does not say that individual data points become normally distributed as you collect more of them — the shape of the raw population distribution never changes no matter how much data you gather. It applies to aggregated statistics like the sample mean, not to the raw observations themselves. It also requires the underlying distribution to have a finite variance; a small number of pathological distributions (the Cauchy distribution is the standard textbook counterexample) have such heavy tails that their variance is infinite, and for those, the CLT in its classical form simply does not apply, no matter how large n gets. Finally, "large n" is doing real work: highly skewed or heavy-tailed populations typically need a considerably larger sample size before the normal approximation becomes trustworthy, compared to populations that are already close to symmetric.