The Law of Large Numbers and the Central Limit Theorem

Two laws, two questions

The Law of Large Numbers (LLN) and the Central Limit Theorem (CLT) are the two results that make averaging meaningful. The LLN answers "where does the average go?" — as you take more and more independent samples from any distribution with a finite mean, the running average converges to that true mean. The CLT answers a different question, "what does the average look like along the way?" — the distribution of sample means, taken over many repeated samples of size n, approaches a normal (bell-shaped) curve as n grows, regardless of what shape the original distribution had.

Why the average of dice rolls has to settle near 3.5

A fair six-sided die has expected value E[X] = (1+2+3+4+5+6)/6 = 3.5. After a handful of rolls the running average can sit almost anywhere — three rolls of 6, 6, 1 average to 4.33. But every additional roll is an independent draw from the same distribution, and the variance of the average of n rolls shrinks as 1/n:

Weak Law of Large Numbers:
  For any ε > 0,   P( |X̄_n − μ| > ε )  →  0   as n → ∞

Var(X̄_n) = σ² / n     →   the average's own spread shrinks with n

Strong Law of Large Numbers:
  X̄_n → μ   almost surely (with probability 1)

The distribution of the running average gets narrower and narrower around 3.5 as n grows, which is why it looks like convergence on a graph even though, technically, nothing forbids an enormous run of sixes at any point — such a run just becomes vanishingly unlikely, and any early streak gets diluted by the overwhelming number of later rolls.

A common misreading: the gambler's fallacy

The LLN is frequently, and wrongly, invoked to justify the belief that a coin "owes" you tails after a run of heads. The die and the coin have no memory: every roll is drawn from the same distribution independent of what came before, and the sequence 6,6,6,6,6,6 is exactly as likely as any other specific sequence of six rolls. The average returns to 3.5 not because future rolls compensate for past ones, but because a long run gets swamped by the sheer volume of new, independent data that follows it — dilution, not correction.

The Central Limit Theorem, precisely

Take samples of size n from any distribution with finite mean μ and finite variance σ² — it can be a fair die, a heavily skewed income distribution, or anything else — compute the mean of each sample, and look at how those sample means are themselves distributed:

Central Limit Theorem:

  X̄_n  ≈  Normal( μ,  σ²/n )     for large n

  standard error of the mean:  SE = σ / √n

This is a remarkable, almost universal result: the shape of the original distribution — uniform like a die, exponential, wildly skewed — essentially disappears from the distribution of the sample mean, which converges toward the same bell curve regardless. It is the theoretical foundation for confidence intervals, margin-of-error calculations in polling, and most of classical hypothesis testing, all of which lean on the assumption that a sample mean is approximately normally distributed even when the underlying data obviously is not.

Diminishing returns: why the standard error shrinks so slowly

Because the standard error scales as σ/√n rather than σ/n, cutting your uncertainty in half requires quadrupling your sample size, not doubling it. That square-root law is the quiet reason polling organisations need thousands of respondents to bring a margin of error down to a couple of percentage points, and why doubling the length of a physics measurement run only shrinks statistical noise by about 41%, not 50%.

Frequently asked questions

Try it live

Everything above runs in your browser — open Law of Large Numbers and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.