Confidence Intervals, the Z-Test, and What 'Coverage' Really Means

What a confidence interval actually claims

A confidence interval is a range of plausible values for an unknown population parameter, built from sample data, paired with a confidence level such as 95%. The single most common misinterpretation is treating a 95% confidence interval as "there is a 95% probability the true parameter lies in this specific interval." That statement is not correct once the interval has already been calculated from a specific sample — the true population parameter is a fixed, if unknown, number, and any particular calculated interval either contains it or it does not.

The correct interpretation is about the procedure, not any single interval: if you repeated the sampling process many times and built a confidence interval the same way each time, approximately 95% of those intervals would contain the true population parameter. This long-run frequency property is called coverage, and it is best understood visually rather than verbally.

Building a Z confidence interval for a mean

When the population standard deviation σ is known (or, more commonly in practice, a large sample lets you substitute the sample standard deviation s with negligible loss of accuracy), a confidence interval for a population mean μ is built as

x̄ ± z_* · (σ / √n)

where x̄ is the sample mean, n is the sample size, and z_* is the critical value from the standard normal distribution corresponding to the chosen confidence level: 1.645 for 90% confidence, 1.96 for 95% confidence, and 2.576 for 99% confidence. The quantity σ/√n is the standard error of the mean, and it should look familiar — it is exactly the standard deviation of the sampling distribution of x̄ guaranteed by the Central Limit Theorem. This is the fundamental reason confidence intervals lean on the CLT: the entire construction assumes x̄ is approximately normally distributed around μ, which the CLT justifies even when the underlying population is not itself normal, provided the sample is reasonably large.

Visualizing coverage through repeated sampling

The clearest way to see what a confidence interval promises is to simulate it directly. Fix a known population mean μ (something you would never know in a real study, but can set arbitrarily in a simulation), then repeatedly draw random samples of size n, compute a 95% confidence interval from each sample, and plot every interval as a horizontal line segment, stacked one above another, with a vertical reference line marking the true μ. In such a plot, some intervals miss the true mean entirely — and crucially, across a large number of repeated simulated samples, close to 95% of the plotted intervals cross the reference line while close to 5% do not. That proportion, not the width or position of any one interval, is what "95% confidence" is actually describing, and it only converges to exactly 95% as the number of repeated samples grows large, itself another manifestation of the law of large numbers.

The Z-test: using the same machinery to test a hypothesis

A Z-test asks a related but different question: is the observed sample mean consistent with some specific hypothesised population mean μ₀, or is it far enough away to cast doubt on that hypothesis? The test statistic is

z = (x̄ − μ₀) / (σ / √n)

which measures how many standard errors the observed sample mean sits away from the hypothesised value. Under the null hypothesis that μ truly equals μ₀, this z statistic follows a standard normal distribution, again by the Central Limit Theorem. The resulting p-value is the probability of observing a z statistic at least as extreme as the one calculated, purely by chance, if the null hypothesis were actually true. If that p-value falls below a pre-chosen significance level (commonly 0.05), the result is called statistically significant, and the null hypothesis is rejected in favour of the alternative.

Common misinterpretations worth avoiding

A p-value of 0.03 does not mean there is a 3% probability the null hypothesis is true, nor a 97% probability the alternative hypothesis is true; it is a statement about how surprising the observed data would be, if the null hypothesis were assumed true from the start — a subtle but important distinction, since it says nothing directly about the probability of either hypothesis itself. Similarly, "statistically significant" is not the same as "practically important": with a large enough sample size, even a trivially small and practically meaningless difference from μ₀ can produce a very small p-value, because the standard error shrinks as n grows, making the test increasingly sensitive to tiny deviations. Reporting both a confidence interval and a p-value side by side is generally better practice than either alone, because the interval directly conveys the magnitude and precision of an effect, while the p-value only conveys whether it clears an arbitrary significance threshold.

Where these tools show up in finance and medicine

In clinical trials, confidence intervals for the difference in outcomes between treatment and control groups are the standard way regulators and clinicians judge both the size and the reliability of a treatment effect, and a Z-test (or its small-sample cousin, the t-test) is typically the formal statistical test behind the trial's headline result. In quantitative finance, confidence intervals are used to bound estimates like expected portfolio returns or Value-at-Risk, while Z-tests and their variants underlie A/B testing frameworks used to decide whether a change to a trading strategy, pricing model, or website design produced a genuine improvement rather than one attributable to random sampling noise. In both domains, the underlying logic is identical to the coin-flip and coffee-cup examples used to teach the method: it is the Central Limit Theorem, applied to a mean or a proportion, that makes the whole machinery valid.