How Regression to the Mean Works

When you select individuals based on an extreme value of one measurement X, you are selecting for a mix of true ability and above-average luck. On the next measurement Y, luck reverts to average, so the group appears to decline even when nothing has changed. This is regression to the mean — a statistical artifact, not a real effect.

E[Y | X = x] = μ_Y + ρ · (σ_Y / σ_X) · (x − μ_X)

The formula shows that the conditional expectation of Y given a specific X lies along a line with slope ρ·(σY/σX). When ρ < 1, this slope is less than the 45° line, meaning extreme X values predict Y values that are closer to the mean of Y than X is to the mean of X.

Francis Galton discovered this in 1886 studying parent-child heights: tall parents had tall children, but those children were on average not as tall as their parents relative to the mean. He called it “regression towards mediocrity.” The scatter plot above demonstrates this with a bivariate normal distribution where you can tune the correlation ρ and see how selection of extreme scorers leads to apparent regression.

Real-world examples: Athletes selected for peak performance who subsequently perform closer to their average; patients who seek treatment when symptoms peak and improve even with a placebo; students selected for remediation who improve due to regression, not the program; countries with the worst economic performance one year showing better growth the next.

Frequently Asked Questions

About this simulation

This simulation generates N pairs of correlated scores (x, y) from a bivariate normal distribution with correlation ρ, using a Box-Muller transform to produce standard normal draws and y = ρ·x + √(1−ρ²)·z. It then selects the top-performing slice on X (by threshold %) and shows how their average Y score falls back toward the population mean — the real statistical phenomenon behind sports slumps, medical placebo effects, and mistaken program evaluations.

🔬 What it shows

A scatter plot (or histogram) of period-1 versus period-2 scores for up to 600 simulated individuals, with the top-X% performers on X highlighted in yellow. A purple line plots the true conditional expectation E[Y|X] = ρ·X, a grey dashed line marks Y=X for comparison, and a red arrow shows the gap between the selected group's average X and their actual average Y — the regression effect itself.

🎮 How to use

Drag Correlation ρ (−1 to 1) to change how strongly period-1 and period-2 scores relate, Sample size N (30–600) to change how many points are drawn, and Selection threshold (5–50%) to change how extreme the highlighted group is. Switch between Scatter, Histogram and All views, hit Resample for a fresh random draw, or load the Sports Curse, Medicine and No Correlation presets to see pre-tuned examples of the effect.

💡 Did you know?

Francis Galton first documented this effect in 1886 studying parent and child heights, calling it "regression towards mediocrity" — the origin of the word "regression" in statistics. When ρ = 0 (the No Correlation preset), any group selected for extreme X scores regresses all the way back to the population mean on Y, because the two measurements share no real information at all.

Frequently asked questions

How does the simulation actually generate the correlated data?

It draws two independent standard normal values z1 and z2 using a Box-Muller transform, sets x = z1, and sets y = ρ·z1 + √(1−ρ²)·z2. This is the standard construction for a bivariate normal pair with correlation ρ: when ρ is close to 1, y tracks x almost exactly, and when ρ is close to 0, y is essentially independent noise.

How is the "selected" group and the regression arrow computed?

The simulation sorts all N points by their X value descending and takes the top slice defined by the Selection threshold slider (5–50%). It then computes that group's average X and average Y, and draws a red arrow from the point (avgX, avgX) on the Y=X identity line down to the actual point (avgX, avgY) — visually showing how far the group's real Y average falls short of matching its inflated X average.

Why does a lower correlation ρ produce stronger regression to the mean?

The purple regression line drawn on the scatter plot is E[Y|X] = ρ·X, so its slope is exactly ρ. At ρ = 1 this line coincides with Y=X and there is no regression at all; as ρ shrinks toward 0, the line flattens toward the horizontal, meaning any extreme X value predicts a Y value pulled almost all the way back to the overall mean of zero.

What do the Sports Curse, Medicine and No Correlation presets model?

Sports Curse sets ρ = 0.5, N = 300 and a tight 10% selection threshold, mimicking how only truly outstanding single-season performances get media attention and then regress hard the next season. Medicine uses ρ = 0.7, N = 150 and a 30% threshold, closer to how patients selecting into treatment at symptom peaks naturally improve afterward. No Correlation sets ρ = 0 to show the extreme case where selected performers regress completely to the population average.

Is the regression effect shown here a real causal effect or an artifact?

It is a statistical artifact of selecting on one noisy measurement, not a real force pulling scores down. The simulation's own info panel states this directly: any extreme score partly reflects genuine ability and partly reflects luck, and since luck by definition does not repeat, the next measurement naturally looks more average even though nothing about the underlying ability changed. This is exactly why the model computes a "Regression %" statistic rather than attributing the change to any real intervention.