Linear Regression: What OLS Is Actually Minimising

Fitting a line by minimising squared error

Ordinary least squares finds the line ŷ = β₀ + β₁x that best fits a scatter of points by minimising the sum of squared vertical distances between each point and the line — the residuals. Squaring does two jobs at once: it makes every residual positive so they can't cancel out, and it penalises large errors far more than small ones, which pulls the fitted line toward reducing any single big miss. The line that minimises this sum has a closed-form solution, no iteration required, which is part of why OLS has stayed the default starting point for two centuries of applied statistics.

The normal equations

Setting the derivative of the sum of squared residuals to zero for both β₀ and β₁ gives two linear equations — the normal equations — with a direct solution in terms of the data's means and covariance:

slope:      beta1 = sum((x_i - x_bar) * (y_i - y_bar)) / sum((x_i - x_bar)^2)
intercept:  beta0 = y_bar - beta1 * x_bar

equivalently, in covariance form:  beta1 = cov(x, y) / var(x)

Two immediate consequences follow directly from that derivation. First, the fitted line always passes through the point of means, (x̄, ȳ) — plug x̄ into the formula for ŷ and beta0 exactly cancels it out. Second, the residuals always sum to zero and are uncorrelated with x by construction, not by luck; those are the equations OLS is explicitly built to satisfy, not properties you need to verify afterward.

R-squared and Pearson's r

Pearson's r measures the strength and direction of the linear association, ranging from -1 (perfect negative) to +1 (perfect positive), with 0 meaning no linear relationship. R-squared, the coefficient of determination, is simply r² for simple linear regression, and it has a direct interpretation: the fraction of the total variance in y that the line explains. An R² of 0.81 means the line accounts for 81% of the spread in y; the remaining 19% is unexplained — noise, omitted variables, or genuine nonlinearity the line cannot capture.

R^2 = 1 - SS_res / SS_tot
SS_res = sum((y_i - y_hat_i)^2)       // squared error the line leaves behind
SS_tot = sum((y_i - y_bar)^2)         // squared error a flat mean-only line leaves behind

Assumptions OLS quietly relies on

The Gauss-Markov theorem guarantees OLS is the best linear unbiased estimator — the lowest-variance linear estimator among all unbiased ones — but only if several assumptions hold: the true relationship is linear in the parameters, residuals have constant variance (homoscedasticity), residuals are uncorrelated with each other, and the predictor is measured without error. Break homoscedasticity and the line itself stays unbiased but its standard errors, and therefore any confidence interval or p-value built on them, become unreliable. This is why plotting residuals against fitted values is standard practice — a residual plot that fans out into a cone, or curves systematically, is telling you the model's assumptions have failed even if R² still looks respectable.

Correlation, causation, and the outlier problem

A high R² says the line fits the observed data well; it says nothing about whether x causes y — both could be driven by a shared third factor, or the causal arrow could run the other way. Separately, OLS's own squaring of residuals is its biggest practical weakness: because large errors are penalised quadratically, a single extreme outlier can drag the fitted slope substantially, far more than a typical point would. Practitioners deal with this by inspecting leverage and influence statistics (like Cook's distance), or by switching to robust regression methods that penalise large residuals less aggressively than squaring does.

Frequently asked questions

Try it live

Everything above runs in your browser — open Linear Regression and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.