Regression Visualized: Least Squares, Overfitting, Ridge and Lasso

Least squares: the closed-form best fit

Ordinary least squares chooses the coefficients β of a linear model y = Xβ that minimise the sum of squared residuals, ‖y − Xβ‖². Setting the gradient of that sum to zero gives the normal equations, which have a closed-form solution whenever XᵀX is invertible:

minimise  ‖y − Xβ‖²
∂/∂β ‖y − Xβ‖² = −2Xᵀ(y − Xβ) = 0
        ⇒   XᵀX β = Xᵀy
        ⇒   β̂ = (XᵀX)⁻¹ Xᵀy

"Linear" regression means linear in the coefficients, not in the visible curve — the same normal equations fit a straight line, a parabola or a degree-9 polynomial, as long as you expand the feature matrix X to include the extra columns x², x³, … x⁹ before solving. That is exactly what polynomial regression is: ordinary least squares on an engineered, higher-dimensional feature set.

Bias, variance, and the degree slider

A degree-1 fit to genuinely curved data is biased: it is the wrong family of function no matter how much data you feed it, and its error will not vanish even as the sample size grows — underfitting. A degree-9 fit to a handful of noisy points has almost no bias (with enough parameters it can pass arbitrarily close to every training point) but enormous variance: refit it on a slightly different noisy sample and the coefficients, and the curve between data points, swing wildly, especially near the edges of the data — overfitting. The expected test error decomposes cleanly into three pieces:

E[test error]  =  Bias²  +  Variance  +  irreducible noise σ²

low degree   →  high bias,  low variance   →  underfit
high degree  →  low bias,   high variance  →  overfit
              somewhere between: the degree that minimises the SUM

There is no free way to reduce bias without raising variance or vice versa by degree alone — that trade-off is the bias-variance trade-off, and the practical way to find the sweet spot is to hold out validation data (or cross-validate) and pick the degree that minimises validation error, never training error, since training error only ever decreases as you add more polynomial terms.

Ridge: shrink instead of drop

Instead of controlling complexity by choosing the polynomial degree, you can fit a high-degree model and penalise large coefficients directly. Ridge regression (Tikhonov regularisation) adds an L2 penalty λ‖β‖² to the least-squares objective:

minimise  ‖y − Xβ‖² + λ ‖β‖²
        ⇒   β̂ridge = (XᵀX + λI)⁻¹ Xᵀy

The extra λI term is also a numerically pleasant side effect: it makes the matrix being inverted better-conditioned (and always invertible, even when XᵀX itself is singular or near-singular from correlated features), which is part of why Ridge was originally introduced as a fix for exactly that numerical problem. As λ grows, every coefficient is pulled smoothly toward zero — but essentially never to zero — so Ridge keeps every feature in the model with a reduced weight, which suits situations where most or all of the features genuinely contribute a little.

Lasso: shrink some coefficients all the way to zero

Lasso (Least Absolute Shrinkage and Selection Operator) swaps the penalty for L1, λ‖β‖₁ = λ Σ|βi|:

minimise  ‖y − Xβ‖² + λ Σ |βi|          (no closed form in general)

Geometrically, the L1 penalty's constraint region is a diamond (in 2D) with sharp corners on the coordinate axes, while the L2 penalty's is a smooth circle; the least-squares solution contour is far more likely to first touch the L1 diamond exactly at a corner, where one or more coefficients are exactly zero, than it is to land on an axis of the smooth L2 circle. That geometric fact is the entire reason Lasso performs automatic feature selection — driving irrelevant coefficients to precisely zero — while Ridge only ever shrinks them close to it. The cost is that the L1 penalty is not differentiable at zero, so there is no normal-equations shortcut; Lasso is solved with coordinate descent or other convex-optimisation methods, and when features are highly correlated Lasso tends to arbitrarily pick one from a correlated group and zero out the rest, which Ridge handles more gracefully by shrinking correlated features together.

Elastic net blends both penalties, αλ‖β‖₁ + (1−α)λ‖β‖², to get sparse solutions like Lasso while keeping Ridge's more stable behaviour under correlated features.

Choosing λ

λ is exactly the same kind of knob as polynomial degree — it trades bias for variance, just continuously instead of in integer steps. λ = 0 recovers ordinary least squares (low bias, potentially high variance on a high-degree model); as λ → ∞ every coefficient is driven toward zero and the model degenerates to a flat line at the mean of y (high bias, zero variance). The standard way to pick it is k-fold cross-validation: fit on k−1 folds, score on the held-out fold, repeat across all folds and average, then pick the λ that minimises average validation error — never a value chosen by eye on the training curve, which is systematically fooled the same way an unregularised high-degree fit is.

Frequently asked questions

Try it live

Everything above runs in your browser — open Regression Visualizer and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.