Linear Regression (2D)

Add or drag points, fit a line by least squares, absolute or Huber loss, and watch gradient descent roll down the loss valley with live statistics.

Presets

Playback

Data

Loss

Gradient descent

Layers

Model check

The science

The model

Linear regression fits a line y = w x + b to points (xi, yi) by minimising a loss of the residuals ri = yi - w xi - b: L(w, b) = (1/n) Σ ρ(ri) + (λ/2) w2. Squared loss uses ρ(r) = r2/2 (ordinary least squares), absolute loss ρ(r) = |r| (pulls towards the median), and the Huber loss is quadratic for |r| ≤ δ and linear beyond, so far outliers have only bounded influence. The term with λ is ridge (L2) shrinkage of the slope.

Closed form and the statistics

For squared loss without ridge the normal equations give w = Sxy/Sxx, b = ȳ - w x̄ (S are centred sums). With ridge, w = Sxy/(Sxx + nλ). The coefficient of determination is R2 = 1 - SSE/SST, which equals the squared correlation r2 = Sxy2/(Sxx Syy). If the noise is independent, Gaussian and of constant variance, the slope has standard error s/√Sxx with s2 = SSE/(n-2), giving the t-interval w ± t0.975,n-2 s/√Sxx; the shaded bands show the corresponding confidence band of the mean line and the wider prediction band for a new point. Absolute and Huber minimisers have no closed form; the page finds them by iteratively reweighted least squares.

Gradient descent

The cyan line is a gradient descent with momentum: v ← μ v - η ∇L, (w, b) ← (w, b) + v, where ∇L = (-(1/n) Σ ψ(ri) xi + λ w, -(1/n) Σ ψ(ri)). Because x and the intercept are strongly correlated, the loss is a long narrow valley (see the contour graph) and descent zig-zags; the Hessian condition number is large. Standardising x makes the valley round, which is why the checkbox converges in a few dozen steps. A learning rate above 2 divided by the largest Hessian eigenvalue makes the iteration diverge, which the verdict line reports honestly.

Bootstrap

Resampling the n points with replacement 500 times and refitting gives an empirical spread of the slope that does not rely on the textbook assumptions. When the data are straight with constant noise the bootstrap spread and the formula agree; with a curved shape, a funnel or outliers they do not, and that gap is the point of the comparison.

Honest simplifications

Points are synthetic and x is in [0, 10]. The bootstrap uses OLS slopes whatever loss is selected. The t quantile is a series approximation (error below 0.1 % for 4 or more degrees of freedom). The reference minimiser for MAE is accurate to about 1e-4 in loss, not to machine precision. Correlation is not causation, and a high R2 says nothing about whether a straight line is the right model: try the curved preset.

What to try

Load "Outliers: squared loss" and then "Outliers: Huber loss" and compare the lines. Drag one point far away and watch R2, the interval and the contour change. Raise the learning rate until descent diverges, then tick standardisation. Click the contour graph to restart descent from any slope and intercept.

Gradient descent: excess loss vs steps
Residuals of the reference fit
Loss contour over (w, b) with the descent path: click to restart
Slope: bootstrap histogram vs textbook interval