Linear Regression 3D

The model

Linear regression fits a line y = w x + b to points (xi, yi) by minimising a loss of the residuals ri = yi - w xi - b: L(w, b) = (1/n) Σ ρ(ri) + (λ/2) w2. Squared loss uses ρ(r) = r2/2 (ordinary least squares), absolute loss ρ(r) = |r| (pulls towards the median), and the Huber loss is quadratic for |r| ≤ δ and linear beyond, so far outliers have only bounded influence. The term with λ is ridge (L2) shrinkage of the slope.

Closed form and the statistics

For squared loss without ridge the normal equations give w = Sxy/Sxx, b = ȳ - w x̄ (S are centred sums). With ridge, w = Sxy/(Sxx + nλ). The coefficient of determination is R2 = 1 - SSE/SST, which equals the squared correlation r2 = Sxy2/(Sxx Syy). If the noise is independent, Gaussian and of constant variance, the slope has standard error s/√Sxx with s2 = SSE/(n-2), giving the t-interval w ± t0.975,n-2 s/√Sxx; the shaded bands show the corresponding confidence band of the mean line and the wider prediction band for a new point. Absolute and Huber minimisers have no closed form; the page finds them by iteratively reweighted least squares.

Gradient descent

The cyan line is a gradient descent with momentum: v ← μ v - η ∇L, (w, b) ← (w, b) + v, where ∇L = (-(1/n) Σ ψ(ri) xi + λ w, -(1/n) Σ ψ(ri)). Because x and the intercept are strongly correlated, the loss is a long narrow valley (see the contour graph) and descent zig-zags; the Hessian condition number is large. Standardising x makes the valley round, which is why the checkbox converges in a few dozen steps. A learning rate above 2 divided by the largest Hessian eigenvalue makes the iteration diverge, which the verdict line reports honestly.

Bootstrap

Resampling the n points with replacement 500 times and refitting gives an empirical spread of the slope that does not rely on the textbook assumptions. When the data are straight with constant noise the bootstrap spread and the formula agree; with a curved shape, a funnel or outliers they do not, and that gap is the point of the comparison.

Honest simplifications

Points are synthetic and x is in [0, 10]. The bootstrap uses OLS slopes whatever loss is selected. The t quantile is a series approximation (error below 0.1 % for 4 or more degrees of freedom). The reference minimiser for MAE is accurate to about 1e-4 in loss, not to machine precision. Correlation is not causation, and a high R² says nothing about whether a straight line is the right model: try the curved preset.

What to try

Load "Outliers: squared loss" and then "Outliers: Huber loss" and compare the lines. Drag one point far away and watch R², the interval and the contour change. Raise the learning rate until descent diverges, then tick standardisation. Click the contour graph to restart descent from any slope and intercept.

What the 3D scene shows

On the green data board each sphere is one data point (red when it lies more than 2s from the reference line); the white rod is the reference minimiser of the chosen loss, the cyan rod is the current gradient-descent line and the orange sticks are its residuals. In the glass case the loss L(w, b) is a bowl: height is the excess loss over the minimum (square-root compressed and clipped so the valley floor stays visible), colour runs from blue (low) to magenta (high), the green ring is the minimiser and the bright ball is the descent state with its trail. Both views use the same JavaScript model as the 2D page; the gallery was modelled and baked in Blender.