Least squares: β minimizes Σ(y − ŷ)², solved via the normal equations (XᵀX)β = Xᵀy.
Ridge (L2): minimizes Σ(y − ŷ)² + λΣβᵢ² → closed form (XᵀX + λI)⁻¹Xᵀy — shrinks all coefficients smoothly.
Lasso (L1): minimizes Σ(y − ŷ)² + λΣ|βᵢ| — solved by coordinate descent with soft-thresholding, which can drive coefficients to exactly zero (automatic feature selection).
The bottom chart plots training error (blue) and held-out test error (orange) against polynomial degree at the current regularization — the classic bias-variance U-curve.