Cross-Validation Lambda Tuner: Ridge Regression

Model & Data

Polynomial degree 5

Noise level 0.25

K (folds) 5

Cross-Validation Result

Optimal λ—

CV error (mean MSE)—

Train error (MSE)—

Legend

Low validation error

High validation error

Column selected by CV (min mean error)

How it works

Ridge regression fits β by minimizing squared error plus an L2 penalty on the coefficients:

minimize  ||y - Xβ||² + λ||β||²
closed form:  β = (XᵀX + λI)⁻¹ Xᵀy

Larger λ shrinks coefficients toward zero (less overfitting, more bias); smaller λ lets the polynomial fit the training noise (less bias, more variance). Picking λ by eye on the training set alone always favors λ→0 — training error keeps dropping as the curve gets wigglier.

K-fold cross-validation avoids that trap: the data is split into K folds. For every candidate λ, the model is trained on K−1 folds and evaluated on the held-out fold, K times, rotating which fold is held out. The bars in the grid are exactly these held-out (validation) MSE values — one column per λ (10 candidates, log-spaced 10⁻³…10²), one row per fold.

CV(λ) = (1/K) Σᵢ MSE( model trained without fold i, evaluated on fold i )

The λ minimizing CV(λ) — the gold-highlighted column — is the one expected to generalize best to unseen data. Push the polynomial degree up and watch the low-λ columns spike (overfitting shows up as high validation error despite low training error); push noise up and watch the optimal column shift toward larger λ.

Bar grid: X = λ (log-spaced), Z = fold, height = validation MSE

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.