Cross-Validation Lambda Tuner: Ridge Regression (2D)

Model & Data

Lambda (log₁₀ λ) 0.10

Polynomial degree 5

Noise level 0.25

K (folds) 5

Cross-Validation Result

Optimal λ (CV)—

CV error at optimal λ—

CV error at current λ—

Train error at current λ—

Legend

Fitted ridge curve (current λ)

Noisy data points

CV error curve (mean over K folds)

CV-optimal λ / current λ marker

How it works

Ridge regression fits β by minimizing squared error plus an L2 penalty on the coefficients:

minimize  ||y - Xβ||² + λ||β||²
closed form:  β = (XᵀX + λI)⁻¹ Xᵀy

Larger λ shrinks coefficients toward zero (less overfitting, more bias); smaller λ lets the polynomial fit the training noise (less bias, more variance). Picking λ by eye on the training set alone always favors λ→0 — training error keeps dropping as the curve gets wigglier. Drag the λ slider yourself and watch the top plot's curve go from a stiff near-flat line at high λ to a wild wiggle chasing every point at low λ.

K-fold cross-validation avoids that trap: the data is split into K folds. For every candidate λ, the model is trained on K−1 folds and evaluated on the held-out fold, K times, rotating which fold is held out, and the K held-out errors are averaged. The bottom plot is exactly that: mean validation MSE recomputed from scratch at 60 log-spaced λ values by genuinely re-splitting, re-fitting and re-evaluating the model at each one — not a canned bowl shape.

CV(λ) = (1/K) Σᵢ MSE( model trained without fold i, evaluated on fold i )

The λ minimizing CV(λ) — marked with the gold dot — is the one expected to generalize best to unseen data. Push the polynomial degree up and watch the low-λ end of the CV curve spike (overfitting shows up as high validation error despite low training error); push noise up and watch the CV-optimal λ shift toward larger values.

Top: data + fitted curve at current λ · Bottom: real re-fitted CV error vs λ