Give a model enough flexibility, a high-degree polynomial, a deep enough neural network, and it can fit every single training point exactly. That perfect fit is rarely good news: with real-world noisy data, a model matching every point exactly has almost certainly memorized the noise along with the signal, and it will perform badly on data it has not seen. Regularization is the family of techniques built to prevent this.
How L2 regularization works
L2 regularization, known as ridge regression when applied to linear models, adds a penalty to the training objective proportional to the sum of squared coefficients. Instead of purely minimizing prediction error on the training set, the model minimizes prediction error plus this penalty. Large coefficients, the kind that let a high-degree polynomial swing wildly to pass through every noisy point, become expensive, so the fitting process favors smaller, smoother coefficients instead.
Why this generalizes better
A smoother function, one that does not chase every noisy fluctuation in the training data, is more likely to reflect the actual underlying pattern rather than this particular sample's idiosyncrasies. That underlying pattern is what shows up again in new data, which is why regularized models often perform worse on the training set but noticeably better on held-out test data.
Choosing the regularization strength
The regularization strength, often called λ, controls this trade-off directly. Set it to zero and you are back to unregularized, overfitting-prone fitting. Set it too high and the model becomes too constrained to capture real signal, underfitting instead. In practice, λ is usually chosen via cross-validation: trying several values and picking whichever minimizes error on data the model was not trained on.
Try it yourself
The Regularization Methods Lab lets you push a 9th-degree polynomial into wild overfitting, then dial in L2 regularization and watch the fit smooth out and test error improve in real time.
🧪 Try it yourself: the Regularization Methods Lab simulation lets you experiment with everything described above directly in your browser.