Polynomial Feature Engineering & the Overfitting Curve

Engineered Features

Polynomial degree d 3

Ridge regularization λ 0.02

Noise level σ 0.35

Fit Diagnostics

φ(x) = [1, x, x², x³]

Train RMSE

0.00

Test RMSE

0.00

Features (d+1)

Fit status

good

How it works

Feature engineering here means literally expanding one raw number x into a vector of new features by taking powers of it: φ(x) = [1, x, x², …, x^d]. A plain linear model on φ(x) can now fit curves, not just straight lines — the "engineering" did the nonlinear work, the regression stays linear in the new features.

Ridge regression:
minimize  ||y - Φβ||² + λ||β||²
solution  β = (ΦᵀΦ + λI)⁻¹ Φᵀy

Φ = matrix of engineered features, one row per data point
λ = regularization strength (shrinks β toward 0)

Degree d — how many engineered polynomial features you add. Too few (d=1) can't capture the curve (underfitting); too many (d=10+) with λ≈0 fits the noise instead of the signal, wiggling wildly between points (overfitting).
λ (ridge) — penalizes large coefficients on the engineered features, damping the wiggle a high degree would otherwise produce. It's the standard fix for feature-engineering-caused overfitting.
Train vs. test RMSE — train error keeps falling as d grows (more features can always fit the training points better); test error on unseen points forms a U-shape. The bottom of that U is the degree that generalizes best — the whole point of feature engineering is to land there, not at the training minimum.

Real-world relevance: this is the exact mechanism behind adding polynomial/interaction terms, log-transforms, or ratio features to a tabular model — every engineered feature raises capacity, and every extra feature needs either enough data or enough regularization to pay for itself.

Train points Test points Fitted curve

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.