Fitting a polynomial of degree k to noisy data always makes the residual sum of squares (RSS) go down as k grows — the curve simply has more freedom to chase the noise. Picking the model with the lowest training RSS therefore always picks the most complex model, even when it overfits. Information criteria fix this by adding an explicit penalty for the number of parameters:
AIC = n·ln(RSS/n) + 2·p
BIC = n·ln(RSS/n) + p·ln(n)
p = k + 1 (coefficients + intercept)
Both terms reward a lower RSS, but the penalty term grows with model size — BIC's penalty (p·ln n) grows faster than AIC's (2p) once n > 7, so BIC prefers a simpler model than AIC on the same data, especially at larger sample sizes.
- Left panel — the scatter is the noisy sample; the yellow curve is the least-squares fit of whichever degree you're inspecting. Drag it past the true complexity and watch it start wiggling through individual points instead of following the underlying shape — that's overfitting, visible directly.
- Right panel — one orange (AIC) and one green (BIC) bar per candidate degree 1–10, height ∝ the criterion's value (shorter is better). A ring marks each criterion's minimum — its actual model pick.
- Noise σ raises variance in the sample, pushing both criteria toward simpler, more conservative picks. Sample size n strengthens BIC's extra penalty. True curve complexity sets the real underlying polynomial degree the data was generated from — compare it to what AIC/BIC actually select.
Real-world relevance: this is the same trade-off behind choosing the number of predictors in a regression, the order of an ARIMA model, or the depth of a decision tree — AIC/BIC are a training-data-only alternative to cross-validation for the same job.