Fitting a polynomial of degree k to noisy data always makes the residual sum of squares (RSS) go down as k grows — the curve simply has more freedom to chase the noise. Picking the model with the lowest training RSS therefore always picks the most complex model, even when it overfits. Information criteria fix this by adding an explicit penalty for the number of parameters:
AIC = n·ln(RSS/n) + 2·p
BIC = n·ln(RSS/n) + p·ln(n)
p = k + 1 (coefficients + intercept)
Both terms reward a lower RSS, but the penalty term grows with model size — BIC's penalty (p·ln n) grows faster than AIC's (2p) once n > 7, so BIC prefers a simpler model than AIC on the same data, especially at larger sample sizes.
- Top panel — the scatter is the noisy sample, the grey dashed curve is the true underlying polynomial it was generated from, and the yellow curve is the least-squares fit of whichever degree you're inspecting. Drag it past the true complexity and watch it start wiggling through individual points instead of following the underlying shape — that's overfitting, visible directly.
- Middle panel — training RSS for every candidate degree 1–10. It only ever goes down (or flattens) as degree rises, which is exactly why RSS alone can never be used to pick a model.
- Bottom panel — one orange (AIC) and one green (BIC) bar per candidate degree, height ∝ the criterion's value (taller is better here — bars are drawn inverted so the winning, lowest-criterion degree stands tallest). A ring marks each criterion's minimum — its actual model pick.
- Noise σ raises variance in the sample, pushing both criteria toward simpler, more conservative picks. Sample size n strengthens BIC's extra penalty. True curve complexity sets the real underlying polynomial degree the data was generated from — compare it to what AIC/BIC actually select.
Real-world relevance: this is the same trade-off behind choosing the number of predictors in a regression, the order of an ARIMA model, or the depth of a decision tree — AIC/BIC are a training-data-only alternative to cross-validation for the same job.