📉 Statistical Learning Theory for Hyperparameter Optimization

About this simulation

Statistical learning theory asks a simple but hard question: if a model fits the training data well, how confident can we be that it will also do well on data it has never seen? PAC (Probably Approximately Correct) learning answers this with generalization bounds — statements of the form "with probability at least 1−δ, the true risk is no more than the training risk plus a gap term that grows with model complexity and shrinks with more training data."

This scene renders that trade-off as a literal landscape. The floor plane spans two axes: model complexity (a stand-in for VC dimension / effective capacity, running left→right) and training-set size N (log scale, running front→back). The cyan sheet is training risk — it always falls as complexity rises, because a more flexible model can memorize more of the training set. The purple translucent sheet layered above it is the PAC bound on true risk: training risk plus a statistical penalty term. Where the two sheets pull apart is the generalization gap — the price you pay for capacity you cannot yet justify with data.

How it works

• Training risk — modeled as decaying toward a family-specific floor as complexity increases: more capacity always fits training data better.
• Generalization gap — approximated as gap ≈ √((complexity·weight + ln(2/δ)) / (2N)), in the spirit of classical VC-dimension and PAC-Bayes bounds: the gap shrinks as 1/√N and grows with model capacity and required confidence.
• True risk (bound) — training risk plus the gap. Its minimum over complexity is the theoretically "just right" model — too little capacity underfits, too much overfits the bound blows up the gap.
• The white marker rides on the current N row and shows exactly where your selected complexity sits on both sheets.

Controls

• Training samples (N) — how much labelled data you have; more data tightens the bound and pushes the best complexity higher.
• Confidence (1−δ) — how sure you want to be that the bound holds; higher confidence widens the gap.
• Model family — linear, polynomial, or neural-network capacity curves, each with a different training-risk decay rate.
• Selected complexity — moves the marker along the complexity axis to read off risk and gap at a chosen hyperparameter setting.
• Show PAC generalization bound — toggle the purple bound sheet on or off to compare against training risk alone.
• ▶ Auto-sweep complexity — animates the marker across the complexity axis so you can watch the bound sheet's U-shape emerge. Drag to rotate, scroll to zoom.

Did you know?

This is exactly why hyperparameter search early-stops on validation curves, why k-fold cross-validation exists, and why "bigger model" is not automatically "better model": statistical learning theory formalizes the bias–variance trade-off as a quantitative bound rather than a rule of thumb, and Bayesian and bandit-based hyperparameter optimizers implicitly try to search along the floor of this exact valley.