🎯 Hyperparameter Tuning: Core Concepts

About this simulation

The left surface is a synthetic loss landscape over two hyperparameters — think learning rate on one axis, batch size or dropout on the other. A search strategy drops trial points onto it, each one a full train + validate cycle. The right chart tracks two lines as the search progresses: the best validation loss found so far, and the true generalization loss of that same winning configuration.

How it works

• Grid search lays trials on a fixed lattice — it spends equal effort on every axis, even ones that barely matter.
• Random search samples uniformly at random. With a fixed budget it explores far more distinct values of the axis that actually matters, which is why it usually beats grid search in practice (Bergstra & Bengio, 2012).
• 2nd hyperparameter's importance controls how much the second axis actually affects loss. At 0% the landscape is a ridge — only one hyperparameter matters, and grid search visibly wastes trials.
• Validation noise simulates a small or noisy validation split (e.g. few cross-validation folds): the score you observe is the true loss plus random error.
• Because you pick the "best" trial by its noisy validation score, the winner is often one that simply got a lucky noise draw — its true loss is usually a bit worse. That gap is the optimism gap, and it grows with more trials and more noise: this is validation-set overfitting from too much searching.

Controls

• Search strategy — grid vs. random sampling of the search space.
• Search budget — how many configurations get evaluated.
• 2nd hyperparameter's importance — reshapes the landscape from a ridge (one axis matters) to a bowl (both matter).
• Validation noise — how much the observed score can mislead you.
• ▶ Run search — regenerates and replays the trial sequence with current settings.

Did you know?

This is why practitioners hold out a separate test set that is never used to choose hyperparameters: with enough trials against the same validation set, you can "search your way" to a low validation score that doesn't generalize — the exact multiple-comparisons pitfall this optimism gap visualizes.