🔍 Scikit-Learn Hyperparameter Optimization Implementation

About this simulation

GridSearchCV and RandomizedSearchCV both search the same space of hyperparameter combinations, cross-validating each candidate to estimate how well it generalizes. This scene turns that abstract search into a landscape: the hilltops are combinations of two hyperparameters (learning rate and max_depth-style regularization) that generalize well, and the low plains are combinations that underfit or overfit.

How it works

• GridSearchCV lays candidates on a fixed, evenly spaced grid and evaluates every single one — exhaustive but the cost grows exponentially with each added parameter.
• RandomizedSearchCV draws the same number of candidates from random positions in the space, so it often stumbles onto a comparably good peak with far fewer evaluations when only a few directions actually matter.
• Each glowing point is one cross-validated candidate; height and colour show its mean CV score. k-fold cross-validation is simulated by adding random per-fold jitter to the true score — more folds average that jitter away, giving a steadier estimate.
• A purple beam marks the best combination found so far — scikit-learn exposes this as .best_params_ and .best_score_ after .fit().

Controls

• Search strategy — switch between exhaustive grid search and randomized sampling.
• Search budget — total candidates evaluated (grid search rounds this to a perfect square, like an n×n param grid).
• CV folds (k) — more folds reduce the noise in each score estimate, at the cost of more model fits per candidate.
• Search speed — how fast candidates are evaluated on screen.
• Show true score surface — toggle the underlying generalization landscape (unknown to the search in real life, shown here for intuition).
• ▶ Run search — restarts the search animation with current settings. Drag to rotate, scroll to zoom.

Did you know?

Bergstra & Bengio (2012) showed that random search often outperforms grid search under a fixed compute budget precisely because most hyperparameters barely affect the outcome — random sampling explores the few important dimensions far more densely than an evenly-spaced grid ever can.