Hyperparameter Optimization Algorithms (2D)

Search Setup

Search algorithm

Evaluation budget 64

Search speed 5

Loss surface ruggedness 45%

Exploration vs exploitation 50%

Live Readout

Trials run

0 / 64

Best val. loss

—

Best learning rate

—

Best reg. strength

—

Trials to best-5%

—

FPS

—

How it works

The heatmap is a validation-loss surface over two normalized hyperparameters — learning rate (x-axis) and regularization strength (y-axis) — generated from a smooth convex bowl plus a seeded ripple of sine terms whose amplitude follows the ruggedness slider. Cyan is low loss, red is high loss.

Grid search — lays a regular lattice across the space and evaluates every point in order. Exhaustive but easy to step over a narrow optimum.
Random search — samples uniformly at random; often reaches a good region faster than grid search since real surfaces rarely depend equally on every dimension.
Bayesian optimization — scores random candidate points by a distance-weighted prediction of nearby trial losses (exploit) plus a bonus for distance from any sampled point (explore), and evaluates whichever candidate scores best. The exploration slider directly weights that bonus.
Trials to best-5% — how many evaluations the running search needed before it first landed within 5% of the best loss it will eventually find; a smaller number means the algorithm converged faster on this surface.

Bergstra & Bengio (2012) showed random search often outperforms grid search at equal budget, because most real loss surfaces are dominated by only a few truly important dimensions.

↑ regularization

learning rate →

Early trial Late trial Best so far