Every hyperparameter search tool — Optuna, Ray Tune, scikit-learn's
GridSearchCV, Hyperopt — is ultimately hunting for the lowest point
on a "loss landscape": a surface where the two horizontal axes are hyperparameters
(say, learning rate and regularization strength) and height is validation loss.
This simulation renders that landscape in 3D and drops search "trials" onto it
using four real strategies, so you can see why smarter samplers reach the valley
in fewer tries.
Optuna's default sampler is Tree-structured Parzen Estimator (TPE), a Bayesian
method that models p(hyperparameters | good) and
p(hyperparameters | bad) separately, rather than a single surrogate —
which makes it cheap enough to run thousands of trials in pure Python.
A 3D hyperparameter loss landscape where four real tuning strategies — Grid Search, Random Search, Optuna-style Bayesian TPE and Ray Tune's ASHA early-stopping — race to find the lowest valley.
The terrain's height is validation loss over two hyperparameters. Each strategy samples that surface differently: exhaustive grids, uniform randomness, probability-guided Bayesian sampling, or parallel trials pruned early if they underperform.
Pick an algorithm, set trial speed, ruggedness and trial budget, then watch markers land on the surface. The glowing beacon marks the best loss found so far; ASHA trials that get pruned fade out mid-search.
Optuna's default TPE sampler and Ray Tune's ASHA scheduler are both designed to spend compute where it matters most — which is why modern AutoML pipelines rarely use plain grid search once the search space grows past a few dimensions.