Optuna searches for hyperparameters by repeatedly proposing a trial, evaluating a
model's loss, and using that history to propose smarter trials next time. This scene
renders a synthetic two-parameter loss surface — like plotting validation loss against
learning_rate and max_depth — as a purple 3D landscape. Low
points (valleys) are good hyperparameter combinations; high ridges are bad ones.
optuna.pruners.MedianPruner: trials trending worse than the running median are stopped early (shown as a small red marker) instead of running to completion, saving compute.
Optuna's TPE sampler is a Bayesian optimization strategy: instead of directly modelling
P(loss | params) like Gaussian-process methods, it models
P(params | good) and P(params | bad) and picks the point that
maximizes their ratio — a trick that scales cheaply to many hyperparameters.
A synthetic two-parameter loss surface rendered as a 3D purple landscape, where trial markers rain down and settle into valleys as a Tree-structured Parzen Estimator (TPE) — the sampler behind Optuna — learns where to search next.
Each falling sphere is one Optuna trial. Under TPE, new trials cluster near the best valley found so far instead of scattering uniformly, and weak trials can be pruned early — exactly the mechanics that make Optuna converge faster than grid or random search.
Pick TPE or random search, adjust γ to shift between exploitation and exploration, toggle pruning, and watch the best-loss beam track the study's optimum. Generate a new landscape to see the sampler adapt to a different loss surface.
Optuna's default sampler models P(params | good) and P(params | bad) separately, then proposes the candidate that maximizes their ratio — a cheap approximation to full Bayesian optimization that scales to dozens of hyperparameters.