Tuning a gradient-boosted model like XGBoost means searching a landscape of
hyperparameter combinations — here simplified to two axes, learning_rate
and max_depth — for the peak that gives the best validation AUC. Two
identical terrains sit side by side: the same underlying scoring surface, searched
by two different strategies at once.
On real tuning problems, TPE-style Bayesian optimisation (as used by Hyperopt) typically reaches a near-optimal validation score in a fraction of the trials that uniform random search needs, because every new candidate is informed by everything tried so far rather than sampled blind.
Two identical hyperparameter landscapes sit side by side, one searched by uniform Random Search, the other by a simplified Tree-structured Parzen Estimator (Bayesian TPE) — watch which strategy climbs to the peak validation score faster.
Random Search samples blindly every trial. Bayesian TPE models which regions of past trials scored well versus poorly, then samples new candidates that maximise the good/bad density ratio — biasing later trials toward the peak.
Set a trial budget and TPE exploitation strength, then press Autoplay (or Step) to run both strategies trial-by-trial. Compare the best AUC found by each and watch how quickly TPE's blue points cluster near the summit.
Hyperopt's TPE sampler underlies many production AutoML pipelines precisely because it usually needs far fewer trials than grid or random search to reach a near-optimal gradient-boosted model configuration.