AutoML automates the parts of the ML pipeline that used to require manual trial and error: choosing hyperparameters, sometimes even model architecture itself. At its core, most of this comes down to a search problem -- explore a space of configurations, evaluate each one, and converge on a good setting within a limited trial budget.
Three search strategies
Grid search
Exhaustively evaluate every combination on a fixed, regular lattice over the hyperparameter ranges. Simple and reproducible, but wastes trials on combinations the search has no reason to think are promising, and its resolution is a step function of trial budget -- adding a few extra trials often changes nothing until the budget crosses a threshold that supports a finer lattice.
Random search
Sample configurations uniformly at random. Counter-intuitively often outperforms grid search at the same budget, especially when only a few hyperparameters actually matter -- random sampling explores the important dimensions more densely by chance than a lattice does by construction.
Bayesian optimisation
Build a probabilistic model of the objective function from trials so far, then choose each next trial to balance exploiting known good regions against exploring uncertain ones. Typically the most sample-efficient of the three on smooth objective surfaces, at the cost of needing an initial exploration phase and more implementation complexity (Optuna and Hyperopt are common implementations).
💡 Key idea: a fixed regular lattice can walk straight past a narrow region of good performance between grid points -- no amount of additional trials fixes this if the lattice spacing stays coarser than the region's width.
Why grid search degrades fastest as dimensions grow
Grid search's trial count grows exponentially with the number of hyperparameters being tuned -- a 5-point grid over 2 parameters is 25 trials, over 4 parameters it's 625. Random and Bayesian search don't have this exponential lattice constraint, which is a large part of why grid search is rarely used beyond two or three hyperparameters in practice.
Choosing a strategy in practice
- Grid search: reasonable only for a small number of hyperparameters with a modest number of values each, where exhaustive coverage is affordable.
- Random search: a strong, simple default, especially early on or when trials are cheap.
- Bayesian optimisation: worth the added complexity when each trial (each full model training run) is expensive enough that sample efficiency matters more than implementation simplicity.
🧪 Try it yourself: the AutoML Lab simulation lets you experiment with everything described above directly in your browser.