Random Search vs Bayesian Optimisation: Tuning XGBoost on a Bank Marketing Dataset

A practical comparison of RandomizedSearchCV and Hyperopt's tree-structured Parzen estimator for tuning a gradient-boosted model, using a bank deposit-response prediction task as the working example.

▶ Open the simulation

Four models, one baseline problem

When building a model to predict whether a bank customer will subscribe to a term deposit after a marketing call, it is standard practice to compare several algorithm families before committing to one. A typical progression starts with logistic regression as an interpretable baseline, adds k-nearest neighbours and a single decision tree as simple non-linear alternatives, and finishes with a gradient-boosted ensemble such as XGBoost, which combines many shallow decision trees, each one trained to correct the errors of the ones before it. On tabular data with a mix of numeric and categorical features like this one, gradient boosting almost always wins on ranking metrics, which is why it becomes the natural target for further hyperparameter tuning once the baseline comparison is done.

Why default hyperparameters are rarely good enough

A gradient-boosted model has many settings that jointly control how it fits: the number of trees, the maximum depth of each tree, the learning rate that scales each tree's contribution, the fraction of rows and columns sampled per tree, and regularisation terms that penalise overly complex trees. Left at their defaults, these settings represent a reasonable generic compromise, not something tuned to a specific dataset's size, noise level, and class balance. Search over this space is what turns a merely adequate model into a genuinely strong one, and it is why a tuned XGBoost model typically outperforms an untuned one by a noticeable margin on validation ROC-AUC even though the underlying algorithm hasn't changed at all.

Random search: cheap, parallel, and surprisingly competitive

RandomizedSearchCV, part of scikit-learn, works by sampling a fixed number of hyperparameter combinations at random from specified distributions, training and cross-validating a model for each, and keeping the best. A common setup draws 50 random combinations and evaluates each with stratified k-fold cross-validation (commonly 3 folds for speed), scoring by ROC-AUC to stay robust to the dataset's roughly 88/12 class imbalance. The appeal of random search is structural: every trial is independent of every other trial, so the whole search parallelises trivially across cores or machines, and unlike an exhaustive grid search, it does not waste evaluations exploring every combination of a coarse grid — a classic finding from the hyperparameter-tuning literature is that for a fixed compute budget, random sampling tends to explore the space more efficiently than a grid, because most hyperparameters matter far less than a few dominant ones, and random sampling naturally spreads the trial budget across the important dimensions instead of wasting draws on redundant fine-grained combinations of unimportant ones.

Bayesian optimisation: learning from every previous trial

Hyperopt takes a different approach built on Bayesian optimisation with a tree-structured Parzen estimator (TPE). Rather than sampling blindly, TPE builds a probabilistic model of which regions of hyperparameter space tend to produce good scores based on the trials it has already run, then proposes the next combination to try in a region likely to improve on the best result so far, balancing that exploitation against continued exploration of less-tested areas. Given the same 50-evaluation budget as the random search, TPE-based search typically converges toward a better final score, because later trials are informed by earlier ones instead of being drawn independently. The tradeoff is that this sequential, information-using process is inherently harder to parallelise than random search, since each new suggestion depends on the results of prior trials (though batched and asynchronous variants exist to soften this constraint), and it introduces its own overhead in modelling the search history.

Comparing the two in practice

On a task like predicting term-deposit subscription, both methods are usually run under an identical evaluation budget and cross-validation scheme so that any difference in the final score can be attributed to the search strategy rather than to unequal effort. In practice, random search establishes a solid, cheap floor with minimal setup, while Bayesian/TPE search tends to close the remaining gap toward the best achievable ROC-AUC and F1-score, especially as the number of tunable hyperparameters and the sensitivity between them grows. Because gradient boosting hyperparameters interact — for instance, a lower learning rate generally wants a higher number of trees to compensate — a search method that can learn correlations between promising values is naturally suited to this kind of dependency, which is exactly the theoretical advantage TPE has over independent random draws.

Practical guidance for choosing between them

Neither method is uniformly the right default. Random search is the sensible first move when compute is cheap and parallel, when the hyperparameter space is small enough that broad coverage matters more than adaptive refinement, or when a quick, reproducible baseline is needed before investing more engineering effort. Bayesian optimisation earns its extra complexity when each training run is expensive enough that squeezing more value out of a fixed number of trials matters, when the hyperparameter space is large or highly interactive, or when the last few percentage points of a ranking metric like ROC-AUC translate into real business value, as they often do in a marketing-response setting where the model's job is to prioritise a limited pool of outbound calls.

Frequently Asked Questions

What is the tree-structured Parzen estimator (TPE)?

TPE is a Bayesian optimisation algorithm used by libraries like Hyperopt. It models the distribution of hyperparameters that led to good versus poor results in past trials, and uses that model to propose the next combination most likely to improve on the current best score, rather than sampling combinations independently at random.

Is random search ever better than Bayesian optimisation?

Yes, particularly when the hyperparameter space is small, when trials are cheap and can be run in large parallel batches, or when the search budget is so limited that the overhead of building a probabilistic model outweighs the benefit of learning from past trials.

Why use stratified k-fold cross-validation during hyperparameter search on this kind of dataset?

Because the target class is imbalanced (roughly 88% no versus 12% yes), plain k-fold splits could accidentally create folds with very few positive examples. Stratified k-fold preserves the original class ratio in every fold, giving a more reliable estimate of how each hyperparameter combination performs.

Why score the search by ROC-AUC instead of accuracy?

With an imbalanced target, a model that predicts the majority class for everything can score high accuracy while being useless. ROC-AUC measures how well the model ranks positive cases above negative ones across all thresholds, which is far more informative than accuracy for this kind of problem.

Does hyperparameter tuning matter more than choosing the right algorithm?

Usually not as much. Comparisons on this kind of tabular marketing-response data typically show gradient boosting outperforming simpler models like logistic regression or k-nearest neighbours before any tuning is even applied. Tuning provides a further, smaller improvement on top of choosing a strong algorithm, rather than substituting for it.