The weak learner: a decision stump
A decision stump is the simplest possible classifier: a single threshold on a single feature. Against a genuinely curved true boundary, no single axis-aligned threshold can do much better than chance in the regions where that boundary cuts across straight lines — a stump is, by design, a weak learner.
Bagging: reduce variance by averaging
Bagging (bootstrap aggregating) trains many stumps independently, each on a random resample of the training data, and combines their votes with equal weight. Because each stump's errors come from the particular quirks of its own resample, averaging across many of them cancels out much of that idiosyncratic noise — reducing variance without needing any single stump to be individually strong.
Boosting: correct mistakes sequentially
Boosting trains stumps sequentially instead of independently. Each new stump is trained with extra weight on the points the previous ensemble got wrong, and the final combination weights each stump by how well it performed. Rather than passively averaging away errors, boosting actively targets the ensemble's remaining blind spots at every step.
Why ensembles dominate tabular machine learning
Gradient-boosted trees (XGBoost, LightGBM, CatBoost) and random forests remain the dominant approach for tabular data in production ML, ahead of deep learning in most such settings. They handle mixed feature types, missing values, and nonlinear feature interactions robustly with comparatively little tuning, and the underlying principle — combine many weak, cheap, diverse models rather than search for one strong, expensive, complex one — keeps winning where a curved or irregular decision boundary would defeat any single simple model.
🧪 Try it yourself: the Ensemble Methods Lab simulation lets you experiment with everything described above directly in your browser.