Decision Trees and Random Forests: From a Single Tree to a Forest

A model made of yes-or-no questions

A decision tree makes predictions by asking a sequence of simple questions about the input data, each one splitting the remaining possibilities in two, until it arrives at a final answer. Structurally, it looks like an upside-down tree: a single root node at the top containing the full dataset, a series of internal nodes where the data is split based on a feature ("Is age over 40?", "Is income above £30,000?"), and leaf nodes at the bottom that hold the final prediction — a class label for classification, or a numeric value for regression.

This structure is what makes decision trees unusually easy to interpret compared with most other machine learning models: you can trace the exact sequence of questions that led to any individual prediction, which matters in domains like credit approval or medical triage where a human needs to be able to explain a decision, not just trust a number.

How the tree decides what to ask

Building a tree from data is a recursive process. At each node, the algorithm considers every available feature and every possible splitting threshold, and picks the single split that does the best job of separating the data into purer subsets — subsets where the examples increasingly agree on the outcome. "Purity" is measured with one of two related metrics:

• Gini impurity measures how often a randomly chosen point would be misclassified if you labeled it according to the distribution of classes in that node. A node containing only one class has zero impurity; a node split evenly between two classes has maximum impurity.
• Entropy and information gain come from information theory and measure the same underlying idea — how much "disorder" is present in a node, and how much a proposed split would reduce it.

Both metrics tend to produce similar trees in practice. The algorithm greedily picks whichever split most reduces impurity, then repeats the process independently on each resulting branch, recursing deeper and deeper until a stopping condition is met — a maximum depth, a minimum number of samples required in a node, or a node that is already pure.

Why a single tree overfits so easily

Left unconstrained, a decision tree will keep splitting until every leaf contains a single training example, achieving perfect accuracy on the data it was trained on. This is a textbook case of overfitting: the tree has not learned the underlying pattern in the data, it has simply memorised the noise, quirks, and coincidences specific to that particular training set. Such a tree will generalise poorly to new data it hasn't seen. Decision trees are also notoriously unstable — because each split is chosen greedily based on the exact data available, a small change to the training set (removing a handful of examples, say) can cause an entirely different sequence of splits to be chosen further down the tree, producing a structurally very different model.

Common remedies include pruning the tree back after it has grown (removing branches that provide little predictive value), or constraining growth up front by limiting the maximum depth, requiring a minimum number of samples per leaf, or requiring a minimum improvement in purity before allowing a split. These help, but a single tree remains a fairly high-variance, unstable model — which sets up the case for combining many of them together.

Random forests: strength through diversity

A random forest is an ensemble of many decision trees, each trained slightly differently, whose predictions are combined by majority vote (for classification) or averaging (for regression). The key insight is that if individual trees make different, somewhat uncorrelated mistakes, those mistakes will tend to cancel out when averaged, even though no individual tree is very good on its own. Two sources of randomness create that diversity:

• Bootstrap sampling (bagging): each tree is trained not on the full dataset, but on a random sample of the same size, drawn with replacement, so some examples appear multiple times and others not at all. This means every tree sees a slightly different version of the training data.
• Random feature selection: at every single split within every tree, the algorithm considers only a random subset of the available features rather than all of them. This prevents a handful of strongly predictive features from dominating every tree in the forest, forcing trees to discover a wider variety of useful splits and reducing the correlation between trees.

The combined effect is a model that trades away some of a single tree's interpretability in exchange for dramatically better accuracy and stability. Random forests also naturally rank features by importance — by measuring how much each feature reduces impurity on average across all the trees and splits where it was used — providing a useful, if approximate, window into which inputs the model actually relies on.

Random forests versus gradient boosting

Random forests build their trees independently and in parallel, then average the results — a strategy called bagging, which primarily reduces variance (overfitting to noise). Gradient boosting (implemented in popular libraries such as XGBoost, LightGBM, and CatBoost) takes the opposite strategy: it builds trees one at a time, sequentially, where each new tree is deliberately trained to correct the mistakes — technically, the residual errors — left behind by the trees built before it. This sequential correction process primarily reduces bias (underfitting), and gradient-boosted models frequently achieve the best raw accuracy of any technique on structured, tabular data, which is why they dominate machine learning competitions on that kind of data. The trade-off is that boosted trees are more sensitive to hyperparameter choices, slower to train because trees cannot be built in parallel, and slightly more prone to overfitting if not carefully regularised, compared with the relatively "turnkey" reliability of a random forest.