XGBoost vs LightGBM: Leaf-Wise Growth and the Speed Trade-off

Why LightGBM trains faster than XGBoost on large tabular datasets, how leaf-wise tree growth differs from level-wise growth, and when each library is the better default choice.

▶ Open the simulation

The shared foundation: gradient boosting

Both XGBoost and LightGBM are implementations of gradient boosting, an ensemble technique that builds a sequence of shallow decision trees, where each new tree is trained to correct the errors left by the trees built before it. Rather than fitting a single tree to the data directly, gradient boosting fits each tree to the gradient of a loss function with respect to the current ensemble's predictions — in effect, each tree focuses on where the model is currently wrong. The final prediction is a weighted sum of all the trees' outputs. This general recipe made gradient boosting the dominant approach for tabular, structured data for years, consistently outperforming both plain decision trees and, on this kind of data, often outperforming deep neural networks as well.

XGBoost (eXtreme Gradient Boosting) and LightGBM (Light Gradient Boosting Machine, from Microsoft) both implement this recipe with production-grade engineering — regularisation to control overfitting, native handling of missing values, and support for parallel and distributed training. Where they diverge is in how each individual tree is grown, and that difference has real consequences for speed, memory use, and behaviour on different dataset shapes.

Level-wise vs leaf-wise: the core architectural difference

XGBoost's default tree-growth strategy is level-wise (also called depth-wise): at each step it expands every leaf at the current depth before moving to the next depth, so the tree grows as a roughly balanced structure, layer by layer. This keeps the tree shape predictable and tends to guard against any single branch overfitting to a narrow slice of the data, but it can waste computation splitting leaves that contribute little to reducing the loss, simply because they happen to be at the current depth.

LightGBM instead grows leaf-wise (also called best-first): at each step it finds the single leaf across the entire current tree that would reduce the loss the most if split, and splits only that leaf, regardless of its depth. This tends to reach a lower loss with fewer splits than level-wise growth, because computation is concentrated on the leaves that matter most rather than spread evenly across a level. The trade-off is that leaf-wise trees can grow deep and unbalanced quickly, which increases the risk of overfitting on smaller datasets unless the maximum depth or number of leaves is constrained explicitly — a parameter LightGBM exposes and expects users to tune.

Why LightGBM trains faster in practice

Leaf-wise growth is only part of LightGBM's speed advantage. It also uses histogram-based split finding by default, bucketing continuous feature values into a fixed number of discrete bins before searching for the best split point, which is far cheaper than XGBoost's original exact greedy method that considers every possible split value (XGBoost has since added its own histogram-based mode, narrowing this particular gap). LightGBM additionally uses techniques like Gradient-based One-Side Sampling, which keeps data points with large gradients — the ones the model is currently getting most wrong — and randomly subsamples the rest, reducing the amount of data scanned per split without discarding the most informative examples.

The practical upshot, reported consistently across benchmarks and by practitioners, is that LightGBM trains noticeably faster than XGBoost on the same data — commonly cited figures are in the range of three to five times faster — with the gap widening as dataset size grows. For rapid experimentation, hyperparameter search, or retraining pipelines that need to run frequently, this speed difference is often decisive on its own, independent of any accuracy difference between the two.

Categorical features and memory use

LightGBM also has native support for categorical features, meaning it can split directly on unordered categories rather than requiring them to be one-hot or target encoded beforehand — a real convenience on datasets with many categorical columns, since one-hot encoding a high-cardinality column can blow up the feature space and memory footprint substantially. XGBoost has added experimental categorical support in recent versions, but one-hot or ordinal encoding upstream of the model remains the more common and battle-tested approach with XGBoost.

Memory efficiency follows a similar pattern: LightGBM's histogram binning and its use of Exclusive Feature Bundling, which merges sparse mutually-exclusive features into single bundles, tend to give it a lower memory footprint on wide, sparse datasets — a genuine advantage when working with very large tabular datasets on constrained hardware.

When to reach for which

On small to medium datasets, the practical difference between the two often shrinks to a matter of a percent or two in accuracy, and the choice comes down to secondary factors — tooling maturity, familiarity, or ecosystem integrations. XGBoost has been in production longer, has broader library support across languages and platforms, and its level-wise growth tends to be more forgiving of default settings on smaller datasets, since it is less prone to the aggressive overfitting that unconstrained leaf-wise growth can produce when there is not much data to support a deep tree.

LightGBM tends to be favoured for large datasets, high-cardinality categorical features, and workflows where training speed itself is a constraint — frequent retraining, large-scale hyperparameter search, or systems where a model must be refreshed on a tight schedule. In practice, many teams train both on a given dataset with comparable tuning effort and let cross-validation performance make the final call, since the theoretical trade-offs described here do not always predict which library will win on a specific dataset.

Frequently Asked Questions

Does leaf-wise growth always produce a more accurate model than level-wise growth?

Not always. Leaf-wise growth can reach a lower training loss for a given number of splits, but on smaller datasets it is more prone to overfitting because the resulting trees can be deep and unbalanced. Level-wise growth is a more conservative default that spreads splits more evenly, which sometimes generalises better when data is limited.

Do I need to encode categorical variables before using LightGBM?

No, not necessarily. LightGBM can take categorical columns directly and find splits on the unordered categories natively, avoiding the feature-space explosion that one-hot encoding can cause on high-cardinality columns. XGBoost has more limited native categorical support and is more commonly used with pre-encoded categories.

How much faster is LightGBM than XGBoost in practice?

Reported speedups are commonly in the range of three to five times faster training on the same dataset, with the gap tending to widen as dataset size increases, largely due to LightGBM's histogram-based splitting and gradient-based sampling of training examples.

Can I use both algorithms' feature importance for the same purposes?

Yes, both provide feature importance scores derived from how much each feature contributed to reducing the loss across all trees, and both are commonly used to explain predictions to non-technical stakeholders or to guide feature selection, though the exact numeric values are not directly comparable between the two libraries.

Is one of these two algorithms simply obsolete compared to the other?

No, both remain actively maintained and widely used in production. The choice is a genuine trade-off between training speed and memory efficiency (favouring LightGBM, especially at scale) versus tooling maturity and slightly more forgiving defaults on smaller datasets (favouring XGBoost), not a case of one clearly superseding the other.