Gradient boosting builds an ensemble one weak learner at a time. Starting from the constant that minimizes squared error — the mean of the targets — each round fits a new shallow regression tree to the current residuals, then adds a shrunk copy of that tree to the running prediction:
F_0(x) = mean(y)
r_i = y_i − F_{m-1}(x_i) (residual = negative gradient of ½·SSE)
h_m(x) = regression tree fit to (x_i, r_i), minimizing Σ(r_i − ŷ_leaf)²
F_m(x) = F_{m-1}(x) + ν · h_m(x) (ν = shrinkage / learning rate)
Every leaf value is the mean residual of the points that land in it — the exact split-and-average rule real regression trees use, found here by an actual greedy SSE-minimizing search over candidate thresholds, not a canned formula. Because each new tree only has to explain what previous trees got wrong, a handful of very shallow (depth 1–3) "weak learners" combine into a strong, smooth approximation.
- Shrinkage ν — how much of each new tree's correction is applied per round. Small ν needs more rounds but generalizes better; ν = 1 fits fastest but overfits noise sooner.
- Tree depth — how many splits each new weak learner may use; depth 1 ("stumps") is the classic boosting weak learner, deeper trees fit faster but risk overfitting individual points.
- Dataset noise — how much random scatter is added to the true curve before fitting; watch the fit start tracking the noise itself once rounds outrun the model's capacity to generalize.
- The translucent vertical walls mark the x-thresholds the most recent tree split on — the actual decision boundaries a real gradient-boosted model (XGBoost, LightGBM, scikit-learn's GradientBoostingRegressor) would report.