💳 Loan Default Predictor — Gradient Boosted Trees Live

Each new tree is fit specifically to reduce the residual error on the training set, so training log-loss decreases almost monotonically as trees are added. The validation set was never used to fit any tree, so its loss only improves as long as the patterns the trees are learning genuinely generalize. Once the ensemble starts fitting noise specific to the training borrowers rather than the true default-risk relationship, validation loss stops improving or creeps back up — this gap is the classic signature of overfitting, and is exactly why boosting round count, tree depth and learning rate are tuned against a held-out validation set rather than the training set itself.

The learning rate scales down how much of each new tree's fitted correction is actually added to the running prediction. A small learning rate (e.g. 0.05) means many trees are needed to reach a good fit, but each one only nudges the model slightly, which tends to generalize better and reduces the risk of any single noisy tree throwing the ensemble off course. A large learning rate (close to 1.0) lets the model fit the training data very quickly, often in just a handful of rounds, but it is far more prone to overshooting and overfitting, since early trees lock in large corrections before later trees can smooth them out.

A single deep tree can memorize the training set almost perfectly by carving it into tiny, homogeneous regions, which generalizes poorly. Gradient boosting instead uses many shallow, high-bias, low-variance trees (often called "stumps" at depth 1) and lets the boosting process itself supply the complexity, one small correction at a time. A depth-1 stump can only ever capture a single feature interaction per tree, but summed over dozens or hundreds of boosting rounds, the ensemble can represent very complex, high-order feature interactions while each individual tree stays simple and resistant to overfitting.

A random forest builds many deep trees independently and in parallel on bootstrapped samples of the data, then averages their predictions — this reduces variance (noise) but each tree is trained without knowledge of the others' errors. Gradient boosting builds trees sequentially, and every new tree is explicitly trained to correct the specific errors the current ensemble is making. This sequential error-correction is what lets boosting often reach lower bias and higher accuracy than a forest of comparable size, at the cost of being more sensitive to the learning rate and number of rounds, and more prone to overfitting if left unchecked.

The synthetic borrower generator bakes in realistic relationships used by real credit-risk models: higher income and longer employment history reduce default risk, while a higher debt-to-income ratio, higher credit-card utilization, and more late payments in the past two years increase it; a longer credit history also modestly reduces risk by giving more track record to assess. These relationships are combined into a true log-odds of default plus irreducible random noise, then a coin flip weighted by that probability generates the observed default label — so even a perfect model cannot reach zero loss, exactly as in real lending data.

AUC (area under the ROC curve) measures how well the model ranks borrowers who actually defaulted above those who did not, regardless of any particular probability threshold. An AUC of 0.5 means the model ranks pairs no better than a coin flip; an AUC of 1.0 means every defaulter is ranked above every non-defaulter. It is computed here directly from its rank-based definition: for every defaulter/non-defaulter pair in the validation set, count whether the defaulter received a higher predicted probability, and average over all pairs.