A model is trained on a baseline feature distribution (grey towers). In production the live distribution (teal towers) drifts: user behaviour, a new market, seasonality. Samples arrive from the data silo as glowing orbs and land on the bucket they belong to. The Population Stability Index compares both histograms bucket by bucket and drives the dial.
PSI = sum_i (live_i - base_i) * ln(live_i / base_i)
accuracy = acc0 - k * max(0, PSI - 0.06) (acc0 = 94 %, k = 0.55)
- Same model as the 2D twin - 24 buckets, baseline Gaussian (mean 0.42, sd 0.16), live mean random-walking away at the drift rate, retrain rebuilds the baseline from live data.
- PSI is a symmetric KL divergence (Jeffreys divergence) of the two bucket distributions. For two Gaussians it has a closed form; the check below uses a continuous-density quadrature instead, which also handles the clipped edges.
- Rule of thumb - PSI below 0.1 stable, 0.1 to 0.25 watch, above 0.25 usually retrain. The gauge bands show exactly that.
- Honest simplifications - one numeric feature, a Gaussian shape, and an accuracy loss that is a fitted linear law, not a measurement of a real model. Real monitoring tracks many features and the label delay.