MLOps Data Drift Monitor | mysimulator.uk

Production Model

Drift rate 0.4

PSI alert threshold 0.25

Retrain cooldown 6 s

PSI (drift)

–

Accuracy

–

Model age

–

Retrains

–

How it works

A model is trained once on a baseline feature distribution (the ghosted histogram). In production the real-world distribution slowly drifts away from that baseline — user behaviour changes, a new market segment appears, seasonality shifts. Every tick, the Population Stability Index compares the live histogram against the baseline bucket by bucket.

PSI = Σᵢ (liveᵢ − baseᵢ) · ln(liveᵢ / baseᵢ)
accuracy ≈ accuracy₀ − k · max(0, PSI − PSI₀)

Drift rate — how fast the live distribution's mean walks away from the training baseline each second.
PSI alert threshold — the industry rule of thumb: PSI < 0.1 is stable, 0.1–0.25 is a watch zone, > 0.25 usually means retrain. Lower it to alert earlier.
Retrain cooldown — minimum time between automatic retrains, so the pipeline doesn't thrash on noisy data.
Auto-retrain — when on, crossing the threshold snaps the live distribution back onto a fresh baseline and accuracy recovers, exactly like a scheduled or drift-triggered retraining job would in a real pipeline.

Real-world relevance: PSI (or the related KS-test) is exactly what monitoring tools like Evidently, WhyLabs or a hand-rolled Airflow DAG compute on every batch of production traffic — it is the signal that turns "the model just feels worse lately" into an automated retraining trigger.

↻ RETRAINING…

Baseline (training) distribution Live (production) distribution Accuracy curve