A deployed model is trained on a fixed reference distribution (shown as the pale blue point cloud). In production, new records stream in continuously — the live feed shown as falling spheres. Over time the real world can drift away from what the model learned in two distinct ways: data drift (the input feature distribution shifts) and concept drift (the true relationship between inputs and the correct label changes, even if the inputs look similar).
Population Stability Index (PSI) and Kolmogorov–Smirnov tests are common production techniques for catching data drift on unlabeled traffic, since ground truth labels for computing real accuracy are often delayed by days or weeks.
An interactive 3D data-distribution plot: a live production stream flows in next to a frozen training reference, and shifting it over simulated time triggers a drift-detection alert before the model's rolling accuracy visibly degrades.
Data drift (shifting input distribution) and concept drift (a changing input-output relationship) are distinct failure modes. A statistical drift score reacts to distribution shift immediately, while accuracy only drops once the true and model decision boundaries genuinely disagree.
Increase data drift magnitude to shift the incoming stream, and concept drift angle to rotate the true decision boundary away from the model's frozen one. Watch the drift score cross its alert threshold, then compare how long it takes rolling accuracy to catch up.
Because ground-truth labels are often delayed in production, most MLOps teams monitor distribution statistics like PSI or KS-distance as an early-warning proxy long before a true accuracy metric is even computable.