Traditional software monitoring asks "is the server up, and is it fast." A deployed machine learning model can pass every one of those checks while quietly making worse and worse predictions, because nothing about a wrong prediction necessarily throws an error. Model monitoring is the discipline of watching signals that are specific to model behaviour, not just infrastructure health.
Three signals that matter
Prediction confidence tracks how sure the model is about its own outputs; a sustained drop often signals the model has drifted outside the range it was trained for. Input distribution shift compares live incoming data against the training data's statistics, catching upstream changes, a new customer segment, a changed data format, before they visibly break predictions. Error rate is the most direct signal, but usually the slowest to arrive, since it requires ground truth labels that often lag behind predictions by days or weeks.
Why alerts need tuning
Set an alert threshold too sensitive and a team drowns in false alarms from ordinary noise, until they start ignoring the channel entirely. Set it too loose and a real incident runs for days before anyone notices. Getting this threshold right is itself an engineering decision, one that depends on how costly a missed incident is versus how disruptive a false alarm is.
Try it yourself
The Model Monitoring Lab simulates a real incident partway through a 60-step timeline and lets you scrub through it, watching which of three monitored signals catches it, and how much of the incident an unmonitored system would have missed entirely.
🧪 Try it yourself: the Model Monitoring Lab simulation lets you experiment with everything described above directly in your browser.