Models are trained on offline features computed in batch from a data warehouse, but served with online features computed in real time from live events. If those two pipelines compute a feature differently — a different aggregation window, a stale batch job, a code path that drifted out of sync — the model sees different inputs at inference time than it saw at training time. This is training-serving skew, and it silently degrades production accuracy.
A feature store fixes this by computing each feature once and serving identical values to both the offline (training) and online (serving) paths, instead of maintaining two separate implementations.
Uber's Michelangelo, Airbnb's Zipline, and open-source projects like Feast were all built specifically to solve training-serving skew at scale — it's one of the most common silent failure modes in production machine learning.
An interactive 3D pipeline showing how a shared feature store keeps the offline features a model trains on identical to the online features it's served in production — and how splitting those paths lets training-serving skew creep in.
Two feature computation paths — batch/offline and streaming/online — either merge into one shared store (values always match) or split apart (the offline path lags behind a batch refresh delay while the online path drifts from independent recomputation).
Toggle the shared feature store on or off, adjust batch refresh lag and serving drift rate, and pick a feature. Watch the live chart and the two value bars converge or diverge as skew appears.
Training-serving skew is one of the most common — and hardest to detect — silent failure modes in production ML, since offline evaluation metrics can look perfectly fine while the live model quietly underperforms.