Feature Stores: Closing the Gap Between Training and Serving in ML Pipelines
Why a feature computed offline for model training can silently differ from the same feature computed online at inference time, and how a feature store is designed specifically to close that gap.
A bug that only shows up in production
Training-serving skew is one of the most insidious classes of bug in applied machine learning precisely because it is invisible during development: the model trains fine, offline evaluation metrics look good, and the failure only surfaces once real traffic hits the live endpoint and predictions turn out systematically worse than the offline numbers promised. The root cause is almost always that the feature values the model saw during training were computed by a different code path, running on a different data source, at a different point in time relative to the event being predicted, than the feature values it sees at inference.
A concrete and common example: a fraud model is trained with a feature "average transaction amount over the last 30 days," computed by a batch SQL job that runs nightly over the full transaction warehouse, using clean, deduplicated, fully-settled data. In production, the same feature is computed by an online service that queries a live transactional database at request time, working from data that may include not-yet-settled transactions, may be missing the most recent few minutes due to replication lag, and may define "last 30 days" relative to a slightly different reference timestamp than the batch job used. Both computations are reasonable implementations of "average transaction amount over the last 30 days" in isolation, but they are not the same function, and the model, having learned its weights against the batch version's particular quirks and biases, receives systematically different inputs in production than it was trained to expect — degrading performance in a way that no amount of offline model tuning can fix, because the bug is not in the model at all.
Why this happens structurally, not just as a coding mistake
It is tempting to treat training-serving skew as simply a case of insufficiently careful engineers writing two versions of the same logic, and code review discipline alone should fix it. In practice it recurs across well-run teams because the two computation contexts have genuinely different constraints that push toward different implementations even when engineers are trying hard to keep them aligned. Offline training feature computation typically runs as a batch job over a data warehouse, optimised for throughput over historical data, often written in Spark or SQL, with the luxury of full table scans and complex joins across large historical windows. Online serving feature computation typically runs under a tight latency budget (often single-digit milliseconds), against operational databases or caches rather than the warehouse, frequently rewritten in a different language for performance, and under pressure to approximate or simplify any computation that would be too slow to run per-request.
Point-in-time correctness is the subtler and more dangerous version of the same problem. When building a training set from historical data, a feature for a given training example must be computed using only information that would genuinely have been available at that historical prediction time — not information that arrived later and got backfilled into the warehouse. A naive batch join that simply pulls "current" feature values for historical examples introduces label leakage: the model trains on information from the future relative to the event it is predicting, produces excellent offline metrics, and then fails in production where that future information is, correctly, not yet available. This is a distinct failure from the online/offline logic mismatch described above, but it compounds the same underlying problem — that training and serving need the exact same feature value as of the exact same logical timestamp, and building that alignment by hand, pipeline by pipeline, is where teams consistently get it wrong.
What a feature store actually does
A feature store's core architectural idea is to make feature computation logic a single, versioned, shared artifact rather than something reimplemented per pipeline, and to split storage into two synchronised layers that serve the two different access patterns without duplicating the definition of what the feature means. An offline store (typically built on a data warehouse or lakehouse like BigQuery, Snowflake or a Parquet-based lake) holds the full historical time series of feature values, optimised for the large, point-in-time-correct joins that training set construction needs. An online store (typically a low-latency key-value store like Redis or DynamoDB) holds only the latest value of each feature per entity, optimised for the millisecond-scale single-row lookups that live inference needs. Crucially, both stores are populated by the same feature transformation definition, usually expressed once in a declarative form (a SQL expression or a Python transformation function registered with the feature store) and then materialised into both places by the store's own pipeline, so the batch and streaming computation paths are generated from one source of truth rather than hand-written twice by two different teams.
This single-definition architecture is what actually eliminates training-serving skew, as distinct from merely making it easier to spot: if there is only one place the feature logic is written, there is no second implementation to drift out of sync with the first. Modern feature stores (Feast, Tecton, Databricks Feature Store, SageMaker Feature Store) also solve the point-in-time correctness problem directly, by supporting "time-travel" joins that, when building a training set, automatically fetch each feature's value as it stood at each training example's specific event timestamp rather than the current value, which is exactly the leakage-preventing join that teams previously had to hand-build and frequently got subtly wrong.
What a feature store does not solve, and when it is worth the overhead
A feature store is infrastructure, not a guarantee, and it is worth being clear-eyed about its limits. It does not eliminate the need for careful feature design, does not automatically catch a feature definition that is conceptually wrong (only that it is consistently applied), and does not remove the latency engineering work needed to make even a well-defined feature computable within an online serving budget — a feature that genuinely requires scanning a year of transaction history will still be slow to serve online no matter how well the store is architected, and may need to be approximated (a decayed running aggregate updated incrementally rather than recomputed from scratch) regardless of the store's presence.
The overhead of adopting a feature store — standing up the online store infrastructure, migrating existing pipelines to the shared definition format, training teams on the new abstraction — is real, and for a single team running one or two models with simple, stable features, hand-managing consistency between a small number of well-tested batch and online implementations can genuinely be less overhead than adopting an entire feature store platform. The trade-off tips toward a feature store once an organisation has multiple teams building multiple models that reuse overlapping features (customer lifetime value, recent purchase count, and similar features tend to get recomputed independently, inconsistently, and wastefully across a company's different ML teams without one), once the cost of a training-serving skew incident has already been paid at least once, or once the volume of models in production is large enough that a shared, governed, discoverable feature catalogue becomes valuable in its own right, independent of the skew problem specifically.
Frequently Asked Questions
What exactly is training-serving skew?
It is a mismatch between the feature values a model was trained on and the feature values it receives during live inference, caused by the two being computed through different code paths, data sources, or timing, even when both are nominally implementing the same feature definition.
How is a feature store different from just caching feature values in Redis?
A cache stores computed values but says nothing about how they were computed or whether the same logic produced the offline training values. A feature store's key contribution is a single, versioned feature definition that generates both the offline historical values and the online cached values, so the two are guaranteed to be logically consistent rather than independently maintained.
What is point-in-time correctness and why does it matter?
It means that when building a training example from historical data, each feature must be computed using only information that would genuinely have been known at that example's timestamp, not information that arrived later. Violating this causes label leakage, where a model appears highly accurate offline because it was trained on future information it will not have access to in production.
Do small teams need a feature store?
Not necessarily. A team running a small number of models with simple, well-tested features can often maintain consistency manually. Feature stores earn their overhead once multiple teams and models start sharing overlapping features, or once training-serving skew has already caused a costly production incident.