Feature Stores in Production

A feature store is a centralized system for storing, managing, and serving ML features. It solves the problem of feature consistency between training and serving, enables feature sharing across teams, and provides a single source of truth for feature definitions. Feature stores bridge the gap between data engineering and ML engineering.

Where It Is Used

Feature stores power ML platforms across industries:

• E-commerce: Real-time personalization, recommendation features
• Finance: Fraud detection features, credit scoring signals
• Ride-sharing: Driver/rider features, ETA prediction inputs
• Ad tech: User profiles, ad relevance features
• Gaming: Player behavior features, churn signals
• Healthcare: Patient features for risk models

Companies with feature stores: Uber (Michelangelo), Airbnb (Zipline), Netflix, Spotify, LinkedIn, DoorDash.

Typical Architecture

┌─────────────────────────────────────────────────────────────┐
│                     Feature Store                          │
├─────────────────────────────────────────────────────────────┤
│  ┌─────────────────┐    ┌─────────────────┐               │
│  │  Offline Store  │    │  Online Store   │               │
│  │  (Historical)   │    │  (Low-latency)  │               │
│  │  - Parquet/S3   │    │  - Redis        │               │
│  │  - BigQuery     │    │  - DynamoDB     │               │
│  │  - Snowflake    │    │  - Cassandra    │               │
│  └────────┬────────┘    └────────┬────────┘               │
│           │                      │                         │
│           ▼                      ▼                         │
│  ┌─────────────────────────────────────────────┐          │
│  │         Feature Registry/Catalog            │          │
│  │  - Feature definitions                      │          │
│  │  - Metadata, lineage                        │          │
│  │  - Versioning                               │          │
│  └─────────────────────────────────────────────┘          │
└─────────────────────────────────────────────────────────────┘
        │                              │
        ▼                              ▼
   Training                       Inference
   (batch)                        (real-time)

What You Do in Practice

1. Define feature requirements - What features, what freshness, what latency?
2. Design feature schema - Entity, features, timestamps
3. Build transformation pipelines - Raw data → features
4. Register features - Definitions, owners, documentation
5. Materialize to stores - Offline and online as needed
6. Validate consistency - Training/serving skew checks
7. Serve features - API for online, point-in-time for offline
8. Monitor usage and quality - Tracking, alerting
9. Version and evolve - Handle schema changes
10. Share across teams - Discovery, reuse

Baseline First

Start with Feast, the most popular open-source feature store:

from feast import FeatureStore, Entity, Feature, ValueType
from feast import FileSource, FeatureView
from datetime import timedelta

# Define entity
customer = Entity(name="customer_id", value_type=ValueType.INT64)

# Define feature source
customer_source = FileSource(
    path="data/customer_features.parquet",
    timestamp_field="event_timestamp"
)

# Define feature view
customer_features = FeatureView(
    name="customer_features",
    entities=["customer_id"],
    ttl=timedelta(days=1),
    features=[
        Feature(name="total_purchases", dtype=ValueType.INT64),
        Feature(name="avg_order_value", dtype=ValueType.FLOAT),
        Feature(name="days_since_last_order", dtype=ValueType.INT64),
    ],
    source=customer_source,
)

# Use feature store
store = FeatureStore(repo_path=".")

# Training: get historical features
training_df = store.get_historical_features(
    entity_df=entity_df,
    features=["customer_features:total_purchases",
              "customer_features:avg_order_value"]
).to_df()

# Serving: get online features
online_features = store.get_online_features(
    features=["customer_features:total_purchases"],
    entity_rows=[{"customer_id": 1001}]
).to_dict()

Why Feast baseline: - Open source, vendor neutral - Supports major cloud providers - Clear separation of offline/online - Active community

Feature Store Options

Decision guide: - Just starting? → Feast - Enterprise with budget? → Tecton - Already on cloud MLOps? → Native feature store - Real-time streaming features? → Tecton or custom - Simple batch features? → dbt + warehouse might suffice

Key Concepts

Point-in-time correctness: Features must be computed as of the prediction time, not current time. This prevents data leakage.

# WRONG: Current feature values
current_purchases = get_total_purchases(customer_id)

# RIGHT: Features as of prediction time
historical_purchases = get_features_at_time(
    customer_id,
    as_of=prediction_timestamp
)

Training/serving skew: Differences between features computed for training vs serving cause model degradation. Feature stores ensure consistency by using the same transformation code.

Feature freshness: - Batch features: Updated hourly/daily (e.g., total purchases) - Streaming features: Updated in real-time (e.g., session count) - On-demand features: Computed at request time (e.g., time since last action)

Feature versioning: Track feature definitions over time to reproduce training data and handle schema evolution.

Evaluation and Monitoring

Monitoring checklist: - Feature value distributions (detect drift) - Null rates (detect pipeline failures) - Serving latency percentiles - Cache hit rates (online store) - Materialization lag (freshness)

Failure Modes and Debugging Checklist

• [ ] Training/serving skew: Different transformations in training vs serving
• [ ] Data leakage: Features computed with future information
• [ ] Stale features: Materialization pipeline failed
• [ ] Missing features: Entity not in feature store
• [ ] Schema mismatch: Feature definition changed
• [ ] Latency spikes: Online store overloaded
• [ ] Inconsistent timestamps: Timezone or format issues
• [ ] Feature drift: Distribution changed, not detected

Production Checklist

Before deployment: - [ ] Feature definitions documented - [ ] Transformations tested and validated - [ ] Point-in-time correctness verified - [ ] Online/offline consistency checked - [ ] Latency requirements met - [ ] Backup/recovery plan defined

After deployment: - [ ] Feature freshness monitored - [ ] Distribution drift tracked - [ ] Serving latency monitored - [ ] Pipeline health alerting - [ ] Usage analytics tracked - [ ] Cost monitoring (storage, compute)

Governance: - [ ] Feature ownership defined - [ ] Access controls configured - [ ] Data lineage tracked - [ ] Deprecation process defined - [ ] Feature discovery enabled

Tools and Stack

Example Mini-Case

Scenario: E-commerce platform needs feature store for personalization models serving 10K requests/sec with < 20ms latency.

Requirements: - 50 features per user (demographics, behavior, transactions) - Real-time session features + batch historical features - Training data for 1 year of history - Feature sharing across 5 ML teams

Architecture:

Raw data (S3) → Spark transformations → Parquet (offline)
                      ↓
               Redis (online, batch features)
                      ↓
Kafka streams → Flink → Redis (online, real-time features)
                      ↓
               Feature serving API → Models

Implementation: 1. Deploy Feast with Redis online store, S3 offline store 2. Batch pipeline: Daily Spark job computes 40 features 3. Streaming pipeline: Flink computes 10 real-time features 4. Feature registry: All features documented with owners

Metrics achieved: - Serving latency: P50 5ms, P99 15ms - Feature freshness: Batch 1 hour, streaming 30 seconds - Training/serving skew: < 0.1% difference - Feature reuse: 30% of features used by multiple teams

Key learnings: - Start with batch, add streaming only where needed - Feature documentation critical for adoption - Monitoring caught 3 pipeline issues before model impact - ROI: 40% reduction in feature engineering time

Related Pages

• Index: ML Field Guide
• MLOps Foundations
• Feature Engineering Fundamentals
• Model Monitoring

Further Reading

• Feast Documentation
• Tecton Documentation
• Feature Store for ML (Google)
• Feature Stores: A Hierarchy of Needs

Try it live

Everything above runs in your browser — open Feature Store Skew Detector and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.