MLOps Foundations

MLOps is the discipline of deploying and maintaining ML models in production reliably. It bridges the gap between notebook experiments and systems that serve predictions 24/7. Without MLOps, models rot, pipelines break silently, and teams cannot iterate.

Where It Is Used

Every organization running ML in production needs MLOps, whether they call it that or not:

• Tech companies: Automated retraining, A/B testing, feature stores
• Finance: Model governance, audit trails, reproducibility requirements
• Healthcare: Regulatory compliance, model versioning, validation pipelines
• E-commerce: Real-time serving, personalization at scale, rapid experimentation
• Manufacturing: Edge deployment, model updates, sensor data pipelines

Companies with mature MLOps: Uber (Michelangelo), Airbnb (Bighead), Spotify (ML Platform), Netflix (Metaflow).

Typical Data You Will See

MLOps deals with metadata, not just training data:

Typical volumes: - Experiment tracking: Thousands of runs per team per month - Serving logs: Millions to billions of predictions per day - Feature computation: Hourly to real-time updates

What You Do in Practice

1. Version everything - Code, data, models, configs. If you cannot reproduce it, you cannot debug it.
2. Build reproducible pipelines - Training should run the same way every time, on any machine.
3. Separate feature logic from training logic - Features are reused across models; keep them modular.
4. Implement model registry - Track which model is in production, who approved it, what it replaced.
5. Automate testing - Unit tests for features, integration tests for pipelines, validation tests for models.
6. Deploy with rollback capability - Blue-green or canary deployments. Never deploy without escape hatch.
7. Monitor predictions - Track distribution, latency, error rates. Alert before users notice.
8. Close the feedback loop - Get labels back, compute production metrics, trigger retraining.
9. Document lineage - Which data, features, and code produced this model?
10. Establish team workflows - Code review for ML code, model review before deployment.

Baseline First

Before building complex infrastructure, establish minimal viable MLOps:

1. Git for code - All training code, feature definitions, configs in version control
2. Manual model registry - Spreadsheet or simple database tracking model versions
3. Scheduled retraining - Cron job that runs training script weekly
4. Basic monitoring - Log prediction counts and latencies, alert on anomalies
5. Manual deployment - Documented process to deploy new model version

This baseline handles 80% of use cases. Add automation incrementally based on pain points.

Common Patterns and When to Choose

Start with scheduled batch retraining. Add triggered retraining when you have drift detection in place.

Infrastructure Patterns

Pipeline orchestration: - DAGs define dependencies between steps - Retry logic handles transient failures - Caching avoids redundant computation - Parameterization enables experimentation

Feature serving: - Batch features: Precomputed, stored in data warehouse - Online features: Low-latency lookup from feature store - On-demand features: Computed at inference time

Model serving architectures: - Embedded: Model in application code (simple, tight coupling) - Model-as-service: Dedicated prediction service (scalable, independent) - Batch prediction: Precompute predictions for all entities (no latency requirement)

Storage layers: - Artifact store: Model files, datasets (S3, GCS, Azure Blob) - Metadata store: Experiments, runs, metrics (MLflow, database) - Feature store: Feature values for training and serving (Feast, Tecton)

Evaluation and Metrics

MLOps success metrics:

Pitfalls: - Optimizing for deployment speed without monitoring - Complex infrastructure for simple problems - Lack of documentation making systems unmaintainable - No testing, leading to silent failures

Failure Modes and Debugging Checklist

• [ ] Pipeline failure: Step failed, no notification, stale model served
• [ ] Training-serving skew: Features computed differently in training vs serving
• [ ] Version mismatch: Model expects features that pipeline no longer produces
• [ ] Silent data corruption: Upstream data changed format, pipeline did not fail
• [ ] Resource exhaustion: Training job OOM, serving pod crashes under load
• [ ] Drift undetected: Model degraded, monitoring did not catch it
• [ ] Rollback impossible: No previous model version available or tested
• [ ] Reproducibility broken: Cannot recreate training results

Production Checklist

Infrastructure: - [ ] Pipeline orchestrator running (Airflow, Prefect, Kubeflow) - [ ] Model registry operational - [ ] Artifact storage configured with retention policy - [ ] Serving infrastructure autoscales - [ ] Logging and monitoring integrated

Processes: - [ ] Code review required for all ML code changes - [ ] Model validation gate before production deployment - [ ] Runbook for common incidents - [ ] On-call rotation established - [ ] Regular review of model performance

Documentation: - [ ] System architecture documented - [ ] Feature definitions catalogued - [ ] Model cards for production models - [ ] Incident postmortems tracked

Tools and Stack

Example Mini-Case

Scenario: Fintech startup with 5 ML models in production, 3 data scientists, growing data volume.

Current state: Models trained in notebooks, deployed manually via SSH, no monitoring.

MLOps implementation: 1. Week 1-2: Git repo for all ML code, basic CI with linting and unit tests 2. Week 3-4: MLflow for experiment tracking and model registry 3. Week 5-6: Airflow for scheduled retraining pipelines 4. Week 7-8: FastAPI serving with Docker, deployed to Kubernetes 5. Week 9-10: Evidently for drift monitoring, PagerDuty alerts

Outcome: Deployment time reduced from 2 days to 2 hours. First detection of drift issue that would have gone unnoticed. Team can now safely experiment and iterate.

Next phase: Feature store, A/B testing framework, automated retraining triggers.

Related Pages

• Index: ML Field Guide
• Feature Stores in Practice
• Model Monitoring and Observability
• ML Pipeline Orchestration

Further Reading

• Google MLOps Whitepaper
• MLflow Documentation
• Kubeflow Documentation
• Evidently AI Documentation
• Chip Huyen - Designing Machine Learning Systems

Try it live

Everything above runs in your browser — open MLOps Lab — Watch Model Drift Get Caught (or Not) and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.