Model deployment is the process of integrating a trained model into a production environment where it processes real data and serves real predictions to real users or systems. It is also, by most estimates, where the majority of ML projects actually die: only 10-20% of trained models ever reach production, and the gap is rarely about modelling skill.
Three deployment patterns
Batch processing
The model processes data in scheduled batches -- suited to tasks without a real-time response requirement, efficient in resource use. Typical example: daily sales forecasts generated overnight.
Real-time API
The model serves predictions through a low-latency API, suited to interactive applications. Typical example: real-time recommendations rendered as a user browses.
Edge deployment
The model runs directly on-device (mobile, IoT) for the lowest possible latency and offline operation, usually requiring the model itself to be compressed or optimised to fit device constraints.
💡 Key idea: a good offline evaluation score is necessary but not sufficient -- production traffic, latency constraints and drift are all things a notebook cannot show you.
The stages between training and production
- Model preparation: serialisation (pickle, joblib, ONNX), versioning (MLflow, DVC), testing, and documentation.
- Containerisation: dependency management, a Dockerfile, image-size optimisation, and container-level testing.
- API layer: a framework (FastAPI, Flask, Django), input validation, graceful error handling, and structured logging.
- Infrastructure: cloud platforms (SageMaker, Vertex AI, Azure ML), Kubernetes orchestration, load balancing, and auto-scaling.
- Monitoring: latency, throughput and error rate; model-quality metrics and drift detection; infrastructure health; and alerting.
Why canary rollouts matter
Rather than switching 100% of traffic to a new model at once, a canary rollout exposes it to a small, controlled slice of traffic first -- 5%, then 25%, then 50%, then everything -- with monitoring at each stage. Pairing this with an auto-rollback rule (revert automatically if the challenger's observed error rate crosses a threshold) turns "did we just break production" from a manual pager-duty scramble into an automated safety net, at the cost of needing to set that threshold correctly: too tight and normal statistical noise triggers false alarms; too loose and a real regression runs at high traffic before anyone notices.
Tooling landscape
MLOps platforms: MLflow (lifecycle management), Kubeflow (ML on Kubernetes), TFX (TensorFlow Extended), Seldon (deployment and monitoring). API frameworks: FastAPI is the most common modern choice for its speed and automatic documentation.
🧪 Try it yourself: the Model Deployment Lab simulation lets you experiment with everything described above directly in your browser.