Model Serving Architectures for Production ML

Fundamentals

Core Concepts

• Interface: synchronous HTTP/gRPC, asynchronous queues, or batch jobs.
• Packaging: container images, serverless functions, or managed model servers.
• Artifact management: immutable model versions, metadata, and signatures.
• Performance: P50/P95/P99 latency, throughput, cold start, and tail behavior.
• Scalability: autoscaling strategies (CPU/GPU utilization, QPS, custom signals).
• Reliability: health checks, circuit breakers, retries, rate limiting, timeouts.
• Observability: logs, metrics, traces, feature/label drift, data quality checks.
• Security: authn/authz, secrets, encryption, tenancy isolation, supply chain.
• Governance: approvals, audit trails, dataset/model lineage, rollback plans.

Serving Architectures

Single Model Service

A dedicated microservice embedding the model runtime. Simple to reason about, ideal for low traffic or prototyping. Tight coupling can complicate upgrades; consider a sidecar for runtime isolation.

Multi-Model Server

A shared server hosting multiple models with dynamic loading and caching. Improves utilization but requires per-model resource isolation, admission control, and per-tenant QoS.

Serverless Inference

Pay-per-invocation with automatic scaling. Great for bursty workloads, but cold starts and limited GPU availability can hurt tail latency. Prewarming and provisioned concurrency mitigate risk.

Batch and Streaming

For offline or nearline scenarios, schedule batch jobs (e.g., nightly scoring) or streaming jobs (e.g., Kafka, Flink) where models run inside stream processors with exactly-once semantics.

Latency and Throughput Trade-offs

• Prefer vectorized inference with micro-batching to improve GPU/CPU efficiency.
• Use request-level timeouts and bounded queues to protect upstream services.
• Cache feature inputs and common responses where feasible; validate freshness windows.
• Enable model quantization and optimized runtimes (ONNX Runtime, TensorRT) for speed.

Scaling Patterns

• Horizontal autoscaling based on QPS, concurrency, or utilization.
• Canary and blue/green rollouts with automated rollback triggers.
• Shadow traffic to validate new models without user impact.
• Traffic shaping by user cohort, geography, or experiment group.

Observability and Quality

Collect structured logs, RED/USE metrics, traces, and model-specific signals (feature ranges, prediction histograms, calibration). Correlate production data with offline validation to detect drift and regressions.

Security and Compliance

• Authenticate callers (mTLS, OAuth) and authorize per endpoint and tenant.
• Protect inputs/outputs at rest and in transit; avoid sensitive data leakage in logs.
• Maintain SBOMs and verify image signatures; pin runtime versions.
• Document data residency and retention policies; enable deletion workflows.

Implementation Guide (Step-by-step)

1. Define SLOs and workloads (QPS, latency, cost targets, concurrency).
2. Choose packaging (container vs serverless) and runtime (CPU/GPU, accelerator).
3. Create a versioned model artifact with schema/signature and metadata.
4. Provision infrastructure: ingress, autoscaler, secrets, observability stack.
5. Implement request validation, timeouts, retries, and rate limiting.
6. Configure canary rollout, shadowing, and automated rollback.
7. Instrument metrics/traces and set alert thresholds (latency, error rate, drift).
8. Establish on-call and incident playbooks with clear ownership.

Examples

• Image classification on GPU with micro-batching and Triton Inference Server.
• Tabular credit scoring via autoscaled HTTP microservice on CPU with ONNX.
• LLM text generation with KV cache, token-level rate limiting, and A/B canaries.

Related Articles

• Model Serving Implementation
• Model Serving Evaluation
• Model Serving Best Practices

How the Algorithm Works

Request Lifecycle

1. Ingress accepts HTTP/gRPC requests and authenticates clients.
2. Validation and schema enforcement filter malformed inputs.
3. Routing selects a model version via rules, traffic splits, or experiments.
4. Queuing buffers requests; admission control rejects excess load.
5. Batcher groups compatible requests within a time/size window.
6. Scheduler assigns batches to CPU/GPU workers based on policy.
7. Postprocessing formats outputs and attaches provenance metadata.

Scheduling Policies

• FIFO/FCFS: Simple, predictable latency distribution.
• Priority: Weighted priority queues by tenant or endpoint.
• Shortest-job-first: Favor small tensors for better tail latency.
• Deadline-aware: EDF using per-request latency budgets.
• Cost-aware: Choose CPU vs GPU based on cost/perf envelope.

Batching Strategies

Micro-batching improves throughput with modest latency cost. Use max batch size, max wait time, and shape compatibility constraints. Employ dynamic padding and pre-allocated memory to reduce overheads.

Backpressure and Flow Control

• Implement bounded queues with immediate shed on overflow.
• Apply token buckets per tenant and per endpoint to enforce fairness.
• Use circuit breakers to prevent cascading failures.

Experimentation

Apply traffic splits (A/B, canary, shadow). Support feature flags and per-cohort routing; store results with run IDs for analysis.

Related Articles

• Model Serving Mathematics
• Model Serving Fundamentals
• Model Serving Best Practices

Real-World Applications

E-commerce Recommendations

• Low latency personalization at scale; heavy A/B testing cadence.
• Feature freshness via streaming updates and session context.

Fraud Detection

• High precision and recall under strict latency budgets.
• Explainability, auditability, and dual-approval workflows.

Healthcare Diagnostics

• Regulatory compliance and rigorous validation.
• Human-in-the-loop and strong safety guardrails.

Industrial IoT and Edge

• Resource-constrained devices; offline tolerance and local fallbacks.
• Robust OTA update mechanisms with rollback.

Related Articles

• Model Serving Examples
• Model Serving Fundamentals
• Model Serving Best Practices

Best Practices

Reliability

• Graceful degradation paths and circuit breakers per dependency.
• Blue/green rollouts with automatic rollback triggers.
• Regular game days and chaos experiments.

Safety

• Strict schema validation and PII scrubbing at ingress.
• Content safety pipelines for generative models.
• Approval workflows and audit logs for sensitive changes.

Performance

• Adaptive batching and concurrency control.
• Runtime optimizations (ONNX Runtime/TensorRT) and quantization.
• Per-tenant QoS and rate limits to isolate heavy users.

Governance and Cost

• Artifact lineage and reproducible builds with SBOMs.
• Reserved capacity for steady loads; serverless for bursty traffic.
• Unit economics dashboards (cost per 1k requests) with alerts.

Related Articles

• Model Serving Fundamentals
• Model Serving Algorithm
• Model Serving Evaluation

Evaluation

SLO Framework

• Latency: P50/P95/P99 targets; budget per stage.
• Availability: monthly/quarterly uptime and error budget.
• Quality: online metrics (CTR, calibration, reward models) with alerts.

Load and Chaos Testing

• Replay real traces; ramp QPS with realistic payload sizes.
• Introduce failures (CPU throttling, network loss, instance kill) to validate resilience.
• Record degradation curves and verify automated rollback triggers.

Guardrails

• Input validation and schema enforcement.
• Content safety filters for text/vision generation tasks.
• Rate limits and timeouts to protect dependencies.

Related Articles

• Model Serving Examples
• Model Serving Best Practices
• Model Serving Fundamentals

Worked Examples

Image Classification on GPU

• Runtime: Triton with TensorRT-optimized engines.
• Batching: 16–64 with dynamic shapes; preallocated device memory.
• Scaling: GPU autoscaling by utilization and queue depth.
• Guardrails: content filters and input validation.

Credit Scoring on CPU

• Runtime: ONNX Runtime with AVX acceleration.
• Batching: small batches (8–16); low latency target.
• Compliance: audit logs, explainability, and lineage.

LLM Text Generation

• Scheduling: separate prefill and decode; cap tokens/request.
• KV Caching: share across requests with eviction policies.
• Safety: prompt filters, jailbreak detection, and red-teaming tests.

Related Articles

• Model Serving Applications
• Model Serving Implementation
• Model Serving Best Practices

Implementation

Prerequisites

• Versioned model artifact with input/output schema and signature.
• Container base image with optimized runtime (ONNX Runtime, TensorRT, PyTorch).
• Observability stack (logging, metrics, tracing) and dashboards.

Step-by-step

1. Build: containerize the model server with health endpoints and readiness checks.
2. Contract: define JSON schema for inputs/outputs; implement strict validation.
3. Security: enforce authn/authz, secrets, and request size limits.
4. Performance: enable batching, thread pools, and pinned memory where applicable.
5. Deployment: provision ingress, autoscaler, and per-tenant quotas.
6. Rollout: use staged canaries and shadow traffic with automatic rollback criteria.
7. Observability: emit RED metrics and model KPIs; set alerts.
8. On-call: define runbooks and incident response procedures.

Operational Checks

• Capacity and SLO validation with load tests before production exposure.
• Data quality gates at ingress to block out-of-contract payloads.
• Periodic drift checks and calibration monitoring.

Related Articles

• Model Serving Fundamentals
• Model Serving Parameters
• Model Serving Best Practices

The Math Behind It

Queueing Theory Essentials

• Little’s Law: L = λW links concurrency, arrival rate, and wait time.
• M/M/1: mean response 1/(μ - λ); tail explodes near high utilization.
• M/G/k: multi-server systems capture GPU worker pools and heterogeneous devices.

Service Time Distributions

Heavy-tailed service times (e.g., LLMs by token length) dominate P99. Use timeouts, max token limits, and preemption to trim tails.

Batching Models

• Throughput increases sublinearly with batch size due to memory bandwidth limits.
• Wait-time penalty grows with batch window; optimize jointly with SLOs.
• Adaptive batching targets a latency budget with feedback control.

Tail Latency Control

• Replicated requests: send to two servers and keep the fastest (hedging).
• Per-class queues: avoid small requests waiting behind large ones.
• Deadline-aware scheduling: prioritize near-deadline jobs.

Capacity Planning

1. Measure arrival rates and concurrency patterns across time windows.
2. Estimate service rates per hardware profile; include variability.
3. Simulate queues under burst scenarios to pick safe utilization targets.

Related Articles

• Model Serving Algorithm
• Model Serving Implementation
• Model Serving Best Practices

Key Parameters

Core Parameters

• Batch size: max items per batch; use adaptive bounds.
• Batch wait: max aggregation delay; keep under 10–20 ms for interactive use.
• Concurrency: in-flight requests or workers per device.
• Timeouts: per stage; budget across ingress, queue, inference, egress.
• Retries: limit attempts; add jittered backoff.
• Queue limits: hard caps to avoid unbounded latency.
• Memory: preallocate buffers; pin critical tensors.
• Autoscaling targets: CPU/GPU utilization or concurrency.

Recommended Defaults

• Interactive CPU: batch 8–16, wait 5–15 ms, timeout 300–800 ms.
• Interactive GPU: batch 16–64, wait 5–20 ms, timeout 150–600 ms.
• LLM decode: cap tokens/sec per worker; enforce max tokens/request.

Tuning Process

1. Profile baseline under representative load with tracing enabled.
2. Increase batch until latency budget breaks; record throughput gains.
3. Raise concurrency carefully; watch tail latency and error rates.
4. Adjust autoscaling targets to stabilize around safe utilization.

Related Articles

• Model Serving Training
• Model Serving Evaluation
• Model Serving Best Practices

Training Strategy

Feature Parity

• Use a single source of truth for feature definitions (feature store).
• Package transforms with the model to avoid leakage or drift.
• Ensure time travel for training data to avoid lookahead bias.

Calibration and Validation

Calibrate probabilities offline, then confirm with online A/B tests. Monitor calibration error in production and retrain when it degrades.

Feedback Loops

• Collect labels or proxies with privacy safeguards.
• Automate data pipelines for frequent retraining or fine-tuning.
• Guard against feedback loops that reinforce bias.

Related Articles

• Model Serving Implementation
• Model Serving Evaluation
• Model Serving Applications