Inference Autoscaling: Queueing Theory for ML Model Serving

Traffic

Arrival rate λ 12.0 req/s

Service rate μ per replica 4.0 req/s

Autoscaler target utilization 70%

Replicas

Live Readouts

Active replicas

1 / 16

Utilization ρ

Queue length

p95 latency

— ms

Dropped (overflow)

Est. cost

$0.00/hr

Queued Processing Completed Dropped

How it works

Inference requests arrive as a Poisson process with rate λ (requests/second). Each of the c active replicas is an independent server that completes a request after an exponentially-distributed service time with rate μ — together this is the classic M/M/c queue used to model any pool of stateless workers behind a load balancer, including an ML model-serving fleet.

Utilization:   ρ = λ / (c·μ)         (must stay < 1 or the queue grows without bound)
Little's Law:  L = λ·W                (avg requests in system = arrival rate × avg time in system)
Erlang C:      P(wait > 0) depends on c, λ/μ — more replicas shrinks both queue and tail latency

The autoscaler mimics a Kubernetes Horizontal Pod Autoscaler: every couple of seconds it compares the fleet's live utilization ρ against your target utilization. If ρ sits above target it adds a replica (up to 16); if it sits well below target it removes an idle replica (down to 1), with a short cooldown so it doesn't flap on every fluctuation.

λ (arrival rate) — how many inference requests hit the fleet per second.
μ (service rate) — how many requests one replica can finish per second (inverse of average inference latency).
Target utilization — the ρ the autoscaler tries to hold; lower targets keep more headroom (lower tail latency) at higher cost.
Queue overflow — once the wait buffer fills, new requests are dropped rather than queued forever, exactly like a real load balancer's bounded request queue.

Real-world relevance: this is the same math behind Kubernetes HPA/KEDA autoscaling policies, AWS SageMaker/GCP Vertex AI endpoint autoscaling, and any capacity-planning spreadsheet for a production inference API — trading replica cost against p95 latency.

Inference fleetλ=12.0/s · μ=4.0/s

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.