Queueing Chain: M/M/c/K Birth-Death Diagram for ML Inference Autoscaling

Traffic

Arrival rate λ 12.0 req/s

Service rate μ per replica 4.0 req/s

Autoscaler target utilization 70%

Replicas

Live Readouts

Active replicas c

1 / 16

Utilization ρ

Queue length

p95 latency

— ms

Dropped (blocked)

Est. cost

$0.00/hr

Current state n Empirical PMF Theoretical Erlang PMF Blocking state (n=N)

How it works

This is the same M/M/c/K queue as the 3D fleet view, drawn instead as its actual mathematical object: a continuous-time birth–death chain on states n = 0..N (N = c active replicas + K = 24 buffer slots). State n climbs by one on every Poisson arrival (rate λ) and drops by one on every service completion (rate n·μ while n≤c, rate c·μ once every replica is busy).

Stationary PMF (finite-buffer Erlang):
  P(n) ∝ (λ/μ)^n / n!            for n ≤ c
  P(n) ∝ P(c) · [λ/(c·μ)]^(n-c)  for c < n ≤ N
Detailed balance:  λ·P(n) = death_rate(n+1)·P(n+1)   for every n
Blocking prob.:     P(N)  (arrivals dropped exactly when the chain sits at n=N)

The top strip is the chain itself — a ladder of circles, one per reachable state, with the live marker riding the actual simulated occupancy. The middle strip is that occupancy traced over time next to the replica count c(t), which the autoscaler moves. The bottom strip is a live histogram: amber bars are the simulator's own time-averaged occupancy, the green line is the closed-form stationary PMF computed straight from the formula above for the current λ, μ, c — they converge because the chain really does spend its time in each state exactly as often as the algebra predicts.

λ (arrival rate) — birth rate of the chain.
μ (service rate) — per-replica death rate.
c (active replicas) — where the death rate stops scaling with n and saturates at c·μ.
Target utilization — the ρ = λ/(c·μ) the autoscaler tries to hold by adding/removing replicas, which redraws the whole chain live.

Real-world relevance: this is the exact birth–death model behind Kubernetes HPA/KEDA autoscaling, AWS SageMaker/GCP Vertex AI endpoint scaling, and any capacity-planning spreadsheet trading replica cost against tail latency.

Birth–death chainλ=12.0/s · μ=4.0/s · c=1