Queueing Chain: M/M/c/K Birth-Death Diagram for ML Inference Autoscaling

Every production ML model behind an API sits inside an M/M/c/K queue whether its operators think about it that way or not: requests arrive at some rate λ, a pool of c replicas each serve them at rate μ, and a finite buffer of K waiting slots decides how many requests get dropped when the fleet falls behind. This 2D companion strips away the 3D warehouse-floor view and renders the queue as what it mathematically is — a continuous-time birth–death chain hopping between occupancy states — alongside a live histogram that checks the simulator's own behaviour against the exact closed-form Erlang stationary distribution. Tune the arrival rate, per-replica service rate and autoscaler target and watch the chain, the occupancy trace and the histogram all respond together, exactly the trade-off behind every HPA policy and SageMaker/Vertex AI autoscaling config in production.