Inference Autoscaling: Queueing Theory for ML Model Serving

Every production ML model behind an API sits inside an M/M/c queue whether its operators think about it that way or not: requests arrive at some rate λ, a pool of replicas each serve them at rate μ, and the gap between those two numbers decides whether latency stays flat or explodes. This simulator renders a live inference fleet in 3D — request particles spawn, queue for a free replica, get processed and either complete or get dropped when the wait buffer overflows — while a Kubernetes-style autoscaler watches the fleet's utilization ρ = λ/(c·μ) and adds or removes replicas to hold it near your chosen target. Tune the arrival rate, per-replica service rate and target utilization and watch p95 latency, queue depth, dropped requests and estimated hourly cost respond in real time, exactly the trade-off behind every HPA policy and SageMaker/Vertex AI autoscaling config in production.