Dynamic Batching for AI Inference Serving

Every production ML model serves requests one at a time in theory, but real inference servers group concurrent requests into batches before running them through the GPU, because the fixed cost of a kernel launch and memory transfer is mostly independent of how many examples ride along with it. This simulator models that scheduler directly: requests arrive at a random rate and queue up as small cubes; once the queue hits the batch-size cap or the oldest request has waited too long, the scheduler flushes a batch into the glowing compute core, which lights up while it processes and then releases the finished requests as sparks. Live readouts track queue depth, GPU utilization, average end-to-end latency, and throughput, so you can feel directly how a larger batch cap or longer timeout trades latency for throughput — and where that trade stops paying off.