Dynamic Batching for AI Inference Serving

Traffic & Batching Controls

Arrival rate λ 10 req/s

Max batch size B_max 8

Max wait (batch timeout) 15 ms

Per-item compute cost 1.2 ms

Live Server State

Queue depth

GPU utilization

Avg latency

0 ms

Throughput

0 req/s

Queued request

In current batch

Compute core (busy = bright)

How it works

Requests arrive at random (Poisson-like) times at rate λ. A single GPU can only run one batch at a time, but launching a kernel and moving weights/activations has a mostly-fixed overhead — so processing 1 request and processing 32 at once cost roughly the same fixed price, amortized over more work as the batch grows. The scheduler here uses the standard "batch until full or a timeout, whichever first" rule:

start batch when:  queue ≥ B_max   OR   oldest wait ≥ W_max

compute time:  T(B) = T_fixed + c · B^0.6
throughput:    X(B) = B / T(B)          (requests / unit time)
latency:       L = queue_wait + T(B)

The exponent 0.6 (<1) is the key: it models sub-linear scaling of GPU compute time with batch size, since a GPU parallelizes most of the extra work across its cores. That sub-linearity is exactly why batching raises throughput — but every request in a batch still has to wait for the whole batch to finish, and a request unlucky enough to arrive right after a batch just launched waits close to W_max in the queue before its own batch even starts.

Arrival rate λ — how many requests per second hit the server; higher λ fills batches faster.
Max batch size — the hard cap the scheduler will pack into one GPU pass.
Max wait — the timeout that flushes a partial batch so low-traffic requests aren't stuck forever; 0 ms means "never wait, process instantly one at a time."
Per-item compute cost — how expensive the model itself is per example (a bigger model raises this and pushes the whole latency/throughput curve up).

This is the exact trade-off behind real inference servers (NVIDIA Triton, vLLM continuous batching, TensorFlow Serving): a tighter timeout or smaller batch cap gives lower worst-case latency but wastes GPU parallelism; a larger batch cap and longer timeout raise throughput and utilization but make individual requests wait longer.

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.