🖥️ Model Serving Architectures for Production ML

About this simulation

Training a model is only half the job — production systems need to expose it as a reliable, low-latency service. This scene models a simplified inference cluster: incoming requests hit a load balancer, which routes each one to a model replica. Each replica batches incoming requests together before running inference, since GPUs process a batch almost as fast as a single item.

How it works

• Request rate — how many inference requests arrive per second, visualized as glowing particles streaming from the client.
• Batch size — how many requests each replica groups together per inference pass. Bigger batches raise throughput per replica but make each request wait longer for the batch to fill, raising tail latency.
• Utilization — total request rate divided by total cluster capacity. As utilization approaches 100%, queues build up and latency rises sharply — the classic queueing "hockey stick."
• Autoscaling — when enabled, the cluster adds or removes replicas to keep utilization near a healthy target, up to the max replicas you set. When disabled, the replica count stays fixed at max, like a statically provisioned deployment.

Controls

• Request rate — traffic arriving at the service, in requests per second.
• Batch size — the dynamic batching window size used by each replica.
• Max replicas — the ceiling on how many model replicas can run (or the fixed pool size with autoscaling off).
• Autoscaling — toggle automatic replica scaling based on load.
• ↺ Reset cluster — clears in-flight requests and restarts. Drag to rotate, scroll to zoom.

Did you know?

Real serving frameworks like NVIDIA Triton, TorchServe and TensorFlow Serving use "dynamic batching" — a short timer window that collects concurrent requests before running one forward pass — to trade a few milliseconds of extra wait for dramatically higher GPU throughput.