Microservices Load Balancer: Request Routing & Failover

Load-Balancing Algorithm

Incoming request rate 6 req/s

Avg. service time per request 180 ms

Service Instances (click to kill/revive)

Live Metrics

Throughput

0 req/s

Avg latency

0 ms

Dropped

Healthy nodes

6 / 6

How it works

An API gateway sits in front of a pool of stateless service instances. Every incoming request must be routed to exactly one healthy instance, which processes it for some service time and returns a response. This is a classic queueing system — each instance behaves like an M/M/1 queue with arrival rate λ and service rate μ:

ρ = λ / μ                 (utilization, must stay < 1)
W = ρ / (μ · (1 − ρ))     (avg wait time in queue)
Latency = W + 1/μ         (wait + actual service time)

Four dispatch policies decide which instance gets the next request:

Round Robin — cycles through instances in a fixed order, ignoring current load. Simple, but can overload a slow node.
Least Connections — always sends the request to whichever healthy instance currently has the smallest queue, adapting to uneven load.
Weighted — instances have different capacities (a faster instance has a higher weight); requests are distributed proportionally to weight.
Random — picks a uniformly random healthy instance; a useful baseline with no coordination overhead.

Killing an instance simulates a real outage: the gateway detects it is unhealthy and stops routing to it (failover), while any requests still queued there are dropped — exactly the trade-off a real load balancer's health checks manage. If every instance's queue is full, new requests are rejected outright (a 503), which is why over-provisioning capacity below ρ ≈ 1 matters.

Gateway → instance (request) Completed response Dropped request

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.