Load Balancing: Round Robin, Least Connections & Power of Two Choices

Why one queue beats many

A load balancer sits in front of a pool of backend servers and decides, for every incoming request, which server handles it. The goal sounds simple — spread the work out — but the policy you pick changes tail latency by an order of magnitude even when total throughput looks identical on a dashboard. The core insight from queueing theory is that one shared queue feeding many servers beats one private queue per server, because a private queue can sit idle behind a slow request while another queue backs up. Most real balancers approximate a shared queue without actually building one, by routing based on live signals from each backend.

Round robin is the simplest policy: request 1 goes to server A, request 2 to B, request 3 to C, then back to A. It is stateless, cheap, and fair by count — but it is blind to how long each request actually takes. If server B is handling a slow report-generation query, round robin keeps sending it new work on schedule anyway, and its queue grows while A and C sit comparatively idle.

Least connections and weighted variants

Least connections fixes that blindness by routing each new request to whichever server currently has the fewest in-flight requests. It needs the balancer to track live connection counts, but the payoff is real: slow servers naturally receive less new work because their connection count stays high. Weighted least connections extends this for heterogeneous hardware — a server rated at twice the capacity gets a weight of 2, and the balancer effectively divides its connection count by that weight before comparing.

score(server) = activeConnections(server) / weight(server)
next request -> argmin(score) over all healthy servers

Weighted round robin is the corresponding fix for round robin: a server with weight 3 receives three requests for every one that a weight-1 server receives, cycling through in a fixed pattern rather than by live load. It is a good middle ground when you know relative capacity in advance but don't want the bookkeeping cost of tracking connections.

Random and power-of-two-choices

Pure random routing is surprisingly close to round robin in the long run (by the law of large numbers each server gets roughly 1/N of the traffic) but has worse short-term variance — nothing stops five requests in a row landing on the same server. The interesting refinement is power of two choices: instead of picking one server at random, the balancer samples two servers at random and sends the request to whichever of the two has fewer active connections. Michael Mitzenmacher's analysis showed this simple tweak reduces the maximum load on any server from O(log n / log log n) (pure random) to O(log log n) — an exponential improvement — for a negligible increase in bookkeeping, which is why it shows up inside real systems like nginx's least_conn mode extensions and many service meshes.

What the queues actually do to latency

Every policy above is trying to approximate the same target: keep each server's queue short and roughly equal. Little's Law, L = λW, ties queue length L to arrival rate λ and average time in system W — for a fixed arrival rate, the only way to cut wait time is to cut queue length, and the only way to cut queue length without dropping requests is to route work where the queue is actually shorter. This is precisely why least-connections and power-of-two-choices outperform blind round robin under bursty or heterogeneous load: they use live queue-length information that round robin discards.

// naive least-connections routing
function route(servers) {
  let best = null;
  for (const srv of servers) {
    if (!srv.healthy) continue;
    if (!best || srv.active / srv.weight < best.active / best.weight) best = srv;
  }
  best.active++;
  return best;
}

Health checks, stickiness and failure

None of this matters if the balancer keeps sending traffic to a dead server. Health checks — periodic pings or synthetic requests — remove unresponsive backends from rotation and add them back once they recover, usually with a slow-start ramp so a just-recovered server isn't instantly hit with a full share of traffic while its caches are cold. Session stickiness (routing a given client consistently to the same backend, often via a cookie or consistent hashing on client IP) trades load-balancing quality for simplicity when backends hold local session state; consistent hashing specifically minimises how many clients get remapped when a server is added or removed, which matters a lot for cache hit rates.

Frequently asked questions

Try it live

Everything above runs in your browser — open Load Balancer and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.