Technology Web Development Simulation

Push the load towards capacity and watch latency blow up. Educational queueing model.

The science

Requests arrive as a Poisson stream of rate λ; each of c servers needs an exponential time with mean s = h·tcache + (1-h)/μ (h = cache hit rate, tcache = 2 ms). With offered load a = λs and utilisation ρ = a/c < 1 the M/M/c queue gives the Erlang C waiting probability C(c,a) = [ac/c! · c/(c-a)] / [∑k<c ak/k! + ac/c! · c/(c-a)], mean wait Wq = C·s/(c-a), and mean response T = Wq + s. As ρ approaches 1 latency diverges, which is why a site that is fine at 70% load collapses at 98%. The page cross-checks the formula with a separate first-come-first-served simulation that uses only random numbers and an earliest-free-server rule. Simplifications: exponential times, no network, no retries, no autoscaling; numbers are illustrative. The live panel runs in slow motion.