Every client call (coloured by HTTP verb) first hits the gateway — a single
choke point that all traffic must pass through before reaching a backend endpoint.
The gateway holds a bucket of tokens (the ring of dots above it): each accepted request
consumes one token, and the bucket refills at a fixed rate. If the bucket is empty when
a request arrives, the gateway rejects it instantly with HTTP 429 Too Many Requests
instead of forwarding it — protecting the backend from overload.
tokens(t+Δt) = min(C, tokens + r·Δt)
if tokens ≥ 1: accept, tokens -= 1, route to endpoint
else: reject → HTTP 429
- Request rate — how many calls per second the client fires; push it above the rate limit to watch the bucket drain and rejections appear.
- Rate limit — the token-bucket refill rate r (tokens/s), i.e. the gateway's configured ceiling on sustained throughput.
- Network latency — the one-way delay of each hop; higher latency means more requests are in flight at once (more concurrent packets visible).
- Send burst — instantly queues 20 extra requests, the classic case where a request-rate limit alone isn't enough and a token bucket's short-term burst capacity C matters.
This is exactly how production API gateways — AWS API Gateway, Kong, NGINX, Cloudflare —
throttle clients: token-bucket (or leaky-bucket) limiting per API key, then path-based
routing of whatever gets through to the right backend microservice endpoint.