Service Discovery: Heartbeat Registry & Stale-Cache Failures

Registry Parameters

Heartbeat interval H 1.0s

Suspect TTL = k·H 2.0×

Instance crash rate 4%/s

Client cache refresh R 2.0s

Instance States

Healthy — heartbeat within TTL Suspect — missed one TTL window Down — evicted from registry

Healthy

Suspect

Down

Avg detect time

—

Requests routed

Sent to dead node

Stale-cache failure rate

0.0%

How it works

Every instance sends a heartbeat to the registry roughly every H seconds (with small random jitter). The registry classifies each instance purely from elapsed time since its last heartbeat, t_e:

TTL = k · H
HEALTHY  if t_e < TTL
SUSPECT  if TTL ≤ t_e < 2·TTL
DOWN     if t_e ≥ 2·TTL   (evicted)

This is the same heartbeat/TTL failure-detector logic used by Eureka, Consul and etcd-based service registries — a dead instance is never told to leave; it is simply forgotten once nobody hears from it for long enough.

The client does not query the registry on every request. It caches the list of HEALTHY instance ids and refreshes that snapshot only every R seconds. Between refreshes the client keeps routing to whatever was healthy at the last refresh — including an instance that crashed moments later. A request landing on such a node is counted as sent to dead node:

expected stale window ≈ R / 2
P(request hits a dead node) grows with crash rate × (R / 2)

Turn the crash rate up or R up and watch the failure counter climb even though every request is still "routed" to an instance the client believed was healthy — the core lesson of client-side service discovery: staleness is a knob, not a bug.