Every subsystem sends a periodic heartbeat pulse to a hardware watchdog timer. Each watchdog independently tracks elapsed time since its last heartbeat:
if (t_now - t_lastHeartbeat) > T_timeout:
force subsystem reboot
log reset event at t_now
A stuck task (deadlock, radiation-induced hang, infinite loop) simply stops sending heartbeats — the watchdog can't tell *why* it hung, only that it did, so it force-reboots the module. That single-subsystem recovery is fast and cheap.
But a spacecraft's fault manager also watches the rate of resets across a sliding time window W. If too many independent reboots pile up, that is itself a signal the vehicle is in trouble (bad radiation environment, a systemic bug, a failing bus), so it escalates:
resets_in_window = count(reset events with t_now - t_event < W)
if resets_in_window >= N_threshold:
enter SAFE MODE
de-power all non-essential subsystems
hold essential bus: power, thermal, attitude (sun-pointing)
Safe Mode is a deliberately minimal, well-tested configuration — everything non-essential is shed so the limited onboard autonomy only has to keep the vehicle alive and communicating until ground control can diagnose the root cause. After a hold period with no further faults, the fault manager repowers subsystems one at a time and returns to Nominal.
- Inject Random Fault — freezes one healthy subsystem's heartbeat for a few seconds, exactly as a hung task would.
- Auto-inject — keeps triggering faults at a random interval so you can watch the reset-rate cascade build up on its own.
- Watchdog timeout — how long a subsystem can go silent before its watchdog force-reboots it. Shorter = faster recovery but more false trips from normal jitter.
- Safe-Mode threshold — how many resets inside the 20 s window trigger the cascade. Lower = more cautious, higher = more tolerant of noisy environments.
- Drag the diagram to rotate it, scroll/pinch to zoom — the radial layout and reset-window strip below it both update live.
Real-world relevance: this two-tier design — per-task watchdogs plus a windowed escalation policy — is standard on CubeSats, deep-space probes and human-rated vehicles alike; it's the same pattern behind the Mars rovers' and Voyager's own safe-mode entries.