Each cybernetic agent approaches a fork: a short, hazardous lane that finishes faster, or a longer lane that is always safe. At the fork the agent's own policy first decides what it wants — more autonomous agents lean harder toward the efficient choice. A hard safety constraint then checks that want against the lane's predicted harm probability: if the risk exceeds the constraint, the system overrides the agent regardless of how autonomous it is, and the trip is forced onto the safe lane. That override is the "human oversight" layer made visible.
wants_risky = random() < autonomy
blocked = wants_risky AND hazard(lane) > safety_constraint
takes_risky = wants_risky AND NOT blocked
efficiency += takes_risky ? gain_risky : gain_safe
if takes_risky AND random() < hazard(lane): incident++
- Autonomy level — how strongly each agent's own policy favors the fast, efficiency-maximizing lane over deferring to the conservative default.
- Safety constraint — a hard ceiling on acceptable predicted harm; any decision above it gets overridden no matter how autonomous the agent is. This is the tunable boundary between machine judgment and imposed human values.
- Traffic density — number of agents deciding concurrently; more agents surface the trade-off faster.
- Trust index — share of risky attempts that resolved without an incident, a rough proxy for whether the current autonomy/safety balance is earning its efficiency gains.
Real-world relevance: this is the same structural tension behind self-driving route planners, algorithmic triage and automated trading — a system optimizing for a measurable goal, wrapped in a hard-coded constraint that a designer chose on someone else's behalf.