Both robots run the same tabular Q-learning: pick an ε-greedy action, move, get a reward, update Q(s,a), repeat. The naive agent (left) executes whatever action it picks — including exploratory moves that step off the table edge or into the obstacle. Every such move ends the episode as a crash and the counter jumps.
The safe agent (right) keeps a learned safety envelope: the orange-tinted tiles bordering the edge and the obstacle. Before any action — exploratory or greedy — executes, it is checked against the envelope. An action that would leave the envelope is intercepted and swapped for the best-valued action that stays inside it (drawn as a green redirect arrow over the rejected red one). The agent still explores and its reward curve still climbs — it just never experiences the crash it was about to make.
a ~ ε-greedy(Q, s)
if SAFE_MODE and a leaves envelope(s):
a ← argmax_safe Q(s, ·) // fallback
s', r ← step(s, a)
Q(s,a) += α[r + γ·max Q(s',·) − Q(s,a)]
- Training speed — simulation steps per animation frame.
- Exploration ε — starting probability of a random action; decays toward 0.05 as episodes accumulate, on both boards identically.
- Safety overrides — total count of times the safe agent's chosen action was intercepted and replaced.
This mirrors constrained/shielded reinforcement learning used to train real robots: a runtime monitor wraps the learning loop and substitutes a safe recovery action whenever the policy's proposal would breach a known constraint, so training never needs to survive an actual fall or collision to learn to avoid one.