Cliff Walking: SARSA vs Q-Learning

Learning Parameters

Exploration ε 0.10

Learning rate α 0.50

Discount γ 0.95

Simulation speed 20 steps/s

Live Stats

Episodes completed0

SARSA last return–

Q-Learning last return–

Legend

SARSA agent (on-policy)

Q-Learning agent (off-policy)

Cliff — step in it: −100 reward, back to start

Goal — episode ends, agent respawns

Tile brightness = learned max Q-value

How it works

Cliff Walking is Sutton & Barto's classic gridworld for comparing on-policy and off-policy temporal-difference control. A 4×12 grid has a start cell, a goal cell, and a row of "cliff" cells between them. Every step costs −1 reward; stepping into the cliff costs −100 and teleports the agent back to start (the episode continues). Both agents use ε-greedy action selection over the same four moves (up/down/left/right).

Q-Learning (off-policy TD control):
Q(s,a) ← Q(s,a) + α·[ r + γ·max_a' Q(s',a') − Q(s,a) ]

SARSA (on-policy TD control):
Q(s,a) ← Q(s,a) + α·[ r + γ·Q(s',a') − Q(s,a) ]
  where a' is the action ε-greedy policy actually takes next

The difference is subtle but produces famously different behaviour: Q-Learning bootstraps off the greedy next action regardless of what the exploring policy will actually do, so it learns the value of the optimal (risky) path hugging the cliff edge — but because it still explores with ε-greedy during training, it occasionally falls off. SARSA bootstraps off the action its own ε-greedy policy will really take, so it "knows" it might slip near the cliff and learns a longer, safer route one row further away. Watch the two heat-maps diverge as episodes accumulate — tile brightness encodes each agent's learned max Q-value for that cell (dark = very negative, bright = near zero).

ε (exploration) — probability of taking a random action instead of the greedy one. Raise it to see both agents fall off the cliff more often.
α (learning rate) — how much each TD error updates a Q-value. Higher α learns faster but noisier.
γ (discount) — how much future reward matters relative to immediate reward.
Reset agents — clears both Q-tables and restarts both agents at episode 0.

SARSA Q-Learning

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.