Cliff Walking: SARSA vs Q-Learning

Two reinforcement-learning agents race across the same 4×12 "Cliff Walking" gridworld at once — one trained with on-policy SARSA, the other with off-policy Q-Learning — so you can watch the textbook divergence between the two temporal-difference control algorithms happen live. Every step costs a small penalty, falling off the cliff costs a large one and resets the agent to start, and reaching the goal ends the episode. Adjust exploration, learning rate and discount factor and watch each agent's Q-value heat-map reshape itself: Q-Learning converges on the fast, risky path that hugs the cliff edge, while SARSA learns to give the cliff a wider berth because its updates account for its own exploratory mistakes.