Cliff Walking: SARSA vs Q-Learning (2D)

Two reinforcement-learning agents train on the same 4×12 "Cliff Walking" gridworld at once — one with on-policy SARSA, the other with off-policy Q-Learning — rendered on a flat 2D canvas with live policy-arrow fields and episode-reward learning curves. Every step costs a small penalty, falling off the cliff costs a large one and resets the agent to start, and reaching the goal ends the episode. Adjust exploration, learning rate and discount factor and watch each agent's Q-value heat-map and arrows reshape themselves: Q-Learning converges on the fast, risky path that hugs the cliff edge, while SARSA learns to give the cliff a wider berth because its updates account for its own exploratory mistakes — a gap you can also see in the two reward curves and the "total training reward" stat, which is typically lower for Q-Learning because it keeps stepping off the cliff during training even as its final greedy policy gets shorter.