Reinforcement Learning Simulation

Building the canyon...

Presets

Learning

Algorithm

View

Greedy-policy arrows Q-value wedges (4 actions per tile) Camera follows the rover

Click a tile to toggle a stone wall (resets the table).

Live readouts

Graphs

Episode return (orange) and 20-episode average (green), clipped to -150..0

Error max |Q - Q*| over visited state-actions, log scale (lower is better)

Learned value (orange) vs optimal value V* (blue) along the optimal path

Frame time (ms), green line = 16.7 ms

Numerical checks

running...

How to use

The rover starts at the pad (front left) and must reach the flag (front right). The glowing lava row between them is the cliff: stepping in costs -100 and sends the rover back to the start; every other step costs -1. Press play and watch the table fill in: tile height and colour show the learned value of each state, the four wedges show the value of each action, and the arrows show the greedy policy. Tune alpha, gamma and epsilon, switch between Q-learning and SARSA (Q-learning learns the risky cliff-edge path, SARSA the safer inland path), add stone walls by clicking tiles, and press Demo greedy run to see the learned behaviour without exploration. Drag to orbit, wheel to zoom.

The science

The task is a finite Markov decision process: states are the 12 x 4 tiles, actions are up, right, down, left, the reward is -1 per step and -100 for falling off the cliff, and the episode ends at the goal. Tabular Q-learning updates one state-action value per step towards the Bellman optimality target:

Q(s,a) <- Q(s,a) + alpha [ r + gamma max_a' Q(s',a') - Q(s,a) ] SARSA: ... + gamma Q(s',a') with a' chosen by the same epsilon-greedy policy

The optimal values solve Q*(s,a) = r + gamma max Q*(s',a') and are computed here by value iteration for comparison. Q-learning is off-policy, so its table converges to the optimum even while the agent explores (with probability epsilon it takes a random action), which makes its online path run along the cliff edge and occasionally fall; SARSA learns the values of the epsilon-greedy policy itself and prefers the longer, safer route (Sutton and Barto, Example 6.6, typical result with epsilon = 0.1). With slip probability p the chosen action is replaced by a random one; the optimum shown (Q*) is for the deterministic problem. Tile heights are drawn as 0.15 + 0.5 clip((max_a Q + 40)/40, 0, 1) and are a visualisation, not a physical quantity.

Simplifications: a tiny discrete world, a tabular agent with constant alpha, no function approximation; the rover animation interpolates between tiles and is not driven by physics. The rover, tiles, stone walls, flag, start pad, lava and policy arrows are modelled and baked in Blender 5.2.

What the checks mean

Bellman fixed point verifies that the value-iteration table satisfies the optimality equation. Shortest path compares -V*(start) at gamma = 1 with a breadth-first search over the tiles (a different algorithm). Convergence runs a hidden tabular Q-learning agent with a decaying learning rate 1/sqrt(n) and exploration for 300,000 steps and compares its table with Q*. Greedy rollout follows the learned policy and expects the shortest path. Rendered heights reads the tile instance matrices of the live agent and compares them with the value-to-height formula.

Keyboard and accessibility

Space pauses learning, R resets, C re-runs the checks (while the page has focus).
All controls are native inputs with labels.