...
running...
The science
This is textbook Q-learning (Watkins, 1989) on Sutton & Barto's cliff-walking task. The agent maintains an action-value table Q(s,a) and, after each transition (s, a, r, s'), applies the temporal-difference update
Q(s,a) ← Q(s,a) + α [ r + γ max_a' Q(s',a') − Q(s,a) ]
Actions are chosen ε-greedily. Q-learning is off-policy: it learns the value of the greedy policy even while exploring, which is why with ε > 0 the learned optimal path runs right along the cliff edge (the optimum of the greedy policy) although the exploring agent often falls off. (SARSA would learn the safer upper path.)
Because the environment is small and deterministic (slip = 0) we can compute the true optimum Q* exactly with value iteration, V(s) ← max_a [r + γ V(s')]. The numerical check does this at start-up and displays the maximum difference |Q − Q*| in the live readouts; it also checks that the optimal undiscounted path from the start has the expected length of 13 moves. Simplifications: tabular, small state space, fixed learning rate; with slip > 0 the dynamics are stochastic and the value-iteration reference below ignores slip.
What is checked and why you can trust it
Every time the page loads, and whenever you press Re-run check, the simulation evaluates its own mathematics against an independent reference, such as a closed-form formula, a finite-difference derivative or an exhaustive count. Each line above reports PASS or FAIL together with the measured numbers, so you can see by how much it agrees. Results are also published on window.__msChecks for automated testing. The check never uses the same code path as the animation it validates, otherwise it would only confirm itself.
How to read a failure: a FAIL line means the measured discrepancy exceeded the stated tolerance for the current slider values. Try Reset, then move the sliders back toward their defaults; if a check still fails at the defaults it is a bug worth reporting. Tolerances are chosen from the numerical method itself, never tuned to make a test pass: round-off level (1e-9) for exact algebra, about 1e-3 for finite differences and Monte Carlo estimates with a few hundred thousand samples, and a documented margin for stochastic learning runs.
Reproducibility: the particle and pen simulations use the browser random generator for initial positions, so exact pixels differ between visits, but the checks depend only on mathematics and not on those positions. Fixed wave tables (curl noise) and fixed grammars (L-system) are deterministic.
This is an educational model. It shows the mechanism clearly, but it is deliberately simplified; the science panel lists what was left out.