A small robot agent lives on a grid world with a goal tile (reward), a hazard tile (penalty), and walls it cannot pass through. It has no map and no instructions — only the ability to try an action, observe a reward, and remember what happened. Over many episodes it builds a Q-table: an estimate of how good each action is from each state. This is the core loop of reinforcement learning — agent, environment, action, reward, state — repeated until a good policy emerges.
max Q(s,a), the agent's current estimate of how valuable that tile is. Watch it "light up" outward from the goal as learning spreads.Q(s,a) ← Q(s,a) + α[r + γ·maxQ(s') − Q(s,a)].This exploration/exploitation trade-off, controlled here by ε, is the same dilemma behind ad recommendation systems, clinical trial design, and AlphaGo's early self-play — try something new, or use what already works?
A robot agent explores a 3D grid world with no map, only trial, error, and reward — building a Q-table of state-action values that reshapes the terrain into a visible value-function landscape as learning progresses.
The core reinforcement learning loop — agent, state, action, reward — and how the Q-learning update rule propagates value backward from the goal tile through the grid, one visited tile at a time.
Adjust learning rate, discount factor, and exploration rate, then watch the agent's episode reward and value-function heatmap evolve live. Switch to policy view to see the greedy action chosen at every tile.
Q-learning, introduced by Chris Watkins in 1989, needs no model of the environment at all — it learns purely from experienced (state, action, reward, next state) tuples, which is why it scales to problems far too complex to hand-code, like Atari games and robotic control.