🤖 Q-Learning in a Grid World

About this simulation

An agent (the glowing sphere) lives on a 6×6 grid. Each episode it starts at the blue tile and tries to reach the green goal tile while avoiding the two red pits and the dark wall blocks. It has no map — only a Q-table of estimated values for every (state, action) pair, which it updates after every single step using the Bellman equation.

How it works

• Exploration vs. exploitation — with probability ε the agent takes a random action; otherwise it picks the action with the highest known Q-value for its current tile.
• Bellman update — after moving, it revises its estimate: Q(s,a) += α · (r + γ · max Q(s') − Q(s,a)).
• Tile color shows the current best Q-value for that state — from dull red (poor) through neutral to bright cyan/green (excellent), while translucent cones show the argmax "policy" direction.
• Reaching the goal gives +10, falling in a pit gives −10 and ends the episode, bumping into a wall costs −1, and every ordinary step costs a small −0.05 to encourage shorter paths.

Controls

• Learning rate α — how much each new experience overwrites the old estimate.
• Discount γ — how much the agent values future reward versus immediate reward.
• Exploration ε — probability of taking a random action instead of the current best guess.
• Step speed — how many grid steps the agent takes per second.
• Show policy arrows — toggle the cones that show the greedy action learned so far in every tile.
• ↺ Reset Q-table — wipes all learned values and starts fresh. Drag to rotate, scroll to zoom.

Did you know?

Q-learning is off-policy: it always bootstraps toward the best possible next action, even while the agent itself is busy exploring randomly. That single design choice is what lets the same algorithm learn a correct, optimal policy even though most of the moves it actually makes during training are deliberately imperfect.