AI & Machine Learning ★★★ Advanced

🤖 Reinforcement Learning — Q-Learning Grid World

Train an agent to navigate a grid world using Q-learning. The agent learns Q(s,a) values via the Bellman equation and ε-greedy exploration, converging to an optimal policy. Visualise the value function and policy arrows.

0.20
0.90
0.30
10
Episode: 0 Steps: 0 Total reward: 0 ε (current): 0.30 Best run: —

How to read the grid

Cell brightness encodes the maximum Q-value (state value). Arrows show the greedy policy direction. Green = goal (+10). Red = trap (−5). Yellow = agent. Walls are dark blue. The agent decays ε by 0.5% each episode so exploration gradually gives way to exploitation.

The Physics

Bellman update: Q(s,a) ← Q(s,a) + α·[r + γ·maxa' Q(s',a') − Q(s,a)]. α = learning rate, γ = discount factor. ε-greedy: choose random action with probability ε, best-known action with 1−ε. As ε → 0 over time, agent exploits learned Q-values. Converges to optimal Q* when all (s,a) visited infinitely often (tabular MDP, ergodic).

About Q-Learning Grid World

This simulation trains an agent to find an optimal route across a grid world using Q-learning, a model-free reinforcement learning method. The world is a Markov decision process with a goal cell (+10), trap cells (−5) and a small step penalty (−0.02). The agent stores a table of action values Q(s,a) and refines them with the Bellman update Q(s,a) ← Q(s,a) + α[r + γ·max Q(s',a') − Q(s,a)] after every move.

The sliders set the learning rate α (how strongly each experience updates Q), the discount γ (how much future reward counts), the exploration rate ε (chance of a random move) and the simulation speed. Cell brightness shows the maximum Q-value and arrows show the greedy policy, so you watch a value map and a route emerge from scratch. The same algorithm underpins game-playing AI, robotics and recommendation systems.

Frequently Asked Questions

What is Q-learning?

Q-learning is a model-free reinforcement learning algorithm that learns the value of taking each action in each state. It builds a table of Q(s,a) values purely from trial and reward, with no prior model of the environment, and eventually those values point to the best action everywhere.

What is the agent trying to do here?

The yellow agent starts at the top-left corner and tries to reach the green goal cell, which gives a reward of +10. Red traps give −5 and end the episode, and every step costs −0.02, so the agent is pushed to find the shortest safe path.

What does the Bellman update do?

After each move the agent applies Q(s,a) ← Q(s,a) + α[r + γ·max Q(s',a') − Q(s,a)]. The bracket is the temporal-difference error: the gap between the reward plus the best discounted future value and the current estimate. The learning rate α controls how much of that error is absorbed.

What do the learning rate, discount and exploration sliders change?

The learning rate α (0.01–1) sets how fast Q-values move toward new estimates; high values learn quickly but can be unstable. The discount γ (0.1–0.99) weights future reward, so values near 1 plan further ahead. Exploration ε (0–1) is the probability of choosing a random action instead of the current best.

What is the ε-greedy strategy?

With probability ε the agent picks a random action to explore, and with probability 1−ε it picks the action with the highest known Q-value to exploit what it has learnt. In this page ε starts at the slider value and decays by 0.5% per episode, so the agent explores boldly early on and settles into exploitation later.

What do the colours and arrows on the grid mean?

Each cell's brightness encodes its maximum Q-value, so brighter cells are more valuable. Arrows show the greedy policy direction from that cell once its value is positive. The green star is the goal (+10), red crosses are traps (−5), dark-blue cells are walls and the yellow dot is the agent.

Why does the agent appear to wander early on?

At the start every Q-value is zero, so the agent has no idea where the goal is and explores almost at random, especially with a high ε. As rewards propagate backwards through the Bellman update, a value gradient forms toward the goal and the wandering gives way to a clear, purposeful path.

Is this a physically or mathematically accurate model?

Yes for the idealised case it represents. Tabular Q-learning is proven to converge to the optimal action-value function Q* in a finite Markov decision process, provided every state-action pair is visited infinitely often and the learning rate decays appropriately. The grid here is a faithful small MDP, though it uses a fixed step count rather than formal decay schedules.

What is the difference between the value function and the policy?

The value function tells you how good each state is, shown here as cell brightness from the maximum Q-value. The policy tells you what to do, shown as the greedy arrows. A good value function makes a good policy easy to read off: simply move toward the neighbouring cell with the highest value.

Where is reinforcement learning used in the real world?

The same principles drive game-playing systems such as AlphaGo and Atari agents, robot control and locomotion, traffic-light and energy management, recommendation engines and the fine-tuning of large language models. Grid worlds like this one are the classic teaching environment because they make the value map and policy easy to visualise.