The agent lives in a gridworld: a Markov decision process where each cell is a state, and up/down/left/right are actions. It picks actions ε-greedily — mostly the best-known action, but a random one with probability ε to keep exploring — then updates its value estimate with the Q-learning (Bellman) rule:
Q(s,a) ← Q(s,a) + α [ r + γ · max_a′ Q(s′,a′) − Q(s,a) ]
- Learning rate α — how much each new experience overwrites the old value estimate.
- Discount γ — how much future reward counts versus the immediate step; higher γ makes the agent plan further ahead.
- Exploration ε — the chance of taking a random action instead of the greedy one (the exploration/exploitation trade-off).
- The rising bars show max_a Q(s,a) per cell — the agent's learned estimate of how good each state is. Watch a ridge of value form along the shortest route to the goal as episodes repeat.
This tabular update is the ancestor of Deep Q-Networks (DQN), and the same reward-driven loop — with a learned value/critic instead of a table — underlies Actor-Critic methods, PPO, and RLHF, where a reward model replaces the gridworld's fixed reward function to fine-tune large language models.