Reinforcement Learning: Teaching an Agent to Act by Trial and Error

The loop: agent, environment, reward

Reinforcement learning is not about labelled examples the way supervised learning is. There is no dataset of "correct" moves to imitate. Instead there is an agent sitting inside an environment, and a loop that repeats every time step: the agent observes the current state, picks an action, and the environment responds with a reward and a new state. Nothing in that loop tells the agent which action was best — it only ever finds out what happened, never what would have happened if it had chosen differently.

The agent's strategy is called a policy, written π(s) — a mapping from states to actions. The entire goal of reinforcement learning is to find the policy π* that maximises the total reward accumulated over time, and to find it purely by trial and error rather than by being told the answer.

Markov decision processes and discounted return

The formal object underneath this loop is a Markov Decision Process: a state space, an action space, transition probabilities, a reward function and a discount factor γ. The name comes from the Markov property — the next state depends only on the current state and action, never on the history that led there. That single assumption is what makes the whole problem tractable: the agent never needs to remember how it arrived somewhere, only where it is.

Because a reward now is worth more than the same reward later, returns are discounted:

G_t = r_t + γ·r_(t+1) + γ²·r_(t+2) + ... = Σ_k γ^k · r_(t+k)

With γ = 0.99, a reward 100 steps away is worth only 0.99¹⁰⁰ ≈ 0.37 of the same reward right now. γ close to 0 makes the agent short-sighted; γ close to 1 makes it plan far ahead but demands far more data before the value estimates settle down. Typical practical values sit between 0.9 and 0.999.

Value functions and the Bellman equation

Two related quantities do all the work. The state-value function V(s) is the expected discounted return starting from state s and following policy π. The action-value function Q(s, a) is the expected discounted return starting from state s, taking action a, and following π afterwards. If you know the optimal Q*(s, a) for every state-action pair, the optimal policy falls out for free — always take the action with the highest Q-value.

Richard Bellman's contribution was to notice that Q* is self-consistent: the value of being in state s and taking action a equals the immediate reward plus the discounted value of behaving optimally from whatever state comes next.

Bellman optimality equation:
Q*(s, a) = E[ r + γ · max_a' Q*(s', a') ]

// "the value of (s,a) is the reward you get now,
//  plus the discounted value of playing perfectly from s' onward"

This turns "find the optimal policy" into a fixed-point problem: start with any guess for Q, repeatedly apply the Bellman update, and under mild conditions the guess converges to Q*. That single equation is the mathematical seed of almost every reinforcement learning algorithm in use today.

Q-learning: turning the equation into an algorithm

Q-learning is what happens when you apply the Bellman equation using sampled experience instead of a known model of the environment. After every (s, a, r, s′) transition, the current estimate is nudged toward the Bellman target by a learning rate α:

Q(s,a) ← Q(s,a) + α · [ r + γ · max_a' Q(s',a') − Q(s,a) ]
                        \_____________ TD error ____________/

The bracketed term is the temporal-difference (TD) error — how wrong the current estimate is relative to the one-step Bellman target. Because Q-learning is model-free, it never needs to know the environment's transition probabilities or reward function; it just needs to be able to act and observe the outcome. For a small, discrete grid like a maze, the Q-values live in a plain table indexed by (state, action); for continuous or high-dimensional states, the table is replaced by a function approximator such as a neural network — a Deep Q-Network, stabilised with a replay buffer of past transitions and a slowly-updated target network so the learning target doesn't move under the agent's feet every step.

Exploration versus exploitation

A purely greedy agent always takes the action with the highest current Q-value. Early in training those estimates are little better than guesses, so blind greed can trap the agent in a mediocre routine it never questions again. The standard fix is ε-greedy exploration: with probability ε take a uniformly random action, otherwise take the greedy one. ε typically starts near 1.0 (mostly random, to map out the space) and is annealed down toward something small like 0.05 as the Q-values become trustworthy.

function selectAction(Q, state, epsilon):
  if random() < epsilon:
    return randomAction()          // explore
  return argmax_a Q[state][a]      // exploit

How the maze simulation here uses it

The simulation on this site runs tabular Q-learning on a grid maze in real time: every cell is a state, every move is an action, and the reward is a small step penalty plus a large bonus for reaching the goal. The three sliders you can drag — ε, α and γ — are exactly the three quantities above, so watching the agent's behaviour change as you move them is watching the exploration/exploitation trade-off, the learning-rate trade-off, and the planning-horizon trade-off happen live rather than staying abstract on a page. Push ε to zero and the agent freezes into whatever policy it currently has, good or bad; push γ toward 1 and it starts taking longer, more roundabout-looking paths that pay off further down the line.

Frequently asked questions

Try it live

Everything above runs in your browser — open Reinforcement Learning and drag ε, α and γ while the agent is training. Nothing is installed, nothing is uploaded, the whole model lives in one tab.