Predator-Prey Q-Learning Pursuit

Learning Parameters

Learning rate α 0.30

Discount γ 0.90

Exploration ε 0.30

Steps per second 6

Live Stats

Episode

Steps this ep.

Avg. catch time (last 20)

—

Caught total

How it works

The red predator is a tabular Q-learning agent. Its state is its own cell plus the prey's cell on an 8×8 grid (4,096 states × 4 actions). Every step it picks up/down/left/right, receives a reward, and updates one table entry with the Bellman equation:

Q(s,a) ← Q(s,a) + α · [ r + γ · max_a' Q(s',a') − Q(s,a) ]

Reward: −1 per step (encourages speed), +50 for catching the prey (same cell).
ε-greedy policy: with probability ε the predator picks a random action instead of the best-known one — this is the exploration/exploitation trade-off. Turn ε-decay on and it shrinks automatically as episodes pass, mirroring a real training schedule.
Prey plays a fixed evasive heuristic: it steps toward whichever open neighbor cell is farthest from the predator, with 20% random noise — it does not learn, so any improvement you see is entirely the predator's policy.
Heatmap on the floor colors every cell by max_a Q(cell, preyCell, a) — the predator's current estimate of how valuable it is to stand there given where the prey is right now. Watch it sharpen into a clear gradient toward the prey as episodes accumulate.

This is the same update rule and ε-greedy exploration used to train the original Atari-playing DQN agents, just with a table instead of a neural network standing in for Q.

steps-to-catch (learning curve)

Predator (Q-learning) Prey (evasive)

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.