The attacker is modelled as a reinforcement-learning agent moving through a directed graph of hosts (state = current host). Each edge is a lateral-movement step with an effort cost and a base detection risk. Taking action a (moving to host s′) from state s gives reward r = −cost, plus −25 if the defender's monitoring catches the hop and +40 on reaching the crown-jewel target. The agent updates its estimate of every state–action pair with the Q-learning / Bellman update:
Q(s,a) ← Q(s,a) + α · [ r + γ · max_a′ Q(s′,a′) − Q(s,a) ]
α (learning rate) controls how fast new experience overwrites old estimates; γ = 0.9 (fixed) discounts future reward. During training the agent picks moves with an ε-greedy policy — a random hop with probability ε to keep exploring, otherwise the highest-Q hop found so far.
- Defender vigilance — multiplies every edge's base detection risk. Push it up and the agent is forced to learn cheaper-but-riskier shortcuts are no longer worth it, shifting its policy toward slower, quieter paths.
- Edge colour/width — brightness and thickness encode the learned Q-value for that hop; as training converges, one bright path from entry to target emerges.
- Live agent token — continuously replays the current greedy policy in real time, one hop at a time, so you can see the strategy change as training and the vigilance slider move.
- Bottom strip — a rolling reward sparkline (last 50 episodes) and a per-edge Q-value bar chart, so convergence is visible as data, not just as glow.
Real-world relevance: this is the same objective automated red-teaming and attack-graph-based penetration-testing tools optimise for — and, mirrored, it is exactly the cost/detection trade-off a defensive AI system (SOC monitoring, segmentation policy) is tuned against.
This 2D build computes the identical Q-learning process independently in its own engine (verified numerically against the 3D original) and renders it as a pannable/zoomable 2D graph plus two data panels, instead of an orbiting 3D scene.