A simulated patient has a discretized tumor-burden level (0 = cured … 5 = severe) and a toxicity level (0 = none … 4 = dose-limiting). Each step the agent picks a dose a ∈ {none, low, medium, high}. The environment updates stochastically:
tumor' = clamp(tumor − killRate(a) + noise, 0, 5)
tox' = clamp(tox + toxRate(a) − recovery, 0, 4)
reward = −(tumor' + toxAversion · tox')
+ 50 if tumor' = 0 (cured, episode ends)
− 50 if tox' ≥ 4 (dose-limiting toxicity, episode ends)
The agent has no model of these dynamics — it learns a state–action value table Q(s,a) purely from trial and error via the tabular Q-learning (Bellman) update:
Q(s,a) ← Q(s,a) + α·[ r + γ·max_a′ Q(s′,a′) − Q(s,a) ]
with learning rate α, discount γ = 0.9, and ε-greedy action selection: with probability ε it explores a random dose, otherwise it exploits the currently best-known dose. ε starts at 1.0 and multiplies by the decay factor after every episode, so exploration fades as the policy improves.
- Value grid — one tile per (tumor, toxicity) state, arranged tumor (rows, top = cured) × toxicity (columns, left = none); the fill height inside each tile is the best known Q-value there, colour is the currently best action.
- Agent marker — the white marble is the agent's current state; it glides to the next tile as each simulated day passes.
- Click any tile to lock the state inspector onto it and see the learned value of all four doses side by side; click it again (or hit the toggle button) to release the lock and follow the agent live.
- Toxicity aversion — scales how harshly the reward punishes toxicity, letting you push the learned policy from aggressive to conservative dosing.
This mirrors real reinforcement-learning approaches to adaptive treatment regimes and closed-loop dosing research, where a policy is learned from simulated or real patient trajectories rather than hand-coded rules — always as a decision-support aid, never as an unsupervised prescriber.