Real REINVENT trains a recurrent policy πθ to emit a molecule token by token (as SMILES characters), then nudges it toward high-scoring chemistry with the augmented-likelihood loss:
AL(m) = log P_prior(m) + σ · S(m)
Loss = ( AL(m) − log π_θ(m) )²
This sim implements the reinforcement-learning half of that loop directly: a token-selection policy (one probability vector over 6 actions — add C / add N / add O / branch / close ring / stop) grows a molecule atom by atom, laid out as a 2D graph you can pan and zoom. Each finished molecule gets a scoring-function reward
S = 0.5·size_fit + 0.35·hetero_fit + 0.15·has_ring
size_fit = 1 − |atoms − pocket_target| / pocket_target
hetero_fit = 1 − |N,O_frac − 0.25| / 0.25
and the policy is updated with the REINFORCE gradient, scaled by σ:
Δθ ∝ σ/1000 · (R − baseline) · (one_hot(action) − π_θ(action))
Actions that led to above-baseline reward become more likely next batch; below-baseline actions become less likely — exactly the credit assignment used to fine-tune REINVENT's SMILES RNN against a docking/ADMET scoring function, just with a compact hand-built policy instead of an LSTM. The bar chart below the graph is the live policy πθ over the six actions — watch it sharpen as training pushes probability mass toward whatever the reward rewards.
- Target pocket size — shifts the ideal atom count the reward rewards, standing in for a docking pocket's real-world volume; the dashed target ring on the graph tracks it live.
- σ — how strongly the score dominates the augmented likelihood; higher σ sharpens and speeds up policy updates.
- Molecules per batch — REINFORCE reward is measured relative to the batch's own mean (the baseline), so batch size trades off gradient noise against wall-clock speed.