The agent is a linear softmax policy πθ(a|s) over two actions (push left / push right), trained with REINFORCE — the Monte-Carlo policy-gradient theorem:
∇_θ J(θ) = E_π[ Σ_t ∇_θ log π_θ(a_t|s_t) · (G_t − b) ]
G_t = Σ_{k≥t} γ^(k−t) r_k (discounted return from step t)
π_θ(a|s) = softmax(Ws + c) (2-action linear policy)
After every episode the agent replays its own trajectory, computes each step's discounted return Gt, and nudges the weights in the direction that makes actions followed by high return more likely — exactly ∇log π scaled by (Gt − b). No reward model, no value network: the gradient comes straight from the log-likelihood-ratio trick, ∇log π(a|s) = (𝟙[a] − π(·|s)), which only requires sampling and differentiating the policy itself.
- Baseline b — subtracting the episode's mean return from every Gt leaves the gradient's expectation unchanged (E[∇log π]=0) but cuts its variance, so learning is faster and steadier. Toggle it off to see raw high-variance REINFORCE.
- Learning rate α — step size of the gradient-ascent update W ← W + α·∇J. Too high causes the policy to overshoot and collapse; too low learns very slowly.
- Cart-pole dynamics — the classic Barto–Sutton–Anderson benchmark: cart mass 1 kg, pole mass 0.1 kg, pole half-length 0.5 m, ±10 N pushes, integrated with the standard nonlinear equations of motion. An episode ends when the pole tips past 12° or the cart leaves the 4.8 m track.
This is the same family of algorithm (Monte-Carlo policy gradient) behind early deep-RL results like AlphaGo's policy network pre-training and OpenAI's early continuous-control agents, before actor-critic methods added a learned value baseline.