This is the flat 2D counterpart of the 3D CartPole REINFORCE simulator: the exact same physics and the exact same policy-gradient training run independently here, but instead of drawing a cart and pole moving in a rendered scene, this page draws the agent's decision field directly — a native 2D grid, not a camera view of anything.
∇_θ J(θ) = E_π[ Σ_t ∇_θ log π_θ(a_t|s_t) · (G_t − b) ]
G_t = Σ_{k≥t} γ^(k−t) r_k (discounted return from step t)
π_θ(a|s) = softmax(Ws + c) (2-action linear policy)
The main panel is a heatmap over the pole's angle θ (horizontal axis) and angular velocity θ̇ (vertical axis), holding cart position/velocity at their centre values. Every cell's colour is literally π(push right | θ, θ̇) computed straight from the current policy weights — orange means the policy pushes right there, blue means it pushes left. The field is recomputed only when the weights actually change (once per finished episode), and the bright trail is the real trajectory of the current episode plotted in this same (θ, θ̇) plane, so you can watch the state wander through the learned field and get steered back toward θ=0 as training progresses.
- Dashed vertical lines — the ±12° angle at which an episode actually ends (the fail boundary), shown for reference; the grid extends a little beyond it for context.
- Baseline b — subtracting the episode's mean return from every Gt leaves the gradient's expectation unchanged but cuts its variance. Toggle it off to see raw high-variance REINFORCE.
- Learning rate α — step size of W ← W + α·∇J. Too high causes the policy to overshoot and collapse; too low learns very slowly.
- Cart-pole dynamics — the classic Barto–Sutton–Anderson benchmark: cart mass 1 kg, pole mass 0.1 kg, pole half-length 0.5 m, ±10 N pushes, integrated with the standard nonlinear equations of motion. An episode ends when the pole tips past 12°, the cart leaves the 4.8 m track, or 500 steps pass.
The thin strip along the bottom of the canvas is the one physical dimension the phase-plane can't show directly: the cart's actual position on its ±2.4 m track, so a track-boundary failure is still visible even though it isn't one of the field's two axes.