Both arms are 2-link manipulators using the exact same architecture — a linear-Gaussian policy π(a|s) = 𝒩(θ·s + b, σ²) where s is the direction from the end-effector to the goal, in the arm's own 2D reaching plane. Both learn from the same sparse reward: r = 1 only if the arm reaches within the goal threshold by the end of the episode, otherwise r = 0.
REINFORCE update (both arms):
θ ← θ + α (G − b̄) Σₜ ∇θ log π(aₜ|sₜ)
∇θ log π(a|s) = (a − μ)/σ² · sᵀ
G = 1 if goal reached, else 0
b̄ = running average return (baseline)
With a sparse reward, most random episodes fail completely — G = 0 for almost every rollout early on, so the baseline arm gets almost no gradient signal and learns very slowly.
Hindsight Experience Replay ("final" strategy,
Andrychowicz et al. 2017):
After every episode, relabel the goal with
the position the arm ACTUALLY reached:
g' = end-effector position at final step
Because g' was reached by construction, this
relabeled episode always looks like a near-success —
giving the HER arm a usable learning signal
on every single episode, not just the rare
real successes.
- Learning rate — step size of the policy-gradient update.
- Exploration σ — standard deviation of the Gaussian action noise; too low and neither arm ever stumbles onto a goal, too high and the update is dominated by noise.
- HER relabel weight — the fixed advantage used for the relabeled-goal update; higher values push the HER arm's policy toward wherever it actually ended up, faster.
- Goal threshold — how close the end-effector must get to count as a real success; smaller thresholds make the sparse-reward problem harder and widen the HER/baseline gap.
- Reset weights — restarts both policies from small random weights so you can watch the success-rate gap open up again from scratch.
This is a simplified, browser-sized version of the real mechanism behind HER-based robotic manipulation training (used in real sim-to-real reaching and grasping pipelines): identical optimizer, identical reward sparsity, the only difference is whether failed trajectories get reinterpreted as successes toward a different goal. This 2D view plots each arm's own reaching plane directly — the same plane its kinematics and policy actually operate in — with no projection or scene dressing involved.