Hindsight Experience Replay: Robot Reaching with Sparse Rewards (2D)

Two identical 2-link robot arms try to reach randomly placed targets using the same sparse binary reward and the same REINFORCE policy-gradient optimizer. The only difference: the blue arm applies Hindsight Experience Replay, relabeling every attempt's actual end position as if it had been the intended goal, so it always has a useful trajectory to learn from. The orange arm gets no such help — it only ever learns from the rare episodes where it happens to land on the real goal by chance. This 2D view draws each arm's reaching plane directly, exactly as its own kinematics compute it, with no 3D projection in between. Watch the live success-rate chart and the growing gap between the two curves, exactly the effect that makes HER a standard technique for training real robotic manipulators under sparse, goal-based rewards.