A neural network's loss function assigns a number — how wrong the model is — to every possible setting of its weights. With only two weights this "loss landscape" can be drawn as a 3D terrain, where height is loss. Training is a ball rolling downhill: at each step it looks at the local slope (the gradient) and steps a little in the downhill direction.
∇L) and moves the ball opposite that slope, scaled by the learning rate.Saddle points, not local minima, are now believed to be the bigger obstacle in training large neural networks — in high dimensions almost every flat critical point is a saddle, and momentum-based optimizers like Adam and SGD-with-momentum exist largely to carry the "ball" through them.
An interactive 3D loss-surface terrain: drop a ball anywhere on the landscape and watch gradient descent roll it downhill, step by step, toward a minimum — or into a valley, or stuck at a saddle.
Each terrain height is a loss value; the ball's motion is literal gradient descent — step opposite the local slope, scaled by the learning rate, with momentum carrying velocity forward between steps.
Pick a surface shape, click anywhere on the terrain to drop the ball, then tune learning rate and momentum to see smooth convergence, valley zig-zagging, overshoot, or a stall near a saddle point.
In the high-dimensional weight spaces of real neural networks, saddle points vastly outnumber true local minima — which is exactly why momentum-based optimizers were invented.