📉 Gradient Descent Visualiser

Gradient descent is an iterative method for minimising a function by repeatedly stepping in the direction of steepest decrease, the negative gradient. Each update is parameter ← parameter − learning_rate × gradient. In machine learning it is the engine that adjusts model weights to reduce a loss, and this page makes that process visible on a 2D-input surface drawn in 3D.

SGD takes a fixed-size step along the raw gradient. Momentum accumulates a velocity so it builds speed in consistent directions and damps oscillation. RMSprop divides each step by a running average of squared gradients, giving every axis its own adaptive scale. Adam combines momentum and RMSprop-style scaling with bias correction, which is why it often converges fastest here.

The learning rate sets how far each step moves; too small and convergence crawls, too large and the marker overshoots or diverges. Gradient Noise adds random perturbation proportional to the gradient magnitude, mimicking the stochastic gradients of mini-batch training and showing how each optimiser copes with imperfect direction estimates.

Yes for the core mechanics. The gradients of all four loss functions are coded analytically, and the four update rules implement the standard equations, including Adam's bias-corrected moments with β₂ fixed at 0.999. It is a faithful 2-parameter teaching model rather than a full deep network, so it omits mini-batches, weight decay and learning-rate schedules.

On the saddle surface the gradient nearly vanishes along one direction, so plain SGD can stall while adaptive methods escape faster. On Himmelblau there are four separate minima, so the starting point decides which basin a path falls into. These behaviours illustrate that gradient descent finds a local minimum near where it begins, not necessarily the global one.