This visualiser drops one or more optimisers onto a 3D loss landscape and animates the iterative path each one carves towards a minimum. At every step the true gradient of a chosen test function is computed analytically, optional noise is injected, and the parameter update rule for SGD, Momentum, RMSprop or Adam moves the marker downhill. Watching the four trails race across surfaces such as the Rosenbrock valley reveals why adaptive methods handle curvature, narrow ravines and saddle points so differently.
A rendered height field of a classic optimisation benchmark (Rosenbrock, a saddle, Beale's function or Himmelblau). Coloured dots descend it using real update equations: plain SGD (x ← x − lr·g), Momentum, RMSprop's per-axis squared-gradient scaling, and Adam with its bias-corrected first and second moments (β₁, β₂ = 0.999). The white ★ marks the global minimum each path is chasing.
Pick an optimiser (or "All four" to compare) and a loss function from the dropdowns. The Learning Rate slider (0.001–0.2), Momentum / β₁ slider (0–0.99) and Gradient Noise slider (0–0.5) reshape the dynamics live. Use Reset to restart, Pause to freeze, and drag to rotate or scroll to zoom the camera. The telemetry panel reports step count, loss, gradient norm and per-step displacement.
The Rosenbrock "banana" function has a global minimum at (1, 1) that sits inside a long, curved, nearly flat valley. Following the gradient slides you into the valley almost instantly, but inching along its floor to the true minimum is notoriously slow — which is exactly why it became a standard stress test for optimisation algorithms.
Gradient descent is an iterative method for minimising a function by repeatedly stepping in the direction of steepest decrease, the negative gradient. Each update is parameter ← parameter − learning_rate × gradient. In machine learning it is the engine that adjusts model weights to reduce a loss, and this page makes that process visible on a 2D-input surface drawn in 3D.
SGD takes a fixed-size step along the raw gradient. Momentum accumulates a velocity so it builds speed in consistent directions and damps oscillation. RMSprop divides each step by a running average of squared gradients, giving every axis its own adaptive scale. Adam combines momentum and RMSprop-style scaling with bias correction, which is why it often converges fastest here.
The learning rate sets how far each step moves; too small and convergence crawls, too large and the marker overshoots or diverges. Gradient Noise adds random perturbation proportional to the gradient magnitude, mimicking the stochastic gradients of mini-batch training and showing how each optimiser copes with imperfect direction estimates.
Yes for the core mechanics. The gradients of all four loss functions are coded analytically, and the four update rules implement the standard equations, including Adam's bias-corrected moments with β₂ fixed at 0.999. It is a faithful 2-parameter teaching model rather than a full deep network, so it omits mini-batches, weight decay and learning-rate schedules.
On the saddle surface the gradient nearly vanishes along one direction, so plain SGD can stall while adaptive methods escape faster. On Himmelblau there are four separate minima, so the starting point decides which basin a path falls into. These behaviours illustrate that gradient descent finds a local minimum near where it begins, not necessarily the global one.
Gradient descent is the workhorse optimization algorithm of machine learning, used to minimize a loss function L(θ) by iteratively moving the parameters θ in the direction of the negative gradient: θ ← θ − α∇L(θ), where α is the learning rate. The gradient ∇L(θ) points in the direction of steepest ascent of the loss surface; subtracting it steps downhill toward lower loss values. For a convex loss function (like mean squared error in linear regression), gradient descent converges to the global minimum; for non-convex functions (like deep neural network losses), it converges to a local minimum or saddle point.
The learning rate α is the most critical hyperparameter: too large and the algorithm overshoots minima, oscillating or diverging; too small and convergence is extremely slow. Adaptive learning rate methods (Adam, RMSProp, AdaGrad) automatically adjust the learning rate for each parameter based on historical gradients, dramatically improving convergence. Stochastic Gradient Descent (SGD) computes gradients on random mini-batches rather than the full dataset, adding noise that paradoxically helps escape sharp local minima and find flatter, better-generalizing minima in deep learning.
This simulator visualizes gradient descent on 2D and 3D loss landscapes, showing the trajectory of iterates from a starting point. You can choose different landscape shapes (bowl, saddle, multi-modal), adjust learning rate, compare SGD vs. Adam vs. momentum variants, and observe how landscape curvature and starting point affect convergence behavior—building intuition for the optimization challenges at the heart of training neural networks.
Why do we use gradient descent instead of directly solving for the minimum?
For many problems, the minimum of the loss function can be found analytically by setting the gradient to zero and solving the resulting system of equations—this is the normal equations solution for linear regression. However, for neural networks with millions of parameters and non-linear activations, the system of equations is non-linear and too large to solve analytically. Gradient descent is an iterative approximation that scales to any parameter count and works for any differentiable loss function, making it universally applicable across machine learning, while exact solutions are limited to small, simple problems.
What is the difference between batch, mini-batch, and stochastic gradient descent?
Batch gradient descent computes the exact gradient using all N training examples, giving the true gradient direction but requiring a full pass through the dataset per step—too slow for large datasets. Stochastic gradient descent (SGD) uses a single random example per step, giving a noisy gradient estimate but updating parameters N times per epoch. Mini-batch gradient descent (the standard in deep learning) uses batches of 32–512 examples, balancing noise reduction (more stable than single-example SGD) with computational efficiency (vectorized GPU computation). The noise in SGD and mini-batch SGD can help escape sharp local minima and find flatter loss landscapes.
What are saddle points and why are they problematic?
A saddle point is a critical point (gradient = 0) that is a local minimum in some directions and a local maximum in others—like the center of a saddle. In high-dimensional spaces (millions of parameters), saddle points are far more common than local minima: a local minimum requires the loss to curve upward in all directions simultaneously, an exponentially rare event. Gradient descent can get stuck near saddle points where the gradient is near zero, slowing convergence dramatically. Stochastic noise, momentum, and second-order methods all help navigate around them.
Momentum accumulates a velocity vector in the direction of past gradients, allowing the optimizer to build speed in consistent downhill directions and dampen oscillations across steep ravines. The update rule becomes: v ← β·v − α·∇L; θ ← θ + v, where β (typically 0.9) controls how much past gradients are retained. In a narrow ravine (high curvature in one direction, low in another), standard gradient descent oscillates back and forth across the ravine while making slow progress along it. Momentum averages out the cross-ravine oscillations while amplifying the along-ravine component, achieving much faster convergence.
Adam (Adaptive Moment Estimation) combines momentum (first moment of gradients) with RMSProp (second moment—mean squared gradient). It maintains per-parameter estimates of gradient mean (m) and variance (v), updating: m ← β₁m + (1-β₁)g; v ← β₂v + (1-β₂)g²; θ ← θ − α·m̂/√(v̂+ε), where m̂ and v̂ are bias-corrected estimates. Adam adapts the effective learning rate for each parameter: parameters with consistently large gradients get smaller learning rates; rarely updated parameters get larger learning rates. This makes Adam robust to hyperparameter choice and highly effective for sparse gradients and non-stationary problems.