🧭 Gradient-Based Methods for Hyperparameter Optimization

About this simulation

The floor grid is a 2D hyperparameter space — one axis is a learning rate, the other a regularization strength, both on a log scale — and height is validation loss. Instead of treating each hyperparameter setting as an isolated, expensive trial the way grid search or random search do, gradient-based hyperparameter optimization estimates a hypergradient — how the validation loss would change if each hyperparameter nudged slightly — and takes a step directly downhill, the same way ordinary gradient descent tunes model weights.

How it works

• At every step the marker computes the local gradient of the loss surface and moves in the direction of steepest descent, scaled by the learning rate (η).
• Momentum (β) carries velocity from previous steps forward, letting the marker coast through shallow flats and small dips instead of stalling in them.
• Gradient noise mimics the fact that a real hypergradient is only ever an approximation (finite differences, unrolled training, or implicit differentiation all add error) — a little noise can jostle the marker out of a shallow local minimum.
• The translucent arrows form the gradient field: at every grid point they show the local downhill direction and steepness the optimizer would follow from there.
• Because gradient methods only ever see local slope, they can converge quickly to a good basin but still get trapped in a local minimum if the landscape is not convex — exactly what you can watch happen with momentum and noise both set low.

Controls

• Learning rate (η) — step size taken along the negative gradient each update.
• Momentum (β) — how much of the previous step's velocity carries into the next one.
• Gradient noise — random perturbation added to the estimated gradient, modeling approximate hypergradients.
• Show gradient field — toggles the arrow field showing downhill direction across the whole space.
• ⏸ Pause / ↺ Restart / 🎲 New landscape — pause the descent, restart the marker from a fixed corner, or generate a fresh loss surface. Drag to rotate, scroll to zoom.

Did you know?

Gradient-based hyperparameter optimization (e.g. hypergradient descent, implicit differentiation through the training loop) can tune dozens of hyperparameters at once with a cost close to a single extra training run — something grid search, random search and even Bayesian optimization struggle to do once the hyperparameter count grows much past ten or twenty.