A machine-learning model's parameters (its weights) are learned automatically from data. Its hyperparameters are the knobs a human sets beforehand that control how that learning happens. This scene turns gradient descent into a purple bead rolling down a 3D loss surface, so you can see three major hyperparameter categories act on the same climb.
learning rate sets the step size taken down the slope each update; momentum lets the ball carry velocity from previous steps, smoothing its path through bumps.batch noise stands in for batch size: small batches give noisy gradient estimates (a jittery path), large batches give smooth, stable ones.L2 regularization penalizes complexity, which here reshapes the jagged, multi-valley loss surface into a smoother, more convex bowl as it increases.
Each frame the bead's height and local slope are computed from the current landscape
function. Its velocity updates by v = momentum·v − learningRate·gradient,
with Gaussian noise scaled by the batch-noise slider added to the gradient estimate,
mimicking stochastic mini-batch training. The purple trail marks recent positions.
Hyperparameters are usually chosen before training via grid search, random search or Bayesian optimization — because unlike weights, they can't be learned by gradient descent on the training loss itself (changing them changes the very shape of the landscape being descended).
A purple bead performs gradient descent across a 3D loss landscape while learning rate, momentum, batch noise and L2 regularization — representatives of the optimization, data and regularization hyperparameter categories — reshape both the terrain and the path taken across it.
Optimization hyperparameters (learning rate, momentum) control step size and smoothing; data hyperparameters (batch size, modeled as noise) control gradient stochasticity; regularization hyperparameters reshape the landscape itself toward convexity.
Drag each slider and watch the bead's path change immediately: raise learning rate to see overshoot, raise momentum to see smoothing through bumps, raise noise for a jittery small-batch path, and raise regularization to flatten the valleys into a single bowl.
Hyperparameters can't be learned the same way as model weights, because changing them changes the shape of the loss surface itself — that's why they're tuned with grid search, random search or Bayesian optimization instead of gradient descent.