The purple surface is a loss landscape with one broad global minimum plus small local wrinkles (a "sharp minima" region). Each step, a random mini-batch of size B is drawn from the dataset (the small dots on the ground plane light up gold) and used to estimate the gradient. The ball rolls downhill following that noisy estimate.
grad_estimate = true_grad + noise / sqrt(B)
updates_per_epoch = dataset_size / B
step: theta -= lr * grad_estimate
- Batch size — small B (1-8) gives a noisy, jittery path that can escape sharp wrinkles but converges unevenly; large B (256+) gives a smooth, stable path that can get stuck in the nearest wrinkle.
- Learning rate — scales each step; too high with a small batch causes overshoot.
- Dataset size — changes how many updates fit in one epoch for a given batch size.
- Reset run — restarts the ball from a fixed high-loss point on the landscape.
This mirrors the real trade-off: bigger batches are more stable and use hardware efficiently per step, but take fewer, smoother steps per epoch and can generalize worse; smaller batches add helpful exploration noise at the cost of stability and wall-clock efficiency.