Batch Size Impact | mysimulator.uk

SGD Training Dynamics

Batch size 32

Learning rate 0.060

Steps / second 24

Gradient noise σ/√B

–

Step

–

Train loss

–

Est. generalization gap

–

Basin found

searching…

How it works

The surface is a real loss landscape L(x,y) with two minima of similar depth: a narrow, steep well and a wide, shallow one. A ball rolls down it by mini-batch gradient descent. Each mini-batch of B samples estimates the true gradient by averaging B noisy per-sample gradients, so the estimator's variance shrinks as 1/B — its standard deviation shrinks as 1/√B:

ĝ = ∇L(θ) + N(0, σ²/B)
θ ← θ − η·ĝ
sharpness ≈ ∇²L(θ)  (local curvature)

Small batch → large √B-scaled noise. The random kicks are big relative to the narrow well's basin, so the ball is repeatedly knocked back out until it drifts into the wide, flat basin — which tolerates noise because its walls are gentle.
Large batch → tiny noise. The ball follows the true gradient almost exactly and drops straight into whichever minimum it started closest to, often the sharp one — which can even reach a lower training loss than the flat one.
Generalization gap here is estimated directly from the landscape's local curvature (a finite-difference Laplacian at the ball's position) — the empirical finding behind this model (Keskar et al., 2016) is that sharp minima have a larger gap between train and test loss than flat ones, because small perturbations to the weights cost much more loss where curvature is high.
Learning rate and steps/second scale the step size and playback speed without changing the underlying noise mechanics.

This is a simplified 2D model, not a literal neural-network loss surface — but the mechanism (gradient variance ∝ 1/B, and its interaction with curvature) is the real math behind why batch size shows up as a generalization knob in practice, not just a speed knob.

SGD iterate Sharp minimum Flat minimum

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.