Neural Network Training — Loss Landscape & Gradient Descent

Dataset

Architecture

Hidden units per layer 16

Hyperparameters

Learning rate η 0.010

Batch size 32

L2 regularisation λ 0.001

Optimiser

Controls

Stats

Epoch

—

Train loss

—

Train acc

—

Val acc

L = −Σ[y log ŷ + (1−y)log(1−ŷ)]
SGD: w -= η·∇L
Adam: bias-corrected m̂/√v̂ updates
Overfitting: train↓ but val↑