The science
The model. A multilayer perceptron maps the features x (the point coordinates, optionally x², y², xy, sin) through layers al = φ(Wl al-1 + bl) to one output logit z. The class probability is the logistic function:
p = σ(z) = 1 / (1 + e-z)
The loss. For labels y in {0,1} the binary cross-entropy, averaged over the N training points, plus an optional weight penalty (biases are not penalised):
J(θ) = (1/N) Σ [ -y ln p - (1-y) ln(1-p) ] + λ · ( ½Σw² or Σ|w| )
It is evaluated in the stable form max(z,0) - z y + ln(1 + e-|z|). A network that outputs p = ½ everywhere has J = ln 2 = 0.693 (the grey line in the loss graph).
Backpropagation. With δL = (p - y)/N at the output (sigmoid and cross-entropy cancel) the error is pushed backwards, δl-1 = (WlT δl) ⊙ φ′, and the gradients are ∂J/∂Wl = δl al-1T plus λ w (L2) or λ sign(w) (L1). The first numerical check compares this with central finite differences.
Optimisers. Gradient descent θ ← θ - η g; with momentum m ← βm + g, θ ← θ - η m; Adam keeps bias-corrected moments θ ← θ - η m̂ / (√v̂ + ε), so its first step has size η. On a quadratic J = (k/2)θ² plain descent gives exactly θn = (1 - ηk)nθ0; it diverges when ηk > 2.
Generalisation. The network is fitted to the training points only. The test points are never used for updates, so the gap Jtest - Jtrain estimates how well the rule transfers. A model that is too simple (high bias) has both losses high; a model with too much freedom for too few noisy points (high variance) drives the training loss to zero while the test loss rises again. A penalty λ shrinks the weights, makes the boundary smoother and usually gives a lower test loss at the end of training.
The loss landscape (3D page). The real loss depends on all weights (tens to hundreds), which cannot be drawn. Following Li et al. (2018) two random directions d1, d2 are drawn in weight space, each neuron's part of the direction rescaled to the norm of that neuron's weights ("filter normalisation", so that scale does not distort the picture), and d2 is made orthogonal to d1. The surface is
f(α,β) = J( θc + α d1 + β d2 ), α,β in [-R, R]
where θc are the weights when the slice was started. Its slope at the centre is exactly the projected gradient, ∂f/∂α = ∇J · d1 (checked numerically), so the green arrow (the current minibatch gradient projected on the slice) points downhill on the drawn surface. The chrome ball sits at the projection of the current weights, with its height equal to the true current objective, so it can float above or below the surface when the weights have left the slice plane; the white line is the path of the most recent weights. Height in metres is the loss times the "height per unit of loss" slider (clipped at 4 m).
Honest simplifications. A two-dimensional slice of a high-dimensional surface can hide saddle points and flat directions and its shape depends on the random directions; in Live mode the slice is recomputed (new directions) after every pass. The data are synthetic two-dimensional toys, the networks are tiny, there is no bias-variance decomposition over repeated data draws, and early stopping is only indicated by the dotted marker at the best test epoch. Nothing here is a statement about large language models or any real product.
Checks (run in the page). (1) backprop against finite differences on 14 weights, (2) chance loss ln 2 for an all-zero network, (3) plain descent on a quadratic against (1 - ηk)n, (4) Adam's first step, (5) the landscape slope at the centre against ∇J · d1, (6) the loss on separable blobs falls below a quarter of its start value; the 3D page adds (7) the slice centre against the objective at the weights.