Data science fits a model to samples by minimising a loss. Here the model is linear, y = w1 x1 + w2 x2 + b, the loss is the mean squared error, and the optimiser is gradient descent with momentum. In a physics reading the weights are the coordinates of a ball, the loss is a potential energy, and each step applies the force -grad L, with momentum beta acting as inertia (the heavy-ball method).
L(theta) = (1/n) sum (w1 x1 + w2 x2 + b - y)^2 = theta'A theta - 2c'theta + mean(y^2)
grad L = 2 (A theta - c), Hessian H = 2A
v <- beta v - eta grad L ; theta <- theta + v
stable iff eta lambda_max < 2 (1 + beta)
mode rate: z^2 - (1 + beta - eta lambda) z + beta = 0, rho = max |z|
- Condition number kappa = lambda_max / lambda_min sets how slow descent is along the flat valley while the steep direction caps the step size. Raise the feature correlation rho and the valley stretches.
- Momentum lets the ball coast through the valley: with beta near 0.9 the heavy ball needs far fewer steps than plain descent at the same stability margin.
- Step size is shown relative to its stability limit 2(1+beta)/lambda_max, so 1.0 is the edge: above it the ball is thrown out of the basin and the loss diverges.
- Noise: with Gaussian noise sigma the best achievable MSE is about sigma^2 (n-3)/n, and the recovered weights scatter around the hidden truth (0.7, -0.4, 0.2) by their standard errors.
Honest simplifications. The loss landscape drawn is the profile loss, with the intercept b re-optimised for every (w1, w2); the ball's own full loss (shown in the stats) is a little higher until b catches up. Noise and features are idealised i.i.d. Gaussians. Real machine-learning losses are rarely a single convex bowl; the quadratic case is exactly solvable, which is why it can be checked against the closed-form least-squares answer.
Seven independent checks run live: analytic gradient versus finite differences of the direct sum, descent versus the normal equations, the chi-square law of the residual variance, coefficient recovery within standard errors, the two-step momentum recursion on exact eigenmodes, the predicted stability radius versus a simulation, and R2 versus squared correlation.