📉 Loss Function Hyperparameters

About this simulation

The surface below is the loss landscape for a tiny linear model ŷ = w1·x + w2 fit to a handful of data points. Height at each (w1, w2) is the loss the model would incur with those weights. A marble performs batch gradient descent on that surface, stepping downhill each tick — exactly what an optimizer does during training, just visualized in 3D instead of as a loss-vs-epoch chart.

How it works

• MSE squares every residual, so one huge outlier dominates the gradient and drags the whole surface (and the marble) toward it.
• MAE uses absolute residuals — the landscape stays a shallow cone, far less sensitive to the outlier, but the gradient near the minimum never shrinks, so descent can jitter.
• Huber blends the two: quadratic (like MSE) for residuals smaller than δ, linear (like MAE) beyond it — smooth near the minimum, robust far from it.
• L2 regularization adds λ·(w1²+w2²) to the loss, pulling the whole bowl's minimum back toward the origin and shrinking the weights.

Controls

• Loss function — switch between MSE, MAE and Huber to reshape the surface live.
• Huber delta (δ) — the residual size where Huber switches from quadratic to linear behaviour (only affects Huber).
• Learning rate (η) — step size for the descending marble; too high and it overshoots or oscillates.
• L2 regularization (λ) — penalty strength on large weights; raises and re-centers the bowl.
• Add outlier point — injects one point with a very large residual into the training data.
• ↺ Restart descent — resets the marble to its start position and clears its trail.

Did you know?

Huber loss was introduced by Peter J. Huber in 1964 specifically to get MSE's smooth, well-behaved gradients near the optimum while keeping MAE's robustness against outliers — a trade-off tuned entirely through the δ hyperparameter.