Feature Scaling Bowls: Why One Step Size Needs Standardized Features (Condition Number)

Four loss bowls on a slate tray, one per scaler, each with a rolling steel marble. A thin trench means a high condition number: the marble zig-zags, crawls or flies off. Drag to orbit.

Switch to 2D
Controls

Presets

Parameters

Quality

Camera

Model checks

The science

Why scaling matters. For least squares the loss L(w) = 1/(2n) ∑(y - w·x)² is a quadratic bowl with Hessian H = D-1 C D-1, where C is the feature covariance and D the divisors of the chosen scaler. Gradient descent w ← w - η(Hw - b) has error ek = (I - ηH)k e0: it converges iff η < 2/λmax, at the rate max|1 - ηλi|, which is slowest when the condition number κ = λmax/λmin is large (at best (κ-1)/(κ+1) per step for the optimal step). Features in very different units make a long thin valley (large κ); one shared step size is then too big for the steep direction (raw: divergence) or tiny for the shallow one (min-max with outliers: crawling).

The four scalers. Raw (none), standardized (divide by the standard deviation), min-max (divide by the range) and robust (divide by the interquartile range). Outliers in feature 2 inflate the standard deviation and especially the range, but not the IQR, which is why min-max slows down and the robust scaler becomes the least stable at a large shared step. All four share η = (slider) / λmax of the standardized problem.

Simplifications. Two features, features and target centred so the intercept is solved exactly (the bias term is not part of the descent), synthetic Gaussian data with a few outliers, plain full-batch GD with a fixed step. Real stochastic or adaptive optimisers (Adam, line search) tolerate bad scaling better, though not for free. Checks: eigenvalues by closed form vs power iteration, scale equivariance of the exact solution from independently built designs, measured GD contraction rate vs max|1 - ηλ|, the 2/λmax stability edge by iteration, and the spectral closed form of the loss gap after 25 steps.

Loading model...
Loading Blender model...
-
-
Made with 🟠 Blender · Three.js