Multicollinearity & VIF: Why Correlated Predictors Break Regression

Predictor Correlation

corr(x₁, x₂) 0.00

Noise σ (in y) 0.60

Ridge penalty λ 0.0

Live Diagnostics

VIF

1.00

sample r(x₁,x₂)

0.00

SE(β₁) OLS

0.00

R²

0.00

OLS fit β̂₁, β̂₂ (true: 2.00, −1.50)

—

Ridge fit β̂₁, β̂₂ (λ shrinkage)

—

OLS plane — tilts wildly as VIF grows

Ridge plane — biased but stable

sample points (x₁, x₂, y)

How it works

Two predictors x₁, x₂ are drawn with a controllable correlation r, then y is generated from a fixed true model plus noise:

x2 = r·x1 + √(1−r²)·z         (so corr(x1,x2) ≈ r)
y  = 2.0·x1 − 1.5·x2 + σ·ε

OLS:   β̂ = (XᵀX)⁻¹Xᵀy
Ridge: β̂ = (XᵀX + λI)⁻¹Xᵀy

VIF_j = 1 / (1 − R²_j)   (R²_j from regressing xⱼ on the other predictors)
Var(β̂1) = σ² · VIF1 / Σ(x1 − x̄1)²

With only two standardized predictors this collapses to VIF = 1/(1−r²): as the correlation slider approaches ±1, VIF explodes, Var(β̂) blows up, and the fitted OLS plane (green) starts flipping to wildly different tilts on every resample even though it still fits the cloud almost as well — the coefficients are no longer identifiable, only their sum along the collinear direction is. The Ridge plane (orange) adds λI to the normal equations, which is exactly the article's point: regularization trades a little bias for a lot less variance, keeping the fit stable when predictors are entangled instead of independent.

corr(x₁,x₂) slider — sets how collinear the two predictors are.
Noise σ — observation noise added to y; higher noise makes the instability easier to see.
Ridge λ — L2 penalty strength; raise it while r is near ±1 to watch the orange plane stop swinging.
Resample — draws a fresh n=220 sample at the same r, σ, λ so you can see the OLS plane jump between resamples once VIF is high, while the Ridge plane barely moves.

⚠ Couldn't load the 3D engineThree.js failed to load from the CDN. Check your connection and reload.