Multicollinearity & VIF: Why Correlated Predictors Break Regression

What Multicollinearity Is

Multicollinearity occurs when two or more predictor variables in a multiple regression model are highly correlated, meaning that one can be linearly predicted from the others with a substantial degree of accuracy. This phenomenon introduces instability and inefficiency into the regression coefficients, making it difficult to interpret their individual effects.

The presence of multicollinearity can lead to inflated standard errors for the regression coefficients, which in turn reduces the statistical power of the model. Consequently, even if a predictor is statistically significant, its coefficient may not accurately reflect the true relationship between that variable and the response.

The Variance Inflation Factor (VIF)

The Variance Inflation Factor (VIF) quantifies how much the variance of an estimated regression coefficient is increased because of multicollinearity. A VIF value greater than 1 indicates that the predictor is correlated with other predictors, and values above 5 or 10 suggest a problematic level of collinearity.

By calculating VIF for each predictor in a model, analysts can identify which variables are causing multicollinearity issues and take appropriate steps to address them, such as removing one of the correlated predictors or using dimensionality reduction techniques.

Impact on Regression Models

When two predictor variables are highly correlated, the least squares estimates of their regression coefficients become unstable. This instability can lead to large fluctuations in the coefficient estimates as small changes in the data occur, making it difficult to trust the model's predictions.

In extreme cases, multicollinearity can cause the regression coefficients to be either positive or negative regardless of the true relationship between the variables and the response, leading to misleading conclusions about the importance of each predictor.

Regularization Techniques

To mitigate the effects of multicollinearity, regularization techniques such as Ridge regression can be employed. Ridge regression adds a penalty term to the sum of squared residuals, which shrinks the coefficients towards zero but does not set them exactly to zero. This shrinkage helps stabilize the estimates and reduces their variance.

By adjusting the strength of the penalty through the parameter λ (lambda), analysts can control the trade-off between bias and variance in the model, leading to more reliable and interpretable results.

Frequently asked questions

Try it live

Everything above runs in your browser — open Multicollinearity & VIF: Why Correlated Predictors Break Regression and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.