🔥 Correlation Matrix Heatmap

Understanding Correlation

Correlation measures the strength and direction of linear relationship between two variables. It's fundamental for understanding feature relationships, detecting multicollinearity, and feature engineering.

Correlation Coefficients

• Pearson Correlation:
  • Measures linear relationships
  • Range: -1 to +1
  • +1: Perfect positive linear
  • -1: Perfect negative linear
  • 0: No linear correlation
  • Assumes normal distribution
  • Sensitive to outliers
• Spearman Correlation:
  • Measures monotonic relationships
  • Rank-based (non-parametric)
  • Robust to outliers
  • Detects non-linear monotonic relationships
  • Use when data not normally distributed
• Kendall's Tau:
  • Also rank-based
  • More robust for small samples
  • Better for ordinal data

Interpreting Correlation Strength

• 0.0 - 0.3: Weak correlation
• 0.3 - 0.7: Moderate correlation
• 0.7 - 1.0: Strong correlation
• Negative values: Inverse relationship
• Important: Correlation ≠ Causation!

Multicollinearity

Multicollinearity occurs when features are highly correlated:

• Problems:
  • Unstable coefficient estimates
  • Difficult to determine individual feature effects
  • Inflated standard errors
  • Reduced model interpretability
• Detection:
  • Correlation matrix (|r| > 0.7-0.9 problematic)
  • VIF (Variance Inflation Factor) > 5 or 10
  • Condition number > 30
• Solutions:
  • Remove one of correlated features
  • Combine correlated features (PCA)
  • Regularization (Ridge, Lasso)
  • Domain knowledge to decide which to keep

Applications

• Feature Selection: Remove redundant features
• Feature Engineering: Create interaction terms
• Data Understanding: Discover relationships
• Model Debugging: Detect unexpected correlations
• Anomaly Detection: Unusual correlations signal issues

Correlation vs Causation

Critical to remember:

• High correlation doesn't imply causation
• Could be reverse causation
• Could be confounding variable
• Could be pure coincidence
• Need experiments/domain knowledge for causation

Beyond Linear Correlation

• Mutual Information: Captures non-linear dependencies
• Distance Correlation: Measures all dependencies
• Maximal Information Coefficient (MIC): Detects various relationships

Experiment with the Heatmap

Use the interactive tool above to:

• Explore correlations in different datasets
• Compare Pearson vs Spearman
• Identify strongly correlated features
• Detect multicollinearity
• Click cells for detailed correlation info