Detecting and Addressing Multicollinearity

Worked Solution

How to Think About It: Multicollinearity means your regressors are nearly linearly dependent -- the design matrix $X$ is close to rank-deficient. Mechanically, $X^T X$ is nearly singular, so when you invert it to get $\hat{\beta} = (X^T X)^{-1} X^T y$, the result is unstable. Small changes in the data cause huge swings in individual coefficients. Think of it like trying to separate the contributions of two signals that always move together -- the combined effect is clear, but the split between them is arbitrary. This is the core issue: the model works fine for prediction, but the coefficient-level interpretation falls apart.

Key Insight: Multicollinearity inflates the variance of individual coefficient estimates without affecting the model's overall predictive power. It is an identification problem, not a fitting problem.

The Method:

*Detection tools, from simple to sophisticated:*

1. Pairwise correlation matrix. Compute $|r_{ij}|$ for all predictor pairs. Values above 0.8 are a red flag. Limitation: this only catches pairwise relationships. If $X_3 \approx X_1 + X_2$, each pair might have moderate correlation but the triplet is perfectly collinear.

1. Variance Inflation Factor (VIF). For each predictor $X_j$, regress it on all other predictors and compute:

$\text{VIF}_j = \frac{1}{1 - R_j^2}$

where $R_j^2$ is the $R^2$ from that auxiliary regression. A VIF of 1 means no collinearity. VIF above 5 is concerning; above 10 is serious. VIF catches multivariate collinearity that pairwise correlations miss.

1. Condition number of $X^T X$. Compute the ratio of the largest to smallest eigenvalue:

$\kappa = \frac{\lambda_{\max}}{\lambda_{\min}}$

Values above 30 indicate serious multicollinearity. This is the most comprehensive diagnostic -- it directly measures how close $X^T X$ is to being singular.

1. Eigenvalue decomposition. Look at the eigenvalues of $X^T X$ directly. Eigenvalues near zero point to specific directions of near-linear dependence. The associated eigenvectors tell you which variables are involved.

*Impact on coefficients:*

• Variance of $\hat{\beta}_j$ inflates. The variance is $\sigma^2 (X^T X)^{-1}_{jj}$. When $X^T X$ is nearly singular, diagonal elements of the inverse blow up. Equivalently, $\text{Var}(\hat{\beta}_j) = \sigma^2 \cdot \text{VIF}_j / \text{Var}(X_j)$.
• Coefficients become unstable. Small perturbations in the data cause large swings. You might see a coefficient flip sign when you add or remove a single observation.
• Individual t-tests lose power. Even if $\beta_j \neq 0$, the inflated standard error makes it hard to reject $H_0: \beta_j = 0$.
• Overall model fit is unaffected. $R^2$ and joint F-tests remain valid. The model predicts well -- you just cannot trust individual coefficient values.
• Predictions are fine as long as the collinearity structure persists in new data. If it breaks (e.g., a regime change decorrelates two factors), out-of-sample performance degrades.

*Remedies and trade-offs:*

1. Drop or combine variables. The simplest fix. If two signals are 0.95 correlated, you probably do not need both. Average them, or drop the one with less economic justification. Trade-off: you lose some information, and the choice can be subjective.

1. Ridge regression (L2 regularization). Replace OLS with:

$\hat{\beta}_{\text{ridge}} = (X^T X + \lambda I)^{-1} X^T y$

The penalty $\lambda I$ pushes the near-zero eigenvalues away from zero, stabilizing the inverse. This introduces bias but dramatically reduces variance. Trade-off: coefficients shrink toward zero, which can distort interpretation. You need to choose $\lambda$ (typically via cross-validation).

1. PCA regression. Project the predictors onto their principal components and regress on the top $k$ components. This eliminates collinearity by construction since PCs are orthogonal. Trade-off: the principal components are linear combinations of original variables, so you lose direct interpretability. The first few PCs may not align with economically meaningful factors.

1. Collect more data. More observations reduce the variance of $\hat{\beta}$, partly offsetting the inflation from collinearity. Trade-off: in practice you often cannot get more data of the same type, and the collinearity itself does not go away -- the variance reduction is proportional to
   /n$, but the VIF multiplier persists.

1. Domain-driven variable selection. Use economic reasoning to choose which variables belong in the model before looking at statistical diagnostics. If you are modeling equity returns, you probably do not need both P/E ratio and E/P. Trade-off: requires genuine domain expertise, and you might discard a variable that matters.

Answer: Detect multicollinearity using VIFs (above 5 is a warning), the condition number of $X^T X$ (above 30 is serious), and eigenvalue analysis. Multicollinearity inflates coefficient variance and makes individual estimates unreliable, but does not hurt overall model fit or predictions (as long as the collinearity structure holds). Fix it by dropping redundant variables, using ridge regression to stabilize the inverse, or projecting onto principal components. The right remedy depends on whether you need coefficient interpretation (use variable selection or ridge) or just predictive accuracy (PCA or ridge).