Effects of Multicollinearity on Regression
In standard linear regression, multicollinearity (high correlation among predictors) is present. Which of the following quantities are typically affected?
- Variance of the fitted regression coefficients
- $t$-statistics for individual coefficients
- Sum of squared residuals (SSR)
- $R^2$
For each, explain whether it is affected and why.
Hints
- Write down the formula for the variance of $\hat{\beta}$. What happens to $(X^TX)^{-1}$ when columns of $X$ are nearly collinear?
- Think about what happens if you duplicate a column of $X$. Does the model fit worse? Do the fitted values change?
- The fitted values depend only on the column space of $X$ (via the hat matrix), not on whether individual columns are correlated. SSR and $R^2$ are functions of fitted values only.
Worked Solution
How to Think About It: Multicollinearity means the columns of your design matrix $X$ are nearly linearly dependent, so $X^TX$ is close to singular. The key formula is the coefficient covariance: $\text{Var}(\hat{\beta}) = \sigma^2 (X^TX)^{-1}$. When a matrix is near-singular, its inverse has very large entries. That is where the damage shows up. But does multicollinearity affect how well the model fits? Think of it this way: if you duplicated a column of $X$, you'd have perfect multicollinearity, but the model can still fit the data equally well -- it just can't decide how to split the credit between the two copies.
Key Insight: Multicollinearity inflates the variance of individual coefficient estimates but does not degrade the model's overall predictive power (fitted values and $R^2$ are unaffected).
The Method:
1. Variance of fitted coefficients -- YES, affected.
$\text{Var}(\hat{\beta}) = \sigma^2(X^TX)^{-1}$. When $X^TX$ is nearly singular, $\det(X^TX) \approx 0$, so $(X^TX)^{-1}$ has very large entries. This means the estimated coefficients can swing wildly with small changes in the data. The variance inflation factor (VIF) quantifies this: VIF$_j = 1/(1 - R_j^2)$, where $R_j^2$ is the $R^2$ from regressing predictor $j$ on all other predictors.
2. $t$-statistics -- YES, affected.
The $t$-statistic for $\hat{\beta}_j$ is $t_j = \hat{\beta}_j / \text{SE}(\hat{\beta}_j)$. Since $\text{SE}(\hat{\beta}_j) = \sigma \sqrt{[(X^TX)^{-1}]_{jj}}$ is inflated by multicollinearity, the $t$-statistics become small even when the true coefficient is nonzero. This makes it hard to reject $H_0: \beta_j = 0$ for individual predictors, even though they may be jointly significant.
3. Sum of squared residuals -- NO, not affected.
SSR $= \|Y - X\hat{\beta}\|^2$. The fitted values $\hat{Y} = X(X^TX)^{-1}X^TY = H Y$ depend on the projection (hat) matrix $H$, which projects onto the column space of $X$. Multicollinearity does not change the column space of $X$ -- it only affects how the projection is decomposed across individual predictors. Think of it concretely: if you duplicate a column, the column space is unchanged, so the fitted values (and hence the residuals) are identical.
4. $R^2$ -- NO, not affected.
$R^2 = 1 - \text{SSR}/\text{SST}$, where SST $= \|Y - \bar{Y}\|^2$. Since SSR is unaffected by multicollinearity (as shown above) and SST depends only on $Y$, $R^2$ is also unaffected.
Practical Considerations:
- Multicollinearity is a problem for interpretation and inference (which variables matter?), not for prediction (how well does the model fit?).
- Common fixes: remove redundant variables, use PCA/ridge regression, or just accept that individual coefficients are not identifiable and focus on predictions.
- A classic trap in interviews: people think multicollinearity makes the model "worse." It doesn't -- it makes the coefficient estimates unstable, but the model's in-sample fit is fine.
Answer: Quantities (1) and (2) are affected. Quantities (3) and (4) are not. Multicollinearity inflates coefficient variances and shrinks $t$-statistics, but leaves fitted values, residuals, and $R^2$ unchanged.
Intuition
Multicollinearity is often misunderstood. It does not mean your regression is broken -- it means your regression cannot tell apart the contributions of highly correlated predictors. The model as a whole fits the data just as well, but if you ask "how important is variable $j$?", the answer is very noisy because the model could substitute correlated variable $k$ and get the same fit. This is directly analogous to portfolio theory: if two assets are perfectly correlated, you can't diversify between them, but a portfolio of the two has the same risk as either one alone. In trading, this shows up when you have multiple correlated signals in a regression-based alpha model -- the individual signal weights are unstable, but the combined PnL is not. Ridge regression (L2 penalty) is the standard fix: it stabilizes the coefficient estimates by shrinking them toward zero, at the cost of a small bias.