Detecting Multicollinearity in OLS
You are examining an OLS regression with design matrix $X$. Four methods have been proposed to detect multicollinearity among the predictor variables:
- Inspect the eigenvalues of $X^T X$ for near-zero values.
- Compute the condition number of $X^T X$ (ratio of largest to smallest eigenvalue) and flag large values.
- Inspect the correlation matrix of the rows of $X$.
- Inspect the correlation matrix of the columns of $X$.
Which of these methods correctly detect multicollinearity? For each, explain why it works or why it fails.
Hints
- Multicollinearity makes $X^T X$ nearly singular -- think about what mathematical properties of a matrix signal near-singularity.
- The determinant of a matrix equals the product of its eigenvalues, so near-zero eigenvalues of $X^T X$ directly indicate near-singularity. The condition number packages the same information.
- Pairwise correlation matrices only capture two-variable linear relationships. Three variables can be multicollinear ($X_3 \approx X_1 + X_2$) even when no single pair is highly correlated.
Worked Solution
How to Think About It: Multicollinearity means some columns of $X$ are nearly linearly dependent -- one predictor can be approximately written as a linear combination of others. The OLS estimator $(X^T X)^{-1} X^T y$ breaks down when $X^T X$ is close to singular. So the right question to ask about any detection method is: does it measure something close to singularity of $X^T X$? Methods that look at pairwise correlations are too crude -- multicollinearity can be a multi-way relationship invisible in pairwise statistics.
Key Insight: Multicollinearity is a property of the column space of $X$. Any reliable detection method needs to examine the joint linear structure of the columns, not just pairwise relationships.
The Method:
Option 1 -- Eigenvalues of $X^T X$: Correct. The matrix $X^T X$ is positive semi-definite. Its eigenvalues are all non-negative, and a zero eigenvalue means $X$ has a linearly dependent column (exact multicollinearity). Near-zero eigenvalues indicate near-multicollinearity. Since $\det(X^T X) = \prod_i \lambda_i$, a tiny determinant signals near-singularity. Inspecting small eigenvalues directly captures whether any linear combination of columns is nearly zero -- exactly what multicollinearity means.
Option 2 -- Condition number of $X^T X$: Correct. The condition number $\kappa = \lambda_{\max} / \lambda_{\min}$ measures how sensitive the matrix inverse is to perturbations. A large $\kappa$ means $\lambda_{\min}$ is near zero relative to the matrix scale -- the same signal as option 1, just normalized. In practice, $\kappa > 30$ is often flagged as moderate multicollinearity, and $\kappa > 100$ as severe.
Option 3 -- Correlation matrix of the rows of $X$: Incorrect. Rows of $X$ correspond to observations, not predictors. Correlations between observations tell you about your data sampling structure (are observations clustered?) but say nothing about whether any predictor can be written as a combination of others. This confuses two entirely different things.
Option 4 -- Correlation matrix of the columns of $X$: Partially useful, but insufficient. Columns of $X$ are the predictors, so their pairwise correlations are at least the right object. A high pairwise correlation between two columns is indeed one form of multicollinearity. However, pairwise correlation matrices miss multi-way multicollinearity: three predictors $X_1, X_2, X_3$ can have moderate pairwise correlations but still satisfy $X_3 \approx X_1 + X_2$, making the system nearly singular. The correlation matrix would not flag this. So option 4 is only a weak screen, not a reliable diagnostic.
Answer: Options 1 and 2 are correct. The eigenvalues (or condition number) of $X^T X$ are the right tools because they directly measure near-singularity of the Gram matrix, which is what multicollinearity causes. Options 3 and 4 are either wrong (rows) or insufficient (columns -- misses multi-way dependencies).
Intuition
The fundamental issue with multicollinearity is that $X^T X$ becomes ill-conditioned, making $(X^T X)^{-1}$ unstable. Small perturbations in the data lead to wildly different coefficient estimates. This is why eigenvalue-based diagnostics are correct: they directly measure the geometry of the column space. The condition number in particular has a clean interpretation -- it tells you how many decimal places of precision you lose when inverting the matrix.
The pairwise correlation trap is a classic beginner mistake. In practice, the Variance Inflation Factor (VIF) is the industry standard for detecting multicollinearity beyond pairwise. VIF for predictor $j$ is $1/(1 - R^2_j)$ where $R^2_j$ is from regressing $X_j$ on all other predictors -- it measures how well each predictor is explained by the rest, capturing multi-way dependencies that correlation matrices miss.