Covariance Matrix Eigenvalue Interpretation

Worked Solution

How to Think About It: The covariance matrix is a compact description of how $p$ variables co-move. The eigendecomposition cracks it open: it rotates your coordinate system into directions of pure, independent variance. Each eigenvalue tells you how much variance lives along that direction, and each eigenvector tells you which direction that is. If you have ever looked at a scatterplot of two highly correlated variables and noticed the cloud stretches diagonally -- that diagonal is the first eigenvector, and the length of the cloud along it is $\sqrt{\lambda_1}$. Once you see it that way, the rest follows naturally.

Key Insight: Eigenvalues are variances in a rotated coordinate system. Small eigenvalues do not mean those directions are unimportant -- they mean those directions have almost no variance, which in turn means the corresponding variables are nearly linearly dependent on the others.

The Method:

1. Eigendecomposition

Any symmetric positive semidefinite covariance matrix $\Sigma$ decomposes as:

$\Sigma = V \Lambda V^{\top}$

where $\Lambda = \text{diag}(\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_p \geq 0)$ and $V = [v_1, \ldots, v_p]$ is the matrix of orthonormal eigenvectors. The columns $v_k$ are the principal components (PCs) -- directions of decreasing variance.

2. Variance explained

If you project the data onto $v_k$, the projected scores have variance exactly $\lambda_k$. The total variance is $\text{tr}(\Sigma) = \sum_{i=1}^{p} \lambda_i$, so the fraction explained by PC $k$ is:

$\text{PVE}_k = \frac{\lambda_k}{\sum_{i=1}^{p} \lambda_i}$

This is why PCA starts with $v_1$: it maximizes the variance of any single linear projection of the data.

3. Condition number and multicollinearity

The condition number of $\Sigma$ is:

$\kappa(\Sigma) = \frac{\lambda_1}{\lambda_p}$

A small $\kappa$ (close to 1) means the matrix is well-conditioned -- variance is spread roughly equally in all directions, variables are not strongly collinear. A large $\kappa$ signals trouble:

• $\kappa \sim 10^3$ or more is commonly flagged as severe multicollinearity in regression diagnostics.
• Numerically, inverting $\Sigma$ amplifies errors in the smallest eigenvalue direction by a factor of $\kappa$, so coefficient estimates become highly sensitive to small perturbations in the data.

4. Near-zero eigenvalues and OLS breakdown

If $\lambda_p \approx 0$, it means there exists a direction $v_p$ along which the data has almost no variance. Equivalently, the linear combination $v_p^{\top} x$ is nearly constant across all observations -- the variables are nearly linearly dependent.

In OLS, the coefficient estimator is $\hat{\beta} = (X^{\top}X)^{-1} X^{\top} y$. The matrix $X^{\top}X$ is proportional to $\Sigma$, so a near-zero eigenvalue of $\Sigma$ makes $X^{\top}X$ nearly singular. Consequences:

• $(X^{\top}X)^{-1}$ is numerically unstable -- small changes in $y$ produce large swings in $\hat{\beta}$.
• Standard errors of coefficients blow up: $\text{Var}(\hat{\beta}) = \sigma^2 (X^{\top}X)^{-1}$, and the diagonal entries of $(X^{\top}X)^{-1}$ scale inversely with the eigenvalues.
• You cannot reliably distinguish the individual contributions of the collinear variables to the outcome.

5. Dimensionality reduction via PCA

Keep the top $k$ PCs where cumulative variance explained crosses a threshold (commonly 95%):

$k^{*} = \min\left\{k : \sum_{i=1}^{k} \lambda_i \geq 0.95 \sum_{i=1}^{p} \lambda_i\right\}$

Projecting onto these $k$ directions gives a rank-$k$ approximation $\hat{\Sigma} = V_k \Lambda_k V_k^{\top}$ that preserves most of the structure in the data while discarding the noisy, low-variance directions.

Practical Considerations:

• Always standardize your variables before PCA if they are on different scales. An eigenvalue of 1,000 on an unstandardized variable (e.g., revenue in millions) tells you nothing useful -- it just reflects the units.
• The 95% threshold is a heuristic. In finance, you often want to keep the number of factors interpretable (e.g., level, slope, curvature for a yield curve), even if that means explaining less variance.
• Condition number is useful but not the only diagnostic. Variance Inflation Factors (VIFs) per variable are often more interpretable for regression diagnostics.
• Ridge regression is one direct fix for near-singular $X^{\top}X$: it adds $\alpha I$ to the diagonal, artificially inflating the smallest eigenvalues by $\alpha$ and bounding $\kappa$.

Answer: The eigenvalues of $\Sigma = V \Lambda V^{\top}$ are the variances along orthogonal directions (principal components). $\lambda_k / \text{tr}(\Sigma)$ is the fraction of total variance explained by PC $k$. The condition number $\kappa = \lambda_1 / \lambda_p$ measures collinearity: large $\kappa$ means near-linear dependence among variables, which makes $(X^{\top}X)^{-1}$ unstable and OLS coefficient estimates unreliable. PCA truncates to the top $k$ eigenvalues to build a low-rank approximation that captures most of the variance.