Eigenvalue Interpretation in PCA
In Principal Component Analysis (PCA), what do the eigenvalues of the covariance matrix represent?
Suppose you have a 100-dimensional dataset. The first three eigenvalues of the covariance matrix are $50$, $30$, and $10$, and the remaining 97 eigenvalues sum to $10$. How much variance is captured by the first 3 principal components?
Hints
- The trace of the covariance matrix equals the total variance. What is the trace in terms of eigenvalues?
- Each eigenvalue $\lambda_j$ is the variance of the data projected onto the $j$-th eigenvector. The fraction of variance explained by the first $k$ components is $\sum_{j=1}^{k} \lambda_j / \sum_{j=1}^{100} \lambda_j$.
- Sum all eigenvalues to get total variance: $50 + 30 + 10 + 10 = 100$. The first 3 capture $90/100 = 90\%$.
Worked Solution
How to Think About It: PCA finds the directions in which your data varies the most. The covariance matrix $\Sigma$ encodes all the second-order structure of the data. When you diagonalize $\Sigma = Q \Lambda Q^T$, each eigenvector $q_j$ is a direction (a principal component), and the corresponding eigenvalue $\lambda_j$ is the variance of the data when projected onto that direction. The eigenvalues are ordered from largest to smallest, so the first PC captures the most variance, the second captures the most of what's left, and so on.
Quick Estimate: Total variance = trace of the covariance matrix = sum of all eigenvalues = $50 + 30 + 10 + 10 = 100$. The first 3 components capture $50 + 30 + 10 = 90$ out of 100. That's $90\%$.
Approach: Use the spectral decomposition of the covariance matrix.
Formal Solution:
The covariance matrix of centered data $X$ (with $n$ observations in $\mathbb{R}^{100}$) is:
$$\Sigma = \frac{1}{n-1} X^T X$$
Since $\Sigma$ is symmetric positive semi-definite, it has an eigendecomposition:
$$\Sigma = Q \Lambda Q^T$$
where $Q$ is orthogonal (columns are the principal components $q_1, \ldots, q_{100}$) and $\Lambda = \text{diag}(\lambda_1, \ldots, \lambda_{100})$ with $\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_{100} \geq 0$.
The $j$-th eigenvalue equals the variance of the data projected onto the $j$-th principal component:
$$\lambda_j = \text{Var}(X q_j)$$
The total variance across all dimensions is:
$$\text{Total variance} = \text{tr}(\Sigma) = \sum_{j=1}^{100} \lambda_j = 50 + 30 + 10 + 10 = 100$$
Variance captured by the first $k$ principal components:
$$\text{Fraction explained} = \frac{\sum_{j=1}^{k} \lambda_j}{\sum_{j=1}^{100} \lambda_j}$$
For $k = 3$:
$$\text{Fraction explained} = \frac{50 + 30 + 10}{100} = \frac{90}{100} = 90\%$$
Answer: Each eigenvalue $\lambda_j$ represents the variance of the data along the $j$-th principal component. The first 3 principal components capture $90\%$ of the total variance.
Intuition
PCA is fundamentally about finding a low-dimensional subspace that preserves as much of the data's variability as possible. The eigenvalues tell you exactly how much variance each direction captures, and the rapid decay of eigenvalues (here: 50, 30, 10, then 97 dimensions sharing just 10) is typical of real-world data -- most of the action happens in a few directions, and the rest is noise.
In quant finance, PCA on a covariance matrix of asset returns typically reveals that the first 3-5 components explain 70-90% of the variance, corresponding to market-wide factors (market beta, sector rotation, term structure shifts). The remaining eigenvalues represent idiosyncratic noise. This decomposition is the foundation of factor models, risk attribution, and dimensionality reduction for portfolio construction. A practical rule of thumb: if you can explain 90%+ of variance with $k$ components, you can safely work in $k$ dimensions instead of 100.