Regression with More Features Than Observations

Worked Solution

How to Think About It: The core issue is that when you have more knobs to turn ($d$ coefficients) than data points ($n$ equations), the system $X\beta = y$ is underdetermined. There are infinitely many $\beta$ vectors that fit the data perfectly -- and "perfectly" is the problem. You are fitting noise, not signal. In quant finance, this comes up constantly: you might screen 500 potential alpha signals across 200 trading days. Without regularization you will get a backtest that looks incredible and a live strategy that bleeds money.

Key Insight: The matrix $X^\top X$ is $d \times d$ but has rank at most $\min(n, d) = n < d$, so it is singular. No inverse exists, and OLS has no unique solution. The fix, in every case, is to constrain the solution space -- either by shrinking coefficients, forcing sparsity, or reducing the dimension of $X$ before fitting.

The Method:

*Why OLS breaks:*

1. Singular normal equations. $X^\top X$ has rank at most $n$, so $(X^\top X)^{-1}$ does not exist. The loss surface has a $(d - n)$-dimensional flat valley of global minima, all with $R^2 = 1$ on the training set.

1. Perfect in-sample fit, zero out-of-sample value. With $d > n$, you can interpolate the training data exactly. The model memorizes noise. Out-of-sample $R^2$ will typically be negative -- worse than just predicting the mean.

*How each method fixes it:*

1. Ridge regression (L2). Solve $\hat{\beta}_{\text{ridge}} = (X^\top X + \lambda I)^{-1} X^\top y.$ Adding $\lambda I$ makes the matrix full-rank and invertible for any $\lambda > 0$. All coefficients shrink toward zero, but none are set exactly to zero. Ridge is optimal when many features contribute small amounts -- common in dense factor models.

2. Lasso (L1). Solve $\min_{\beta} \|y - X\beta\|_2^2 + \lambda \|\beta\|_1.$ The L1 penalty produces sparse solutions: many $\beta_j$ are driven to exactly zero. This performs automatic feature selection, which is useful when you believe only a handful of signals matter and the rest are noise.

1. PCA regression. Compute the top $k$ principal components of $X$ (where $k < n$), project the data onto this lower-dimensional subspace, and run OLS in the reduced space. This sidesteps the rank problem entirely and handles multicollinearity well. The drawback: the principal components are linear combinations of all original features, so interpretability suffers.

4. Elastic Net. Combine L1 and L2: $\min_{\beta} \|y - X\beta\|_2^2 + \lambda_1 \|\beta\|_1 + \lambda_2 \|\beta\|_2^2.$ This gets you the sparsity of Lasso plus the stability of Ridge. It handles correlated features better than pure Lasso, which tends to arbitrarily pick one feature from a correlated group and drop the rest.

Practical Considerations:

• Choosing $\lambda$: Use cross-validation, but be careful with time series data -- standard K-fold leaks future information. Walk-forward CV is the right choice in finance.
• Standardization matters. Ridge and Lasso penalties are scale-dependent. Always standardize features before fitting, or you penalize large-scale features disproportionately.
• Lasso instability. When features are highly correlated, Lasso's variable selection is unstable -- small changes in the data can flip which features survive. Elastic Net or stability selection helps.
• Bias-variance trade-off. All regularization methods introduce bias to reduce variance. The right amount of bias depends on how noisy your data is, and in finance, it is almost always very noisy.

Answer: OLS fails when $d > n$ because $X^\top X$ is singular (rank $\leq n < d$), yielding infinitely many solutions that overfit perfectly. Ridge regression restores invertibility by adding $\lambda I$; Lasso enforces sparsity via L1 penalty for automatic feature selection; PCA regression reduces dimensionality below $n$; and Elastic Net combines L1 and L2 to get sparsity with stability. In practice, choose based on whether you believe the true signal is dense (Ridge), sparse (Lasso), or sparse with correlated groups (Elastic Net).