OLS vs Ridge vs Lasso Regression
You are building a factor model for stock returns. You have $p = 200$ candidate factors and $n = 500$ daily observations. Let $X$ be the $n \times p$ design matrix and $Y$ the $n \times 1$ vector of returns.
- Derive the OLS estimator $\hat{\beta}_{\text{OLS}}$ in closed form. Under what conditions does it break down, and why is $p = 200$, $n = 500$ already a warning sign?
- Write down the Ridge regression objective (with penalty parameter $\lambda$) and derive its closed-form solution. Then show that Ridge is equivalent to the MAP estimator under Bayesian linear regression with a specific Gaussian prior on $\beta$. What is that prior?
- Why might Lasso ($L^1$ penalty) be preferred over Ridge for this factor model? What practical advantage does it give you that Ridge cannot?
Hints
- Think about what happens to the eigenvalues of $X^T X$ when factors are correlated and $p/n$ is not small.
- For the Bayesian connection, write the log-posterior and compare term-by-term with the Ridge objective. What prior distribution makes them match?
- The geometric difference between the $L^1$ and $L^2$ constraint regions (diamond vs circle) is why Lasso hits the axes and Ridge does not.
Worked Solution
How to Think About It: You have 200 factors and 500 observations -- that is a ratio of $n/p = 2.5$, which is uncomfortably close to the regime where OLS becomes unstable. In practice, many of those 200 factors will be correlated (value and earnings yield, momentum and reversal, etc.), so the effective degrees of freedom are even worse than the raw numbers suggest. The interviewer wants you to walk through the spectrum from "no regularization" (OLS) to "shrink everything" (Ridge) to "kill the irrelevant ones" (Lasso), and understand the Bayesian underpinnings.
Part (a): OLS Estimator
We minimize the squared residuals:
$$\hat{\beta}_{\text{OLS}} = \arg\min_{\beta} \|Y - X\beta\|^2$$
Taking the gradient and setting it to zero gives the normal equations $X^T X \beta = X^T Y$, so:
$$\hat{\beta}_{\text{OLS}} = (X^T X)^{-1} X^T Y$$
This requires $X^T X$ to be invertible. It fails when:
- $p > n$: The matrix $X^T X$ is at most rank $n$, so it is singular when $p > n$. No unique solution exists.
- Multicollinearity: Even when $p < n$, if factors are highly correlated, $X^T X$ has near-zero eigenvalues. The inverse blows up, and $\text{Var}(\hat{\beta}) = \sigma^2 (X^T X)^{-1}$ becomes enormous.
- Overfitting: With $n/p = 2.5$, OLS will fit the in-sample noise. The Marchenko-Pastur law tells us that the sample covariance eigenvalues are badly distorted when $p/n$ is not small. Expect poor out-of-sample performance.
Bottom line: OLS "works" here in the sense that $X^T X$ is technically invertible, but the estimates will be noisy and unreliable.
Part (b): Ridge Regression
The Ridge objective adds an $L^2$ penalty:
$$\hat{\beta}_{\text{Ridge}} = \arg\min_{\beta} \left\{ \|Y - X\beta\|^2 + \lambda \|\beta\|^2 \right\}$$
Taking the gradient:
$$-2X^T(Y - X\beta) + 2\lambda \beta = 0$$
Solving:
$$\hat{\beta}_{\text{Ridge}} = (X^T X + \lambda I)^{-1} X^T Y$$
The key: adding $\lambda I$ shifts every eigenvalue of $X^T X$ up by $\lambda$, so even zero eigenvalues become $\lambda > 0$. The matrix is always invertible.
Bayesian equivalence: Consider the Bayesian linear model:
- Likelihood: $Y \mid \beta \sim \mathcal{N}(X\beta, \, \sigma^2 I)$
- Prior: $\beta \sim \mathcal{N}\!\left(0, \, \frac{\sigma^2}{\lambda} I\right)$
The log-posterior is:
$$\log p(\beta \mid Y) = -\frac{1}{2\sigma^2} \|Y - X\beta\|^2 - \frac{\lambda}{2\sigma^2} \|\beta\|^2 + \text{const}$$
Maximizing this (the MAP estimate) is equivalent to minimizing $\|Y - X\beta\|^2 + \lambda\|\beta\|^2$, which is exactly the Ridge objective. So Ridge regression is MAP estimation under a zero-mean isotropic Gaussian prior on $\beta$ with variance $\sigma^2 / \lambda$ per component. Larger $\lambda$ means a tighter prior -- you believe the coefficients are small.
Part (c): Why Lasso over Ridge?
The Lasso objective uses an $L^1$ penalty:
$$\hat{\beta}_{\text{Lasso}} = \arg\min_{\beta} \left\{ \|Y - X\beta\|^2 + \lambda \|\beta\|_1 \right\}$$
The critical difference: Lasso produces sparse solutions -- it drives irrelevant coefficients to exactly zero. Ridge shrinks everything toward zero but never reaches it.
For a 200-factor model, you almost certainly believe that only a handful of factors (maybe 10-30) truly drive returns. Ridge keeps all 200 in the model with small but nonzero weights. Lasso automatically selects the relevant subset. This gives you:
- Interpretability: You can see which factors matter and which are noise.
- Out-of-sample robustness: Fewer parameters means less overfitting. Estimating 20 nonzero coefficients is much more stable than estimating 200 small ones.
- Execution simplicity: A sparse model is cheaper to trade -- you only need exposure to a few factors, not all 200.
The Bayesian analogy: Lasso corresponds to a Laplace (double-exponential) prior on $\beta$, which has a sharp peak at zero -- encoding the belief that most coefficients are exactly zero.
Answer: OLS is $(X^T X)^{-1} X^T Y$ but is unstable when $p/n$ is not small. Ridge adds $\lambda I$ to guarantee invertibility and is equivalent to MAP estimation with a $\mathcal{N}(0, \sigma^2/\lambda \cdot I)$ prior. Lasso replaces the $L^2$ penalty with $L^1$, producing sparse solutions that select the few factors that actually matter -- a critical advantage when most of your 200 candidates are noise.
Intuition
The core idea here is the bias-variance tradeoff made concrete. OLS is unbiased but can have enormous variance when $p/n$ is not small. Ridge introduces bias (shrinking coefficients toward zero) to dramatically reduce variance -- and the Bayesian interpretation makes this precise: you are encoding a prior belief that coefficients are small. The tradeoff is controlled by $\lambda$, which is just the strength of your prior.
The OLS-to-Ridge-to-Lasso progression is one of the most important concepts in quantitative finance. In practice, most factor models are sparse -- out of hundreds of candidate signals, only a few carry real predictive power. Ridge keeps all of them alive with tiny weights, which sounds harmless but means you are estimating (and trading on) 200 noisy coefficients instead of 20 meaningful ones. Lasso enforces the discipline of variable selection, which is why it (and its cousin Elastic Net) dominates in production factor models. The deeper lesson: regularization is not a statistical trick -- it is a statement of belief about the structure of the problem, and choosing the right penalty is choosing the right prior.