Ridge vs. Lasso Regression

Worked Solution

How to Think About It: The core question is what each penalty actually does geometrically to the coefficients. Ridge shrinks all $\hat{\beta}_j$ smoothly toward zero -- like a spring pulling every coefficient back, hardest along the directions the data pin down least. Lasso shrinks too, but it also has a kink at zero, which means it can push coefficients all the way to exactly zero and drop features entirely. That difference -- continuous shrinkage vs. hard zeroing -- drives every practical trade-off. A quant interviewer asking this wants to see you understand the geometry, the closed form, and the real-world selection logic, not just recite the formulas.

Key Insight: The $L_2$ penalty is smooth (differentiable everywhere), which is why Ridge has a closed form. The $L_1$ penalty has a kink at $\beta_j = 0$, which breaks differentiability and forces iterative solvers -- but that same kink is exactly what causes exact sparsity.

The Method:

*Part 1: Optimization Problems*

Both methods minimize the usual residual sum of squares plus a penalty on $\beta$:

$$\hat{\beta}^{\text{Ridge}} = \arg\min_{\beta} \|Y - X\beta\|_2^2 + \lambda\|\beta\|_2^2$$

$$\hat{\beta}^{\text{Lasso}} = \arg\min_{\beta} \|Y - X\beta\|_2^2 + \lambda\|\beta\|_1$$

where $\|\beta\|_2^2 = \sum_j \beta_j^2$ and $\|\beta\|_1 = \sum_j |\beta_j|$. The tuning parameter $\lambda \geq 0$ controls how hard the penalty bites -- larger $\lambda$ means more shrinkage.

*Part 2: Closed Form for Ridge*

For Ridge, expand the objective and take the derivative with respect to $\beta$:

$$\frac{\partial}{\partial \beta}\left[\|Y - X\beta\|_2^2 + \lambda\|\beta\|_2^2\right] = -2X^T(Y - X\beta) + 2\lambda\beta = 0$$

Rearranging:

$$X^TX\hat{\beta} + \lambda\hat{\beta} = X^TY$$

$$(X^TX + \lambda I)\hat{\beta} = X^TY$$

$$\hat{\beta}^{\text{Ridge}} = (X^TX + \lambda I)^{-1}X^TY$$

Why the inverse always exists. $X^TX$ is symmetric positive semidefinite: for any $v$, $v^TX^TXv = \|Xv\|_2^2 \geq 0$, so its eigenvalues satisfy $\mu_1 \geq \cdots \geq \mu_p \geq 0$. Write $X^TX = Q\,\text{diag}(\mu_i)\,Q^T$ with $Q$ orthogonal; since $I = QQ^T$, $$X^TX + \lambda I = Q\,\text{diag}(\mu_i + \lambda)\,Q^T,$$ and every eigenvalue is $\mu_i + \lambda \geq \lambda > 0$. So $X^TX + \lambda I$ is positive definite and invertible for every $\lambda > 0$, with no assumption on the rank of $X$: collinear columns or more features than observations ($p > n$, where $X^TX$ is singular and OLS has no unique solution) are fine. The Hessian of the objective is $2(X^TX + \lambda I) \succ 0$, so the objective is strictly convex and this stationary point is the unique minimizer.

What $\lambda$ buys numerically and statistically. The condition number drops from $\mu_1/\mu_p$ (infinite when $\mu_p = 0$) to $(\mu_1 + \lambda)/(\mu_p + \lambda) \leq (\mu_1 + \lambda)/\lambda$, finite whatever the data. When $X^TX$ is invertible, each component of $\hat{\beta}$ along an eigenvector is the OLS component multiplied by $\mu_i/(\mu_i + \lambda)$: high-variance directions pass almost untouched, while the fragile low-variance directions that OLS would fit to noise are shrunk hard. That is the bias-for-variance trade. (A random $5 \times 8$ design with $\lambda = 0.3$ makes this concrete: $X^TX$ has rank $5$, yet $X^TX + 0.3I$ has smallest eigenvalue exactly $0.3$ and the closed form matches a numerical minimizer.)

Lasso has no closed form because $\|\beta\|_1$ is not differentiable at $\beta_j = 0$. The first-order condition involves the subgradient of the $L_1$ norm, and coordinate descent (or LARS) must be used. The one exception is an orthonormal design ($X^TX = I$): there the problem separates by coordinate and Lasso is soft-thresholding of the OLS coefficients, $\hat{\beta}_j = \text{sign}(z_j)\max(|z_j| - \lambda/2,\, 0)$ with $z = X^TY$, which shows the exact zeros directly.

*Part 3: When to Use Each*

• Use Lasso when you believe the true model is sparse -- only a handful of features actually matter. Lasso performs feature selection automatically by driving irrelevant coefficients exactly to zero. Especially useful when $p \gg n$ (many more features than observations).

• Use Ridge when you expect all (or most) features to contribute and multicollinearity is present. Ridge keeps all features in the model and shrinks them smoothly, most strongly along low-variance directions. The closed form is also computationally cheap -- important when $p$ and $n$ are both large.

• Elastic Net ($\lambda_1\|\beta\|_1 + \lambda_2\|\beta\|_2^2$) combines both: it does feature selection like Lasso but handles correlated predictors more gracefully (Lasso tends to pick one arbitrarily among a group of correlated features and zero out the others).

Practical Considerations: In financial applications, Ridge is often preferred in factor models where all factors plausibly contribute (e.g., PCA-based factor returns). Lasso shows up in sparse signal detection -- e.g., identifying which of 500 macro variables actually predict returns. One common mistake is forgetting to standardize features before applying either method: both penalties treat all $\beta_j$ symmetrically, so a feature measured in dollars will be penalized very differently from one measured in basis points unless you standardize first.

Answer: Ridge ($L_2$) shrinks all coefficients toward zero without zeroing them and has closed form $\hat{\beta} = (X^TX + \lambda I)^{-1}X^TY$, always well defined because $\lambda I$ lifts every eigenvalue of $X^TX$ to at least $\lambda > 0$; use it for multicollinear problems where all features contribute. Lasso ($L_1$) produces exact zeros and performs feature selection; use it when the model is sparse. Lasso lacks a closed form because the $L_1$ norm is non-differentiable at zero.