Ridge vs. Lasso Regression
You are comparing two common regularization approaches for linear regression: Ridge ($L_2$) and Lasso ($L_1$).
- Write down the optimization problem each method solves. What penalty term does each add to the ordinary least squares objective?
- Ridge regression has a closed-form solution. Derive it. Why does Lasso not have one?
- When would you prefer Ridge over Lasso, and vice versa? What does each method actually do to the coefficient vector $\hat{\beta}$?
Hints
- Think about the geometry: the $L_2$ penalty ball is smooth and round, while the $L_1$ ball has corners at the axes. What happens when the loss function contours touch each shape?
- For the Ridge closed form, treat the objective as a quadratic in $\beta$ and set its gradient to zero. The key identity is $\nabla_{\beta}\|Y - X\beta\|_2^2 = -2X^T(Y - X\beta)$.
- The choice between Ridge and Lasso comes down to your prior on the coefficient vector: do you believe most coefficients are small but nonzero (Ridge), or that most are exactly zero with a few large ones (Lasso)?
Worked Solution
How to Think About It: The core question is what each penalty actually does geometrically to the coefficients. Ridge shrinks all $\hat{\beta}_j$ toward zero proportionally -- like a spring pulling every coefficient back. Lasso shrinks too, but it also has a kink at zero, which means it can push coefficients all the way to exactly zero and drop features entirely. That difference -- continuous shrinkage vs. hard zeroing -- drives every practical trade-off. A quant interviewer asking this wants to see you understand the geometry, the closed form, and the real-world selection logic, not just recite the formulas.
Key Insight: The $L_2$ penalty is smooth (differentiable everywhere), which is why Ridge has a closed form. The $L_1$ penalty has a kink at $\beta_j = 0$, which breaks differentiability and forces iterative solvers -- but that same kink is exactly what causes exact sparsity.
The Method:
*Part 1: Optimization Problems*
Both methods minimize the usual residual sum of squares plus a penalty on $\beta$:
$$\hat{\beta}^{\text{Ridge}} = \arg\min_{\beta} \|Y - X\beta\|_2^2 + \lambda\|\beta\|_2^2$$
$$\hat{\beta}^{\text{Lasso}} = \arg\min_{\beta} \|Y - X\beta\|_2^2 + \lambda\|\beta\|_1$$
where $\|\beta\|_2^2 = \sum_j \beta_j^2$ and $\|\beta\|_1 = \sum_j |\beta_j|$. The tuning parameter $\lambda \geq 0$ controls how hard the penalty bites -- larger $\lambda$ means more shrinkage.
*Part 2: Closed Form for Ridge*
For Ridge, expand the objective and take the derivative with respect to $\beta$:
$$\frac{\partial}{\partial \beta}\left[\|Y - X\beta\|_2^2 + \lambda\|\beta\|_2^2\right] = -2X^T(Y - X\beta) + 2\lambda\beta = 0$$
Rearranging:
$$X^TX\hat{\beta} + \lambda\hat{\beta} = X^TY$$
$$(X^TX + \lambda I)\hat{\beta} = X^TY$$
$$\hat{\beta}^{\text{Ridge}} = (X^TX + \lambda I)^{-1}X^TY$$
The matrix $X^TX + \lambda I$ is always invertible for $\lambda > 0$ -- even if $X^TX$ is singular or near-singular (multicollinearity). This is a nice bonus: Ridge regularizes the normal equations by adding $\lambda$ to every diagonal entry.
Lasso has no closed form because $\|\beta\|_1$ is not differentiable at $\beta_j = 0$. The first-order condition involves the subgradient of the $L_1$ norm, and coordinate descent (or LARS) must be used.
*Part 3: When to Use Each*
- Use Lasso when you believe the true model is sparse -- only a handful of features actually matter. Lasso performs feature selection automatically by driving irrelevant coefficients exactly to zero. Especially useful when $p \gg n$ (many more features than observations).
- Use Ridge when you expect all (or most) features to contribute and multicollinearity is present. Ridge keeps all features in the model but shrinks them uniformly. The closed form is also computationally cheap -- important when $p$ and $n$ are both large.
- Elastic Net ($\lambda_1\|\beta\|_1 + \lambda_2\|\beta\|_2^2$) combines both: it does feature selection like Lasso but handles correlated predictors more gracefully (Lasso tends to pick one arbitrarily among a group of correlated features and zero out the others).
Practical Considerations: In financial applications, Ridge is often preferred in factor models where all factors plausibly contribute (e.g., PCA-based factor returns). Lasso shows up in sparse signal detection -- e.g., identifying which of 500 macro variables actually predict returns. One common mistake is forgetting to standardize features before applying either method: both penalties treat all $\beta_j$ symmetrically, so a feature measured in dollars will be penalized very differently from one measured in basis points unless you standardize first.
Answer: Ridge ($L_2$) shrinks all coefficients proportionally and has closed form $\hat{\beta} = (X^TX + \lambda I)^{-1}X^TY$; use it for multicollinear problems where all features contribute. Lasso ($L_1$) produces exact zeros and performs feature selection; use it when the model is sparse. Lasso lacks a closed form because the $L_1$ norm is non-differentiable at zero.
Intuition
The deeper lesson is that the shape of the penalty region determines the qualitative behavior of the estimator -- not just how much you shrink, but whether you ever hit zero. The $L_2$ ball is a sphere: the optimal point is always in its interior, so no coefficient ever lands exactly on an axis. The $L_1$ ball is a diamond with sharp corners sitting on the coordinate axes. When the elliptical loss contours are tangent to that diamond, they almost always touch a corner -- meaning one or more $\beta_j = 0$ exactly. This geometry is the entire reason Lasso does feature selection and Ridge does not, and it is far more illuminating than memorizing the formulas.
In practice, regularization choice is a modeling assumption about sparsity. In quant finance, factor models typically assume dense structure -- many small contributions from many sources -- favoring Ridge. Signal identification problems (e.g., scanning hundreds of macro predictors or alternative data signals) assume sparse structure, favoring Lasso or Elastic Net. Getting this wrong costs you: applying Ridge to a sparse problem keeps noise factors in the model; applying Lasso to a dense problem arbitrarily discards factors that genuinely contribute. Always standardize your features first -- both methods are scale-sensitive, and forgetting this is the most common implementation mistake.