Ridge vs. Lasso Regression: Theory and Practice
Consider a linear model $y = X\beta + \varepsilon$ with design matrix $X \in \mathbb{R}^{n \times d}$ and response $y \in \mathbb{R}^n$.
- Write the optimization problems for ridge regression and lasso, including how the intercept is typically handled.
- Derive the closed-form solution for ridge regression when $X^\top X$ is invertible.
- Explain mathematically (not just heuristically) why lasso can produce exact zeros in the coefficient vector while ridge typically does not.
- Describe a cross-validation scheme to select the regularization hyperparameter $\lambda$, and recommend a metric you would optimize when building a trading signal.
Hints
- The intercept is typically excluded from the penalty by centering the data first. Write the penalized least-squares objectives with $L_2$ and $L_1$ norms on the coefficient vector.
- For the ridge closed form, differentiate the objective and solve for $\beta$. Notice that $X^\top X + \lambda I$ is always invertible for $\lambda > 0$.
- For lasso sparsity, look at the subgradient optimality condition at $\beta_j = 0$. The subdifferential of $|\beta_j|$ at zero is the interval $[-1, 1]$, which gives a non-trivial region where zero is optimal.
Worked Solution
How to Think About It: Ridge and lasso are the two canonical regularized regression methods, and understanding their differences is essential for any quant who builds predictive models. The core tension is bias vs. variance: both methods shrink coefficients toward zero to reduce overfitting, but they do it in geometrically different ways. Ridge uses an $L_2$ penalty (a sphere), which shrinks everything proportionally. Lasso uses an $L_1$ penalty (a diamond), whose corners sit on the axes -- and that geometry is exactly why lasso can zero out coefficients while ridge cannot.
Key Insight: The reason lasso produces exact zeros is not heuristic -- it comes from the subdifferential of the absolute value function. At $\beta_j = 0$, the $L_1$ penalty has a kink, and the subgradient condition can be satisfied at zero. The smooth $L_2$ penalty has no such kink, so the gradient condition at zero is generically not satisfied.
The Method:
Part (i): Optimization problems
Assume columns of $X$ are centered (mean-subtracted) and the intercept $\beta_0$ is fit separately as $\bar{y}$. Then regularize only the slope coefficients $\beta \in \mathbb{R}^d$.
Ridge: $$\hat{\beta}_{\text{ridge}} = \arg\min_{\beta} \left\{ \frac{1}{2n} \|y - X\beta\|_2^2 + \lambda \|\beta\|_2^2 \right\}$$
Lasso: $$\hat{\beta}_{\text{lasso}} = \arg\min_{\beta} \left\{ \frac{1}{2n} \|y - X\beta\|_2^2 + \lambda \|\beta\|_1 \right\}$$
The intercept is handled by centering: set $\hat{\beta}_0 = \bar{y} - \bar{X}^\top \hat{\beta}$, where $\bar{X}$ is the vector of column means. This way the intercept is never penalized.
Part (ii): Ridge closed-form
The ridge objective is differentiable. Setting the gradient to zero: $$-\frac{1}{n} X^\top(y - X\beta) + 2\lambda \beta = 0$$ $$X^\top X \beta + 2n\lambda \beta = X^\top y$$ $$(X^\top X + 2n\lambda I)\beta = X^\top y$$
With the common convention $\lambda' = 2n\lambda$: $$\hat{\beta}_{\text{ridge}} = (X^\top X + \lambda' I)^{-1} X^\top y$$
Note: $X^\top X + \lambda' I$ is always invertible for $\lambda' > 0$, even when $X^\top X$ is singular. This is one of ridge's key practical benefits.
Part (iii): Why lasso zeros out coefficients
Consider the lasso subproblem for a single coefficient $\beta_j$, holding all others fixed. The optimality condition uses the subdifferential of $|\beta_j|$:
$$\frac{\partial}{\partial \beta_j} \left[ \frac{1}{2n}\|y - X\beta\|_2^2 \right] + \lambda \cdot \partial |\beta_j| \ni 0$$
Let $r_j = y - X_{-j}\beta_{-j}$ be the partial residual. The condition becomes:
- If $\beta_j > 0$: $-\frac{1}{n} X_j^\top r_j + \frac{1}{n} \|X_j\|^2 \beta_j + \lambda = 0$
- If $\beta_j < 0$: $-\frac{1}{n} X_j^\top r_j + \frac{1}{n} \|X_j\|^2 \beta_j - \lambda = 0$
- If $\beta_j = 0$: the subgradient condition requires $\left|\frac{1}{n} X_j^\top r_j\right| \leq \lambda$
So $\hat{\beta}_j = 0$ whenever the correlation between feature $j$ and the residual is small enough (below $\lambda$ in magnitude). This is the soft-thresholding operator: $$\hat{\beta}_j = \text{sign}(z_j) \max(|z_j| - \lambda, 0) / (\|X_j\|^2/n)$$ where $z_j = X_j^\top r_j / n$.
For ridge, the optimality condition at $\beta_j = 0$ is $-X_j^\top r_j / n + 2\lambda \cdot 0 = 0$, which requires $X_j^\top r_j = 0$ exactly. This happens with probability zero for continuous data. So ridge generically never produces exact zeros.
Part (iv): Cross-validation for trading signals
Use time-series cross-validation (walk-forward or expanding window), never standard K-fold, because financial data is serially correlated.
- Define a grid of $\lambda$ values (log-spaced, e.g., $10^{-4}$ to $10^{2}$).
- For each $\lambda$, run walk-forward CV: train on data up to time $t$, predict on period $[t+1, t+h]$, slide forward, repeat.
- Optionally add a purge/embargo gap between train and test to prevent leakage from overlapping labels.
Metric choice: For a trading signal, do not optimize $R^2$ or MSE -- these reward accurate forecasts of magnitude, which is not the same as profitability. Instead, optimize: - Information coefficient (IC): Spearman rank correlation between predicted and realized returns. This measures whether the signal ranks assets correctly. - Alternatively, risk-adjusted PnL (Sharpe ratio of a simple long/short portfolio formed from the signal), though this is noisier.
The key is that a trading signal needs to get the direction and ranking right, not the exact magnitude.
Answer: Ridge has a closed-form $(X^\top X + \lambda I)^{-1} X^\top y$ and shrinks coefficients toward zero but never exactly to zero. Lasso produces exact zeros via soft-thresholding, driven by the non-differentiability of $|\beta_j|$ at zero. For trading signals, use walk-forward CV with an information coefficient metric.
Intuition
The geometric picture makes the ridge-vs-lasso difference click. Ridge's $L_2$ ball is smooth -- the OLS solution gets pulled toward the origin along a ray, but generically never lands on an axis. Lasso's $L_1$ diamond has corners on the axes, and the contours of the least-squares objective will generically first touch the diamond at a corner, which corresponds to some coefficients being exactly zero. This is why lasso does automatic variable selection while ridge does not.
For quants, this distinction matters because financial features are often noisy and redundant. Lasso's ability to zero out irrelevant features is valuable for building parsimonious trading signals. But lasso has a downside: when features are highly correlated (common in finance -- think sector factors), lasso arbitrarily picks one and zeros out the rest. Elastic net (a mix of $L_1$ and $L_2$) handles this better. In practice, you almost always want to combine regularization with a sensible CV scheme that respects temporal ordering, because the biggest risk in quant modeling is not bias or variance in isolation -- it is overfitting to historical patterns that do not persist.