Ridge vs. Lasso Shrinkage in the SVD Basis

Worked Solution

How to Think About It: Ridge and Lasso are the two fundamental regularization tools for regression, and understanding how they shrink coefficients is essential for any quant working with predictive signals. The key insight comes from looking at these estimators in the SVD basis, where the geometry becomes transparent: Ridge smoothly shrinks all directions, while Lasso hard-thresholds weak directions to zero. This difference has enormous practical consequences when you are building trading signals from noisy, correlated features.

Key Insight: In the SVD basis, Ridge applies a smooth multiplicative shrinkage factor $d_j^2 / (d_j^2 + \lambda)$ to each principal direction, while Lasso applies soft-thresholding $\text{sign}(z_j)(|z_j| - \lambda/2)_+$ in the orthonormal case. Ridge keeps every direction alive (just dampened); Lasso kills weak directions entirely.

Derivation:

(i) Ridge closed form and SVD shrinkage.

Take the gradient of the ridge objective and set it to zero:

$$\nabla_{\beta} \left[ \|y - X\beta\|_2^2 + \lambda \|\beta\|_2^2 \right] = -2X^T(y - X\beta) + 2\lambda \beta = 0$$

Solving:

$$\hat{\beta}^{\text{ridge}} = (X^TX + \lambda I)^{-1} X^T y$$

Now write the SVD of $X = UDV^T$, where $U \in \mathbb{R}^{n \times r}$, $D = \text{diag}(d_1, \ldots, d_r)$, $V \in \mathbb{R}^{p \times r}$, and $r = \text{rank}(X)$. Then $X^TX = VD^2V^T$, and:

$$(X^TX + \lambda I)^{-1} = V(D^2 + \lambda I)^{-1}V^T$$

So:

$$\hat{\beta}^{\text{ridge}} = V(D^2 + \lambda I)^{-1}DU^Ty = \sum_{j=1}^{r} \frac{d_j}{d_j^2 + \lambda} (u_j^T y) \, v_j$$

Compare with OLS, which gives $\hat{\beta}^{\text{OLS}} = \sum_j \frac{1}{d_j}(u_j^T y) \, v_j$. Ridge multiplies the $j$-th OLS component by the shrinkage factor:

$$s_j = \frac{d_j^2}{d_j^2 + \lambda}$$

When $d_j^2 \gg \lambda$, $s_j \approx 1$ (strong directions are barely touched). When $d_j^2 \ll \lambda$, $s_j \approx 0$ (weak/noisy directions are crushed). The transition happens around $d_j^2 \approx \lambda$.

(ii) Lasso shrinkage contrast.

Lasso has no closed form in general, but in the orthonormal design case ($X^TX = I$), the OLS solution is $\hat{\beta}^{\text{OLS}}_j = X_j^Ty$, and the shrinkage operators are:

• Ridge: $\hat{\beta}^{\text{ridge}}_j = \frac{1}{1+\lambda} \hat{\beta}^{\text{OLS}}_j$ (proportional shrinkage)
• Lasso: $\hat{\beta}^{\text{lasso}}_j = \text{sign}(\hat{\beta}^{\text{OLS}}_j)(|\hat{\beta}^{\text{OLS}}_j| - \lambda/2)_+$ (soft-thresholding)

The critical difference: Ridge scales every coefficient toward zero but never to exactly zero. Lasso sets coefficients to exactly zero when $|\hat{\beta}^{\text{OLS}}_j| < \lambda/2$. This is why Lasso produces sparse models -- it performs automatic variable selection.

Geometrically, the $L_1$ ball has corners aligned with the coordinate axes, so the constrained optimum tends to land on a corner (some coordinates exactly zero). The $L_2$ ball is smooth, so the optimum is never exactly on an axis unless the unconstrained solution already is.

(iii) Cross-validation structure.

The correct procedure uses nested cross-validation (double CV):

1. Outer loop (evaluation): Split the data into $K$ outer folds. Each outer fold provides an honest out-of-sample prediction.
2. Inner loop (tuning): Within each outer training set, run a second $K$-fold CV over a grid of $\lambda$ values. Select the $\lambda$ that minimizes inner CV error.
3. Fit and predict: Using the selected $\lambda$, fit the model on the full outer training set and predict on the held-out outer test fold.
4. Aggregate: The out-of-sample $R^2$ is computed from the outer fold predictions.

Why naive CV fails: if you use a single CV loop to both pick $\lambda$ and estimate out-of-sample performance, you are evaluating performance at the $\lambda$ that looked best on the test folds. This introduces selection bias -- you picked the $\lambda$ that got lucky on the test data. The resulting $R^2$ is systematically inflated. Nested CV breaks this circularity by ensuring the data used to evaluate performance was never involved in any tuning decision.

For time-series returns, use walk-forward (expanding or rolling window) CV rather than random folds, to respect temporal ordering and avoid lookahead bias.

(iv) Ridge vs. Lasso in practice for quant signals.

In a typical quant setting with $p \gg n$ and many weak, correlated predictors, Ridge is usually the better default. Here is why:

• Correlated features: Lasso tends to pick one feature from a correlated group and zero out the rest, which is unstable -- small data perturbations change which feature survives. Ridge averages across correlated features, giving more stable coefficient estimates.
• Dense weak signals: If predictive power is spread across many features (each contributing a small amount), Lasso's hard thresholding kills too many useful signals. Ridge keeps them all, appropriately dampened.
• Stability: Ridge coefficients are smooth functions of the data; Lasso coefficients can jump discontinuously. For a trading strategy, this means Ridge produces more stable portfolio weights over time.

Lasso is preferred when you believe the true signal is genuinely sparse (only a handful of features matter) and interpretability is important.

Answer: The ridge estimator is $\hat{\beta}^{\text{ridge}} = (X^TX + \lambda I)^{-1}X^Ty$, which in the SVD basis applies shrinkage factors $d_j^2/(d_j^2 + \lambda)$ to each principal direction. Lasso applies soft-thresholding, producing exact zeros and thus sparse solutions. Honest out-of-sample evaluation requires nested CV (inner loop for $\lambda$ selection, outer loop for performance estimation). In the $p \gg n$ regime with correlated weak signals, Ridge is generally preferred for its stability and ability to aggregate diffuse signal across many features.