Overfitting via Feature Search

Worked Solution

How to Think About It: This is the fundamental multiple-testing trap in quantitative research. You test thousands of features, pick the winner, and report its in-sample IC as if you had only tested that one feature. The problem is that even when every single feature is pure noise, the best-looking one will have a positive IC just by luck. The more features you search over, the bigger this spurious IC becomes. This is not a subtle statistical curiosity -- it is the number one reason quant alphas fail out of sample. Before doing any math, your gut should say: the expected max grows with $P$ and shrinks with $T$.

Quick Estimate: The in-sample ICs are $\hat{\rho}_p \sim N(0, 1/T)$, so after standardizing, $Z_p = \sqrt{T}\,\hat{\rho}_p \sim N(0,1)$. The expected maximum of $P$ i.i.d. standard normals is well-approximated by $E[\max Z_p] \approx \sqrt{2 \ln P}$ for large $P$. For $P = 1000$: $\sqrt{2 \ln 1000} = \sqrt{2 \times 6.91} \approx \sqrt{13.8} \approx 3.72$. Unstandardizing, the expected spurious IC is $E[\max \hat{\rho}_p] \approx \sqrt{2 \ln P / T}$. For $P = 1000$, $T = 2500$: $\sqrt{13.8 / 2500} = \sqrt{0.00552} \approx 0.074$. That is a 7.4% spurious IC, which is enormous -- many real alpha signals have true ICs in the 2-5% range. So searching 1000 features on 10 years of daily data can easily produce a "signal" that is pure noise.

Approach: We use the classical result on the expected maximum of i.i.d. Gaussian random variables, then refine with the Cramer correction.

Formal Solution:

Let $Z_1, \ldots, Z_P \overset{\text{i.i.d.}}{\sim} N(0,1)$. The CDF of $M_P = \max_p Z_p$ is $F_{M_P}(z) = \Phi(z)^P$.

The expected value satisfies:

$E[M_P] = \int_0^\infty [1 - \Phi(z)^P]\,dz - \int_{-\infty}^0 \Phi(z)^P\,dz$

For large $P$, the classical asymptotic result gives:

$E[M_P] = a_P + \frac{\gamma}{a_P} + o(1/a_P)$

where $a_P = \sqrt{2 \ln P}$ and $\gamma \approx 0.5772$ is the Euler-Mascheroni constant. A more refined expression uses the normalizing constants for the Gumbel limit:

$a_P = \sqrt{2 \ln P} - \frac{\ln(4\pi \ln P)}{2\sqrt{2 \ln P}}$

Since $\hat{\rho}_p = Z_p / \sqrt{T}$, we have:

$E\!\left[\max_p \hat{\rho}_p\right] = \frac{E[M_P]}{\sqrt{T}} \approx \sqrt{\frac{2 \ln P}{T}}$

This is the expected spurious IC from searching over $P$ features on $T$ days of data, even when no feature has any real predictive power.

Part 2: Leakage-Safe Protocol

The key problem is that feature selection and performance evaluation must use non-overlapping data. Here is a concrete protocol with three time-disjoint periods:

1. Split the data into three non-overlapping periods: - Selection set (e.g., first 40% of days): Used to screen all $P$ features and pick the top $K$ candidates. - Validation set (next 30%): Used to re-estimate IC on the $K$ shortlisted features and pick the final winner. This limits the multiple-testing penalty to $K \ll P$. - Test set (final 30%): Used exactly once to estimate the out-of-sample IC of the single chosen feature. This IC estimate is unbiased because the test data was never seen during selection or validation.

2. Procedure: - On the selection set, compute $\hat{\rho}_p^{\text{sel}}$ for each $p = 1, \ldots, P$. Rank and keep the top $K$ (e.g., $K = 10$). - On the validation set, compute $\hat{\rho}_p^{\text{val}}$ for each of the $K$ survivors. Select the single best feature $p^{*}$. - On the test set, compute $\hat{\rho}_{p^{*}}^{\text{test}}$. This is your unbiased IC estimate.

3. Adjustments: - Apply a Bonferroni or Benjamini-Hochberg correction at the selection stage to control false discovery. - The time splits must respect chronological order (no future data leaking into past periods). In practice, use walk-forward splits rather than random splits. - Optionally, add an embargo period (e.g., 5-20 days) between the selection and validation sets, and between the validation and test sets, to eliminate autocorrelation leakage.

Answer: The expected spurious IC from selecting the best of $P$ null features over $T$ days is:

$E\!\left[\max_p \hat{\rho}_p\right] \approx \sqrt{\frac{2 \ln P}{T}}$

For $P = 1000$ and $T = 2500$, this is roughly $0.074$, large enough to mimic a real signal. To get an honest IC estimate, use a three-way chronological split (select/validate/test) where the final test set is touched exactly once, with embargo gaps between periods to prevent autocorrelation leakage.