False Discovery in Alpha Mining
You are mining for trading alphas. You test $M$ independent strategies, each producing a t-statistic $Z_1, \ldots, Z_M \sim N(0,1)$ under the null hypothesis that the strategy has no alpha.
You declare a strategy "significant" if $Z_j > z$, where $z = \Phi^{-1}(1 - \alpha)$ is the $(1 - \alpha)$-quantile of the standard normal.
- Compute the expected number of false discoveries.
- Compute the probability of at least one false discovery.
- Choose $\alpha$ (in terms of $M$ and a target $\delta \in (0,1)$) so that the probability of any false discovery is at most $\delta$ (Bonferroni-style control).
Hints
- Under the null, each test independently rejects with probability $\alpha$. What distribution does the total number of rejections follow?
- The probability of at least one event happening equals one minus the probability of none happening. Use independence to factor the joint probability.
- For part (iii), either solve $(1 - \alpha)^M \geq 1 - \delta$ exactly, or apply the union bound $P(\bigcup A_j) \leq \sum P(A_j)$ for the Bonferroni form.
Worked Solution
How to Think About It: This is the bread-and-butter problem of quantitative research: you test a huge number of strategies, and even if none of them are real, some will look significant just by chance. If you test $M = 1000$ strategies at $\alpha = 0.05$, you expect about 50 false discoveries. That is not a bug in the math -- it is why most published alphas do not survive out of sample. The fix is to tighten your significance threshold so that the chance of even one false positive stays small. This is exactly what Bonferroni does.
Quick Estimate: Take $M = 1000$ and $\alpha = 0.05$. Each test has a 5% false positive rate, so the expected number of false discoveries is $1000 \times 0.05 = 50$. The probability that none of the 1000 tests falsely reject is $(1 - 0.05)^{1000} = 0.95^{1000}$, which is essentially 0. So the probability of at least one false discovery is nearly 1. To bring this down to, say, $\delta = 0.05$, Bonferroni says use $\alpha = 0.05 / 1000 = 0.00005$, corresponding to a z-threshold around $3.89$.
Approach: Each part follows directly from properties of independent Bernoulli trials.
Formal Solution:
Part (i): Expected number of false discoveries.
Under the null, each $Z_j \sim N(0,1)$, and a false discovery occurs when $Z_j > z$. The probability of this for any single test is:
$$P(Z_j > z) = 1 - \Phi(z) = \alpha$$
Let $D = \sum_{j=1}^{M} \mathbf{1}\{Z_j > z\}$ be the total number of discoveries. Since the tests are independent, $D \sim \text{Binomial}(M, \alpha)$. By linearity of expectation:
$$E[D] = M\alpha$$
Part (ii): Probability of at least one false discovery.
The probability that test $j$ does not falsely reject is $1 - \alpha$. Since the $M$ tests are independent, the probability that none of them falsely reject is:
$$P(D = 0) = (1 - \alpha)^M$$
So the probability of at least one false discovery is:
$$P(D \geq 1) = 1 - (1 - \alpha)^M$$
For small $\alpha$ and large $M$, this is well approximated by $1 - e^{-M\alpha}$. When $M\alpha \gg 1$, this is essentially 1 -- false discoveries are nearly certain.
Part (iii): Bonferroni correction.
We want $P(D \geq 1) \leq \delta$, i.e.:
$$1 - (1 - \alpha)^M \leq \delta$$
Solving:
$$(1 - \alpha)^M \geq 1 - \delta$$
$$\alpha \leq 1 - (1 - \delta)^{1/M}$$
This is the exact solution. The Bonferroni approximation uses the simpler bound from the union inequality:
$$P(D \geq 1) = P\!\left(\bigcup_{j=1}^{M} \{Z_j > z\}\right) \leq \sum_{j=1}^{M} P(Z_j > z) = M\alpha$$
Setting $M\alpha \leq \delta$ gives the Bonferroni threshold:
$$\alpha = \frac{\delta}{M}$$
This is slightly more conservative than the exact answer $1 - (1 - \delta)^{1/M}$, but the two coincide for small $\delta/M$ since $1 - (1-\delta)^{1/M} \approx \delta/M$ when $\delta/M \ll 1$.
Answer:
- $E[D] = M\alpha$.
- $P(D \geq 1) = 1 - (1 - \alpha)^M$.
- Choose $\alpha = \delta / M$ (Bonferroni correction). The exact threshold is $\alpha = 1 - (1 - \delta)^{1/M}$, which is slightly less conservative.
Intuition
This problem captures the fundamental curse of large-scale testing: when you run enough independent experiments, rare events become common. At a 5% significance level, one in twenty null strategies will look real. Run a thousand tests and you drown in false positives. This is not an abstract statistical concern -- it is arguably the central challenge in quantitative finance. Every quant shop runs thousands of backtests, and the ones that "work" are overwhelmingly noise unless you correct for multiplicity.
The Bonferroni correction is the simplest fix: divide your significance level by the number of tests. It is conservative (it controls the probability of even one false discovery, which is a strict standard), but it is robust and requires no assumptions beyond the ability to bound marginal p-values. In practice, quant firms often use less conservative methods like Benjamini-Hochberg (which controls the false discovery rate rather than the family-wise error rate), or they use out-of-sample validation as the ultimate multiplicity correction. But Bonferroni is the baseline that every researcher should understand, and it teaches the key lesson: significance thresholds must scale with the number of tests you run.