Nonparametric Walk-Forward Backtest Protocol

Worked Solution

How to Think About It: The core challenge is building a backtest that does not fool you. Every quant has seen strategies that look great in-sample and die in production. The two main enemies are look-ahead bias (using future information to make today's decision) and multiple testing (testing $N$ signals and cherry-picking the winner). A walk-forward protocol handles the first; a multiplicity correction handles the second. The nonparametric requirement means we cannot lean on normal distribution assumptions for returns -- we need procedures that work for heavy-tailed, autocorrelated, skewed data.

Key Insight: The walk-forward structure must strictly respect the arrow of time, and the multiple testing correction must account for the dependence structure across signals -- standard Bonferroni is too conservative because signal returns are correlated.

The Method:

*Part 1 -- Walk-Forward Scheme:*

1. Fix a training window of length $W$ days (e.g., $W = 252$ for one year) and a test window of length $T$ days (e.g., $T = 21$ for one month).
2. At each rebalance date $t$, use only data from $[t - W, t - 1]$ to rank stocks on signal $f_i$ and form the long-short portfolio.
3. Record the portfolio return over $[t, t + T - 1]$. This is one out-of-sample observation.
4. Slide the window forward by $T$ days and repeat, producing a time series of non-overlapping OOS returns $\{r_{i,1}, r_{i,2}, \ldots, r_{i,K}\}$ where $K = \lfloor (\text{total days} - W) / T \rfloor$.
5. The out-of-sample Sharpe ratio for signal $i$ is:

$$\widehat{SR}_i = \frac{\bar{r}_i}{s_i} \cdot \sqrt{252 / T}$$

where $\bar{r}_i$ and $s_i$ are the sample mean and standard deviation of the $K$ OOS return blocks.

1. Choosing $W$: too short and your ranking is noisy; too long and you miss regime changes. A practical default is 1-2 years. You can run the protocol for multiple $W$ values and check stability.
2. Choosing $T$: must be long enough that portfolio turnover costs are amortized. Monthly ($T = 21$) is standard for cross-sectional equity signals.

*Part 2 -- Multiple Testing Correction:*

1. For each signal $i$, compute a p-value for the null $H_0^{(i)}: SR_i \leq 0$. Under the nonparametric requirement, use a block bootstrap or a circular bootstrap on the OOS return blocks to estimate the null distribution of $\widehat{SR}_i$. Specifically: - Resample the $K$ OOS return blocks with replacement (use stationary block bootstrap with expected block length $\ell$ to preserve any residual autocorrelation). - Recompute $\widehat{SR}_i^{*}$ on each bootstrap sample. - The p-value is the fraction of bootstrap samples where $\widehat{SR}_i^{*} \geq \widehat{SR}_i$, centered under the null (shift the mean to zero before resampling). 2. Collect the $N$ p-values $\{p_1, \ldots, p_N\}$. 3. Apply the Benjamini-Hochberg (BH) procedure at level $q$ (e.g., $q = 0.05$): - Sort: $p_{(1)} \leq p_{(2)} \leq \cdots \leq p_{(N)}$. - Find the largest $k$ such that $p_{(k)} \leq k \cdot q / N$. - Reject $H_0$ for signals with $p_{(i)} \leq p_{(k)}$. 4. BH controls the expected FDR at level $q$ and is less conservative than Bonferroni, especially when signals are correlated.

*Part 3 -- Pitfalls and Mitigations:*

• Look-ahead in signal construction: If the signals themselves were designed using the full dataset (e.g., you picked which factors to test after seeing the data), the walk-forward only protects the portfolio formation step, not the signal selection step. Mitigation: treat the signal universe as given before the backtest begins, or use a separate holdout period for signal discovery vs. signal evaluation.
• Transaction costs: Decile sorts can produce high turnover. Always deduct realistic transaction costs (bid-ask spread, market impact) from returns before computing Sharpe.
• Survivorship and universe bias: Make sure the stock universe at each rebalance date only includes stocks that were tradeable at that time. No future knowledge of delistings.
• Non-stationarity: A signal might work in one regime and fail in another. Check whether the OOS Sharpe is stable across sub-periods, not just positive on average.
• Dependence across signals: If many signals are highly correlated, you are not really testing $N$ independent hypotheses. BH still controls FDR, but you lose statistical power. Consider clustering signals first and testing one representative per cluster.

Answer: Use a rolling walk-forward with non-overlapping OOS windows, compute per-signal Sharpe ratios from the OOS returns, obtain p-values via stationary block bootstrap under the null of zero Sharpe, and apply Benjamini-Hochberg to control the false discovery rate across all $N$ signals. Always deduct realistic trading costs, avoid look-ahead in signal construction, and verify stability across time.