Comparing Forecasting Models for Daily Asset Returns
You are comparing two forecasting models for daily asset returns: Model A, a simple linear factor model, and Model B, a regularized machine-learning model. Both output next-day predicted returns $\hat{r}^{(A)}_t$ and $\hat{r}^{(B)}_t$ for a universe of assets. Each day, you form a zero-cost long-short portfolio by going long the top-predicted assets and short the bottom-predicted assets, scaled so that each portfolio has unit ex-ante volatility.
(a) Describe how you would construct a rolling cross-validation scheme on this time-series data to fairly compare the out-of-sample Sharpe ratios of the two models. Be specific about the fold structure, what is trained on what, and how you prevent look-ahead bias.
(b) Propose a statistical test to assess whether Model B's Sharpe ratio is significantly higher than Model A's. State the null and alternative hypotheses clearly, define the test statistic, and discuss its distributional assumptions.
(c) You are evaluating not just two models but an entire library of candidate models. Discuss how transaction costs, turnover constraints, and multiple comparisons should factor into the evaluation.
Hints
- Time series CV must respect the arrow of time -- training folds must always precede test folds. Think about what an embargo period buys you.
- To test whether Model B has a higher Sharpe, frame it as testing whether the mean of the daily return difference $d_t = r^{(B)}_t - r^{(A)}_t$ is positive. Use a HAC (Newey-West) standard error to handle autocorrelation.
- For multiple comparisons across a model library, apply Benjamini-Hochberg to control the false discovery rate, and always compute net-of-cost Sharpe -- gross Sharpe rankings can invert after accounting for turnover.
Worked Solution
How to Think About It: This is a model evaluation design problem, not a calculation. The central difficulty is that financial time series are not i.i.d. -- there is autocorrelation in returns, volatility clustering, and regime shifts. Standard K-fold CV is broken here because it randomly assigns observations to folds, which means you can train on tomorrow and test on yesterday. The other trap is multiple comparisons: if you test 100 models and pick the best, you are almost guaranteed to find a spurious winner even if none of them work. A good answer to this question shows you have felt the pain of these issues in practice.
Key Insight: The organizing principle for all three parts is the same: never use future information when making a decision, and account for the cost of every choice you make.
The Method:
(a) Rolling Walk-Forward Cross-Validation
- Partition the timeline into $K$ consecutive non-overlapping test windows, each of length $T_{\text{test}}$ (e.g., 6-12 months). Walk them forward chronologically: fold 1 tests on months 1-6, fold 2 on months 7-12, etc.
- Training set for fold $k$: all data strictly before the test window. Use an expanding window (all history) or a fixed rolling window (e.g., last 3-5 years) to avoid stale data dominating later folds.
- Embargo period: insert a gap of $\tau$ days between the end of training and start of testing (e.g., $\tau = 5$ to 20 trading days). This prevents feature leakage from overlapping return windows or slow-updating signals.
- Purging: if features for day $t$ use data up to day $t+k$ (e.g., forward returns in labels), remove observations within $k$ days of the test boundary from training entirely.
- Compute out-of-sample returns for each model's portfolio in each test fold, then concatenate across folds to get a full out-of-sample PnL series. Compute Sharpe ratio on this concatenated series -- not an average of fold-level Sharpes, which ignores serial structure.
(b) Testing Sharpe Ratio Differences
Let $d_t = r^{(B)}_t - r^{(A)}_t$ be the daily difference in portfolio returns (already scaled to unit vol). Define: $$\text{SR}_\Delta = \frac{\bar{d}}{\hat{\sigma}_d}$$ where $\bar{d}$ is the sample mean and $\hat{\sigma}_d$ is an autocorrelation-robust standard error (use Newey-West with $L$ lags, e.g., $L = \lfloor 4(T/100)^{2/9} \rfloor$).
- $H_0$: $E[d_t] \leq 0$ (Model B Sharpe $\leq$ Model A Sharpe)
- $H_1$: $E[d_t] > 0$ (Model B Sharpe $>$ Model A Sharpe)
Under $H_0$, by CLT (with HAC correction), $\sqrt{T} \cdot \bar{d} / \hat{\sigma}_d \xrightarrow{d} N(0,1)$, so use a one-sided z-test. Reject $H_0$ at level $\alpha$ if the z-statistic exceeds $z_\alpha$ (e.g., 1.645 for $\alpha = 5\%$).
Alternatively, use the Diebold-Mariano test which is designed for this exact setting -- it directly tests equal predictive accuracy between two models using a loss differential series, and is robust to autocorrelation.
For robustness, complement with a bootstrap: resample the $d_t$ series using a block bootstrap (block length matching the autocorrelation structure), compute the Sharpe differential on each resample, and read off the empirical p-value.
(c) Transaction Costs, Turnover, and Multiple Comparisons
Transaction costs and turnover: Compute a net-of-cost Sharpe by deducting estimated transaction costs. For each model, track daily turnover $\tau_t$ (fraction of portfolio rebalanced). Net daily return is: $$r^{\text{net}}_t = r^{\text{gross}}_t - c \cdot \tau_t$$ where $c$ is the one-way cost in basis points (e.g., 5-10 bps for liquid equities). A model with a slightly higher gross Sharpe but 3x the turnover may lose to Model A after costs. Turnover constraints can be imposed directly: penalize or disqualify models exceeding a turnover budget.
Multiple comparisons: If you test $M$ models, the probability of finding at least one spurious winner under the null of no skill is $1 - (1-\alpha)^M$. Fix this by: - Bonferroni correction: use significance level $\alpha/M$ for each individual test. Conservative but simple. - Benjamini-Hochberg (BH) procedure: controls the false discovery rate (FDR) rather than the family-wise error rate. More powerful when $M$ is large. - The Sharpe ratio haircut (Harvey-Liu): adjust the minimum required Sharpe upward based on the number of strategies tested. A commonly cited rule: require $\text{SR} \geq 1.0$ after a Bonferroni-type adjustment, not just $\text{SR} \geq 0.5$. - Out-of-sample holdout: reserve a final holdout period (never touched during model selection) and evaluate the winning model on it exactly once.
Answer: Use walk-forward CV with an embargo gap for out-of-sample evaluation; test Sharpe differences with a HAC-robust z-test or Diebold-Mariano; correct for multiple comparisons with BH or a dedicated holdout; always report net-of-cost Sharpe with turnover disclosed.
Intuition
The deeper lesson here is that model evaluation in finance is itself a statistical inference problem with all the same pitfalls as the models being evaluated. Walk-forward CV is not just good practice -- it is the only methodologically honest way to simulate what your model would have done had you deployed it. Every shortcut (peeking at future data, not embargoing, comparing on in-sample fits) inflates performance estimates in ways that are hard to unwind later.
The multiple comparisons issue is pervasive and underappreciated. The empirical finance literature has documented that a large fraction of published factor premia disappear out-of-sample, largely because researchers implicitly search over many specifications and report the best. In a live trading context, this means a strategy that looks compelling on a backtest may simply be the winner of a noisy lottery. Internalizing this -- and building evaluation pipelines that account for it explicitly -- is what separates rigorous quant research from data mining.