Cross-Sample Correlation and Model Transferability

Worked Solution

How to Think About It: This is a model generalization problem dressed up as a correlation question. Before touching any formulas, get the conceptual map straight. Each sample has its own "true" relationship between $X$ and $y$ -- if those relationships differ (a structural break), then a model fit on one sample will not transfer cleanly to the other. The three sub-questions each probe a different aspect: (1) can in-sample fit differ across regimes? (2) does out-of-sample performance match in-sample performance? (3) how do you quantify the generalization gap? Think of this the way you would think about a strategy that works in one market regime but not another -- the ratio in part (3) is basically your "regime stability score."

Part 1: Can the correlations differ across samples?

Yes, absolutely. Two mechanisms can cause this:

• Structural break: The true relationship $y = f(X) + \varepsilon$ is different in the two subpopulations. If sample A is drawn from a regime where the predictors are only weakly related to $y$, and sample B is drawn from a regime with a stronger relationship, then the in-sample $R^2$ (and hence correlation of $\hat{y}$ to $y$) will be higher in sample B even with fewer observations.

• Overfitting with fewer observations: A model fit on 3,000 observations has more degrees of freedom per observation than one fit on 7,000. If the number of predictors is the same, the smaller sample model can "memorize" noise more easily, inflating its in-sample correlation. In other words, the 0.25 in sample B might partly reflect overfitting, not a better underlying signal.

In practice, both effects can coexist: a genuine structural difference in regimes plus some overfitting in the smaller sample. The hint in the problem points primarily to the structural break interpretation, which is the more interesting and practically relevant one.

Part 2: Will $\text{corr}(\hat{y}'', y)$ exceed 0.25?

Almost certainly not -- and it will likely be lower than 0.25.

Here is why. The sample-A model was calibrated on a population that may have a different signal structure than sample B. When you apply it to sample B, three things work against you:

1. Regime mismatch: If there is a structural break, the coefficients learned from sample A are optimized for the wrong distribution. They may point in the right direction qualitatively but are mis-scaled or miss key interactions present only in regime B.

1. No in-sample advantage: The sample-B model ($\hat{y}'$) was trained on sample B data and thus exploits whatever signal is available there. The sample-A model gets no such head start on sample B.

1. Estimation noise from sample A: Even if the two regimes were identical, transferring estimated coefficients introduces noise. The sample-A estimates have their own sampling error, and that error propagates into the predictions on sample B.

Formal bound: Under the assumption of the same true model across both samples, the out-of-sample correlation of the sample-A model applied to sample B is bounded above by $\text{corr}(\hat{y}', y) = 0.25$ in expectation (since the sample-B model is the MLE for that population). With a structural break, it can be much lower.

Part 3: Interpreting the ratio $\dfrac{\text{corr}(\hat{y}'',\, y)}{\text{corr}(\hat{y}',\, y)}$

This ratio is a direct measure of model transferability -- or equivalently, regime consistency.

• Ratio = 1: Perfect transferability. The model trained on sample A performs just as well on sample B as sample B's own model. This would mean the two samples are exchangeable draws from the same population. No structural break.

• Ratio close to 1 but below 1: Minor regime shift or just estimation noise. The sample-A model loses a little on sample B, but the underlying signal structure is largely the same. This is the best you can hope for in practice.

• Ratio well below 1 (say, < 0.7): Significant regime inconsistency. The model trained on sample A is materially worse at predicting sample B than sample B predicts itself. The two populations have different $X$-to-$y$ relationships -- a structural break is likely.

• Ratio near 0 or negative: Severe regime mismatch. The sample-A model is essentially useless (or perversely wrong) on sample B.

In machine learning language, this ratio is like a cross-dataset transfer score. In quant finance, it maps directly to the question: "Does my model trained on historical data from one market regime generalize to a different regime?" A ratio well below 1 means your model is regime-specific and will break down during regime transitions -- exactly the failure mode that kills many systematic strategies.

Answer:

1. Yes -- structural breaks across subpopulations or overfitting in the smaller sample (or both) can produce different in-sample correlations.

1. No -- $\text{corr}(\hat{y}'', y)$ will typically be lower than 0.25 due to regime mismatch and the fact that sample B's own model has an inherent advantage.

1. The ratio measures model transferability. A ratio below 1 means the cross-sample model underperforms the native model, indicating regime inconsistency. The further it is below 1, the more severe the structural break between the two populations.