Diagnosing Overfitting and Cross-Validation

Worked Solution

How to Think About It: When training performance is high but test performance is poor, the model has memorized the training set rather than learning the underlying pattern. This is overfitting -- the model captures noise and idiosyncrasies specific to the training data that do not generalize. In a quant context, this is exactly what happens when you over-optimize a trading strategy on historical data: it looks amazing in backtest and blows up live. The interviewer wants you to both diagnose the root causes and explain the standard remedy.

Key Insight: Overfitting means the model's complexity exceeds what the data can support. The fix is to estimate out-of-sample performance during training (via cross-validation) and use it to control model complexity.

The Method:

*Root causes of overfitting:*

1. Model too complex -- too many parameters relative to the amount of training data (e.g., a deep neural net on 100 data points, or a polynomial of degree 50 fit to 60 observations).

1. Insufficient training data -- even a reasonable model can overfit if the dataset is too small to distinguish signal from noise.

1. No regularization -- without penalties on model complexity (L1, L2, dropout, early stopping), the optimizer will happily fit every data point exactly.

1. Noisy or irrelevant features -- including features that have no true predictive power gives the model extra degrees of freedom to memorize noise.

1. Data leakage -- information from the test set accidentally bleeds into training (e.g., normalizing features using the full dataset before splitting, or look-ahead bias in time series).

*Using cross-validation to detect and fix it:*

1. $k$-fold CV setup -- split the training data into $k$ folds (typically $k = 5$ or
   0$). For each fold, train on the remaining $k-1$ folds and evaluate on the held-out fold. Average the $k$ validation scores to estimate true out-of-sample performance.

1. Diagnose the gap -- compare the average training score to the average CV validation score. A large gap (high train, low validation) confirms overfitting. A small gap with both scores low indicates underfitting.

1. Model selection -- sweep over complexity parameters (tree depth, number of features, regularization strength $\lambda$, network width) and pick the value that maximizes the CV validation score, not the training score.

1. Regularization tuning -- use CV to select the regularization hyperparameter. For example, in ridge regression, plot CV error vs. $\lambda$ and choose the $\lambda$ at the elbow.

1. Feature selection -- use CV-based metrics (or embedded methods like Lasso) to drop features that do not improve validation performance.

Practical Considerations:

• For time series data, standard $k$-fold CV introduces look-ahead bias. Use walk-forward (expanding window) or purged CV instead.
• Nested CV is needed if you are both selecting hyperparameters and estimating final performance -- the inner loop tunes, the outer loop estimates.
• Stratified folds are important for imbalanced classification problems.
• Beware of "CV overfitting" -- if you try thousands of model configurations and pick the best CV score, you can overfit to the CV folds themselves.

Answer: Poor test performance with good training performance is overfitting, caused by excessive model complexity, insufficient data, lack of regularization, or data leakage. Cross-validation detects it by estimating out-of-sample error during training, and fixes it by guiding model selection and regularization tuning toward the complexity level that generalizes best.