Bayesian ML, Overfitting, and Error Metrics

Worked Solution

How to Think About It: This is a breadth question -- the interviewer wants to see that you understand the landscape of ML methodology, not just one narrow technique. The key is to connect the three parts: Bayesian methods are themselves a form of overfitting prevention (part a feeds into part b), and the choice of error metric (part c) determines what "good generalization" even means in practice. A strong answer weaves these threads together: what prior beliefs do you bring (Bayesian), how do you keep the model honest (regularization / overfitting control), and how do you measure success (error metrics matched to the cost structure). The unifying theme is: how do you build models that *generalize*, and how do you *measure* that they do?

Quick Estimate: Before the detail, the one-line skeleton you want to land: (a) a regularization penalty *is* a log-prior, so MAP = regularized MLE, and you go full-Bayes when uncertainty/small-sample/cost-of-overconfidence matters; (b) overfitting = fitting noise, fought with a layered toolkit (validation splits, regularization, simpler models, early stopping, dropout, ensembling, more/augmented data, leakage control) chosen by model class and data structure; (c) error metrics split into regression (MAE, MSE/RMSE, MAPE/SMAPE, Huber, quantile, R^2, likelihood) and classification (accuracy, precision/recall/F1, ROC-AUC, PR-AUC, log loss, Brier, calibration), and you pick the one whose implied loss matches the *asymmetric business cost*.

Approach: Answer each sub-question in turn, but explicitly cross-reference: the prior/penalty equivalence from (a) is the first tool in (b), and the cost structure in (c) is what tells you which errors the regularization in (b) should be tolerant of. Keep each part 'when and why', not just a list.

Formal Solution:

(a) Bayesian Methods in ML

The Bayesian framework treats model parameters $\theta$ as random variables with a prior distribution $p(\theta)$. After observing data $D$, you update via Bayes' rule:

$$p(\theta \mid D) = \frac{p(D \mid \theta)\,p(\theta)}{p(D)} \propto p(D \mid \theta)\, p(\theta).$$

Here $p(D\mid\theta)$ is the *likelihood* and $p(\theta\mid D)$ the *posterior*. Predictions use the full *posterior predictive*, integrating over parameter uncertainty rather than plugging in a single point estimate:

$$p(y_{\text{new}} \mid D) = \int p(y_{\text{new}} \mid \theta)\, p(\theta \mid D)\, d\theta.$$

Practical connections:

• Regularization as a prior (the key equivalence): maximizing the *log-posterior* $\log p(D\mid\theta) + \log p(\theta)$ is maximum-likelihood plus an additive penalty equal to the log-prior. A Gaussian prior $\theta_i \sim N(0,\sigma^2)$ has log-density $\propto -\theta_i^2/(2\sigma^2)$, i.e. a *squared-norm* penalty -- this is exactly L2 / Ridge regularization. A Laplace prior $\theta_i \sim \text{Laplace}(0,b)$ has log-density $\propto -|\theta_i|/b$, i.e. an *absolute-value* penalty -- this is L1 / Lasso, and the Laplace density's non-differentiable peak at zero is what drives exact-zero coefficients (sparsity). So every regularization penalty is implicitly a Bayesian prior; a flat/uniform prior recovers plain MLE (no penalty).
• MAP vs. full Bayes: Maximum a posteriori (MAP) estimation finds the single most probable $\theta$ (the mode) -- this is what regularized MLE gives you, a *point estimate*. Full Bayesian inference instead retains the whole posterior, giving *calibrated uncertainty*. In quant finance this matters: "expected return 2% with a wide posterior" is a very different trade from "2% with a tight posterior."
• When to go full Bayesian: (i) limited data / small-sample regimes (common in finance), where the prior does real work and point estimates are unstable; (ii) when you need uncertainty quantification for risk management or position sizing; (iii) hierarchical / multilevel structure (e.g. partial pooling across assets or regimes), where a hierarchical prior shares strength across groups; and (iv) when the cost of overconfidence is high (tail-risk decisions). When data is abundant and you only need a fast point prediction, MAP / regularized MLE is usually enough and far cheaper than sampling the posterior.

(b) Overfitting Prevention Toolkit

Overfitting happens when the model fits *noise* in the training data rather than the underlying signal -- training error keeps falling while validation/out-of-sample error rises. The core trade-off is bias vs. variance: overfit models are low-bias, high-variance. The toolkit, organized by *when and why* you reach for each:

1. Proper data splitting / cross-validation (the diagnostic that comes first). You cannot fight overfitting you cannot measure. Use a clean train/validation/test split, and K-fold CV for i.i.d. data. Crucially, for time series (most of quant finance), use walk-forward / expanding-window CV so the model is never trained on the future -- ordinary K-fold leaks look-ahead information and flatters the model.
2. Regularization (L1/L2/Elastic Net) -- your first modeling line of defense. L2 (Ridge) shrinks all coefficients toward zero: use when you believe most features contribute a little (the Gaussian-prior view from part a). L1 (Lasso) drives some coefficients exactly to zero: use when you suspect many features are irrelevant and want built-in feature selection (the Laplace-prior view). Elastic Net blends both when features are correlated.
3. Simpler models / reduced complexity. The cheapest fix is often a lower-capacity model: fewer features, lower polynomial degree, shallower trees, smaller networks. Prefer this when data is scarce relative to parameters -- a simpler model that generalizes beats a complex one that memorizes.
4. Feature selection / dimensionality reduction. Drop or combine features (filter/wrapper methods, PCA) to cut variance when you have many weakly-informative or collinear predictors.
5. Early stopping. For iterative learners (gradient boosting, neural nets), monitor validation loss and stop when it starts rising. It implicitly caps effective complexity and is essentially free.
6. Dropout. Specific to neural networks: randomly zero out units during training so the net cannot rely on any single neuron, forcing redundant representations. Mathematically akin to training an ensemble of sub-networks; use when a deep net overfits and you cannot easily get more data.
7. Ensemble methods. Bagging (e.g. Random Forest) averages over bootstrap samples to reduce *variance* -- good for high-variance base learners. Boosting (e.g. XGBoost) sequentially corrects errors to reduce *bias*, but its later rounds overfit, so it needs regularization (shrinkage, max depth, subsampling) and early stopping.
8. More data / data augmentation. The most reliable cure for variance is more data. When you cannot get it, synthesize examples via label-preserving transformations -- ubiquitous in images/NLP, less natural for tabular financial data (though resampling, jittering, or simulation can help).
9. Leakage control. Overfitting's evil twin: features that secretly encode the target (e.g. using future information, target-derived features, or fitting scalers on the full dataset before splitting). Audit the pipeline so all preprocessing is fit on training folds only -- otherwise every metric is a lie.
10. Hyperparameter tuning. Choose regularization strength, tree depth, learning rate, etc. on validation data (grid/random/Bayesian search), and report final performance on a held-out test set you never touched during tuning.

The meta-point: there is no single fix. You diagnose with proper validation, then pick tools by *model class* (dropout for nets, depth/shrinkage for boosting, L1/L2 for linear) and *data structure* (walk-forward for time series, augmentation when data is scarce).

(c) Error Metrics and Cost Structure

The metric you optimize *is* the implicit definition of a good model, so it must match the problem's cost structure. Two families:

*Regression metrics:* - MAE (mean absolute error): robust to outliers, treats all errors linearly; corresponds to predicting the conditional *median*. - MSE / RMSE: squares errors, so it penalizes large misses heavily and corresponds to predicting the conditional *mean*; sensitive to outliers (which can be a feature or a bug). RMSE is in the units of the target. - MAPE / SMAPE (percentage errors): scale-free, useful when relative error matters across different magnitudes; MAPE blows up near zero and is asymmetric, SMAPE partially fixes the symmetry. - Huber loss: quadratic for small residuals, linear for large ones -- a tunable compromise between MSE's sensitivity and MAE's robustness; good when you want mean-like behavior but with outlier protection. - Quantile (pinball) loss: targets a specific quantile, giving asymmetric penalties for over- vs. under-prediction -- exactly what you want when the cost of over- and under-shooting differ (e.g. inventory, VaR, demand). - $R^2$: variance explained, a unitless goodness-of-fit summary (not a loss to optimize directly); useful for communicating but can mislead with non-stationary data. - Likelihood-based losses (e.g. Gaussian/Poisson negative log-likelihood): the principled choice when you have a probabilistic model and care about the full predictive distribution.

*Classification metrics:* - Accuracy: fraction correct -- fine for balanced classes, badly misleading under imbalance (99% accuracy is trivial if 99% of cases are one class). - Precision / Recall / F1: precision = of predicted positives, how many are right (cost of *false positives*); recall = of actual positives, how many you caught (cost of *false negatives*); F1 is their harmonic mean. Choose based on which error is costlier -- e.g. fraud/disease detection prioritizes recall, spam filtering may prioritize precision. - ROC-AUC: ranking quality across all thresholds, threshold-independent; can look optimistic under heavy class imbalance. - PR-AUC (precision-recall AUC): more informative than ROC-AUC when positives are rare, because it ignores the abundant true negatives. - Log loss (cross-entropy) and Brier score: *proper scoring rules* that reward well-*calibrated probabilities*, not just correct labels -- use when you act on the probability itself (position sizing, Kelly bets, expected-value decisions). - Calibration (reliability diagrams): a diagnostic that predicted probabilities match observed frequencies -- essential whenever downstream decisions multiply probability by payoff. - Cost-weighted / confusion-matrix loss: assign explicit dollar costs to each cell of the confusion matrix and minimize expected cost; this is the most direct way to encode an *asymmetric* cost structure and to set the decision *threshold* accordingly.

*How cost structure changes the choice:* when errors are asymmetric (a false negative on fraud costs far more than a false positive), move from accuracy toward recall, cost-weighted loss, or quantile loss, and shift the decision threshold off 0.5. Under class imbalance, prefer PR-AUC, precision/recall, and resampling over raw accuracy. When you need *tail-risk* control, square the errors (MSE/RMSE) or use quantile loss at extreme quantiles. When downstream decisions consume the *probability* (sizing a bet), optimize and check a proper scoring rule (log loss, Brier) plus calibration, not just a hard-label metric.

Answer: (a) Bayesian ML treats parameters as random with a prior, updates to a posterior via Bayes' rule, and predicts with the posterior predictive; a regularization penalty is exactly a log-prior, so MAP = regularized MLE, with L2/Ridge $\Leftrightarrow$ Gaussian prior and L1/Lasso $\Leftrightarrow$ Laplace prior (sparsity) -- go full Bayes when you need uncertainty, have small samples or hierarchy, or when overconfidence is costly. (b) Overfitting is fitting noise; fight it with proper (time-series-aware) validation, regularization, simpler models, feature selection, early stopping, dropout, ensembling, more/augmented data, leakage control, and tuning -- chosen by model class and data structure. (c) Regression metrics (MAE, MSE/RMSE, MAPE/SMAPE, Huber, quantile, $R^2$, likelihood) and classification metrics (accuracy, precision/recall/F1, ROC-AUC, PR-AUC, log loss, Brier, calibration, cost-weighted loss) should be selected so the implied loss matches the problem's asymmetric costs, class imbalance, tail risk, calibration needs, and decision thresholds.