Gradient Boosting vs. Random Forests, Batch Normalization, and SGD Momentum

Worked Solution

How to Think About It: These are conceptual ML questions where the interviewer wants practical intuition, not a recitation of Wikipedia. For each, lead with the core distinction -- what problem does this technique solve, and what is the mechanism? Then give a concrete use case or failure mode that shows you understand it deeply.

---

Part (a): Gradient Boosting vs. Random Forests

Both are ensemble methods that combine many decision trees, but they differ fundamentally in *how* the trees are built and *what error* they address.

Random Forests (bagging): - Build many trees *independently* on random bootstrap samples of the data, with random subsets of features. - Average (or majority-vote) predictions across all trees. - Each tree is fully grown -- high variance, low bias individually. The ensemble reduces variance through averaging. - Effect: reduces overfitting. A single deep tree wildly overfits; averaging 500 of them mostly cancels out the noise. - Robust, parallelizable, few hyperparameters to tune. Hard to seriously overfit with a large forest.

Gradient Boosting (boosting): - Build trees *sequentially*. Each new tree fits the negative gradient of the loss function -- effectively the residual errors of the current ensemble. - Trees are shallow (depth 3-6) -- high bias, low variance individually. The ensemble reduces bias through accumulation. - Effect: reduces bias. It keeps correcting systematic errors, round by round. - More powerful but more fragile: learning rate, number of trees, and depth all need tuning. Can overfit badly with too many rounds or too high a learning rate.

When to use which: - Random Forests: when you need something that works out of the box, when data is noisy, or when training speed matters (parallelizes trivially). - Gradient Boosting (XGBoost, LightGBM): when you need maximum predictive accuracy and have time to tune. Dominates tabular data competitions.

---

Part (b): Batch Normalization

Batch normalization (BN) normalizes each layer's pre-activation inputs to have zero mean and unit variance, computed over the current mini-batch. It then applies learnable scale and shift parameters $\gamma$ and $\beta$:

$$\hat{x}_i = \frac{x_i - \mu_B}{\sqrt{\sigma_B^2 + \epsilon}}, \quad y_i = \gamma \hat{x}_i + \beta$$

Why does it help? Two mechanisms:

1. Reduces internal covariate shift: as the weights in earlier layers change during training, the distribution of inputs to later layers shifts. This forces later layers to constantly readjust, slowing learning. BN stabilizes these distributions, so each layer can assume its input distribution is roughly constant.

1. Implicit regularization: the mean and variance computed per mini-batch are noisy estimates of the true statistics. This noise acts like a regularizer and often reduces the need for dropout.

Practical effect: allows much higher learning rates, faster convergence, and less sensitivity to initialization. Almost universally used in deep vision architectures (ResNets, etc.).

---

Part (c): Momentum in SGD

Standard SGD update: $$\theta \leftarrow \theta - \eta \nabla_{\theta} L$$

With momentum parameter $\gamma$ (typically 0.9): $$v \leftarrow \gamma v + \eta \nabla_{\theta} L$$ $$\theta \leftarrow \theta - v$$

The velocity $v$ is an exponentially weighted moving average of past gradients. This helps in two ways:

1. Acceleration in consistent directions: if the gradient consistently points the same way (say, left), the velocity builds up and steps get larger. You accelerate through long ravines in the loss surface.

1. Dampening in oscillating directions: in directions where the gradient flips sign (common in narrow, curved regions of the loss surface), positive and negative contributions cancel in the moving average, reducing oscillation.

The physical analogy: a heavy ball rolling downhill builds momentum and rolls through small bumps and shallow local minima, rather than getting stuck or bouncing erratically.

Answer: Random Forests reduce variance by averaging independent trees; Gradient Boosting reduces bias by sequentially correcting residuals. Batch normalization stabilizes layer input distributions during training, allowing faster learning. SGD momentum accumulates a gradient moving average, accelerating progress along consistent directions and dampening oscillations.