Mean as the Minimizer of Squared Loss
You have a set of data points $x_1, x_2, \ldots, x_n \in \mathbb{R}$. You want to find the single number $y$ that is closest to all of them in the least-squares sense.
Find $\arg\min_{y \in \mathbb{R}} \sum_{i=1}^{n} (x_i - y)^2$.
Then: what changes if you use absolute value instead of squared differences?
Hints
- This is a simple unconstrained optimization -- take the derivative of the sum with respect to $y$ and set it to zero.
- When you differentiate $\sum_i (x_i - y)^2$ with respect to $y$, you get $-2\sum_i (x_i - y)$. Setting this to zero gives a condition on the sum of residuals.
- Setting $\sum_i (x_i - y) = 0$ means $ny = \sum_i x_i$, which directly gives $y = \bar{x}$. Check the second derivative to confirm it is a minimum.
Worked Solution
How to Think About It: You are finding the best single-number summary of a dataset under a specific loss function. This is a one-dimensional unconstrained optimization -- just differentiate and set to zero. The result connects directly to a fundamental principle: the sample mean is the point that minimizes squared distance to the data. This is not a coincidence; it is the definition of the mean from an optimization perspective.
Quick Estimate: Intuition check -- if all $x_i$ are equal to some constant $c$, the answer is obviously $c$. If $x_i$ are spread around, the minimizer should be "in the middle" -- the mean is the right guess.
Formal Solution:
Let $f(y) = \sum_{i=1}^{n} (x_i - y)^2$. Differentiate with respect to $y$:
$$f'(y) = \sum_{i=1}^{n} 2(x_i - y) \cdot (-1) = -2 \sum_{i=1}^{n} (x_i - y) = -2\left(\sum_{i=1}^{n} x_i - ny\right)$$
Setting $f'(y) = 0$:
$$\sum_{i=1}^{n} x_i - ny = 0 \implies y = \frac{1}{n} \sum_{i=1}^{n} x_i = \bar{x}$$
Confirm this is a minimum, not a maximum: $f''(y) = 2n > 0$. Confirmed.
The $L_1$ analog: For $g(y) = \sum_{i=1}^n |x_i - y|$, the minimizer is the sample median $\tilde{x}$. Intuitively, the mean is pulled by extreme values (outliers create large squared errors) while the median is not (absolute value grows linearly, not quadratically, so outliers have less leverage).
Answer: $\arg\min_y \sum_i (x_i - y)^2 = \bar{x}$ (the sample mean). The analogous minimizer for $L_1$ loss is the sample median.
Intuition
The mean and median are not just descriptive statistics -- they are optimizers of specific loss functions. The mean minimizes $L_2$ loss (sum of squared deviations), and the median minimizes $L_1$ loss (sum of absolute deviations). This optimization framing is not just a theoretical curiosity: it tells you exactly why the mean is sensitive to outliers (a single extreme point can shift it arbitrarily far) while the median is robust (moving one point far from the others does not change it at all).
In quant work, this shows up in regression. Ordinary least squares (OLS) minimizes the sum of squared residuals -- so the fitted model is pulled toward outlier observations just like the mean is pulled toward outlier data points. Quantile regression minimizes a weighted $L_1$ loss and produces estimates robust to extreme residuals. Understanding which loss function you are minimizing tells you immediately what your estimator is sensitive to.