OLS Coefficient Confidence Intervals
Consider the simple linear regression model $Y_i = \beta_0 + \beta_1 x_i + \varepsilon_i$ for $i = 1, \ldots, n$, where the errors $\varepsilon_i$ are i.i.d. $N(0, \sigma^2)$.
- Derive the distribution of the OLS estimator $\hat{\beta}_1$ and use it to construct a confidence interval for $\beta_1$.
- What assumptions are required for the confidence interval to be valid? What happens if they are violated?
Hints
- Start by writing $\hat{\beta}_1$ as a linear function of the $Y_i$'s. Since linear combinations of normals are normal, you immediately know the distribution's shape.
- The variance of $\hat{\beta}_1$ is $\sigma^2 / S_{xx}$ where $S_{xx} = \sum(x_i - \bar{x})^2$. Since $\sigma^2$ is unknown, replacing it with the residual estimate gives a $t$-distribution rather than a $z$.
- The critical assumptions are: normality of errors (for exact $t$-distribution), independence (for the variance formula), and homoscedasticity (for the single $\sigma^2$ to make sense). Think about which of these can be relaxed asymptotically.
Worked Solution
How to Think About It: The OLS estimator $\hat{\beta}_1$ is a linear combination of the $Y_i$'s, which are normal under our assumptions. So $\hat{\beta}_1$ is itself normal. The confidence interval comes from pivoting: take $\hat{\beta}_1$, subtract the true $\beta_1$, divide by the standard error, and you get a $t$-distribution (because we estimate $\sigma^2$ from the residuals). The only real work is computing the variance of $\hat{\beta}_1$.
Key Insight: The width of the confidence interval is driven by two things: the noise level $\sigma^2$ and the spread of the $x_i$'s. More spread in $x$ means a tighter estimate of the slope -- this is why designed experiments space out their treatment levels.
The Method:
Step 1: OLS estimator. In simple linear regression:
$$\hat{\beta}_1 = \frac{\sum_{i=1}^n (x_i - \bar{x})(Y_i - \bar{Y})}{\sum_{i=1}^n (x_i - \bar{x})^2} = \frac{S_{xY}}{S_{xx}}$$
where $S_{xx} = \sum (x_i - \bar{x})^2$.
Step 2: Distribution of $\hat{\beta}_1$. Since $\hat{\beta}_1$ is a linear combination of the $Y_i$'s and each $Y_i \sim N(\beta_0 + \beta_1 x_i, \sigma^2)$:
$$\hat{\beta}_1 \sim N\left(\beta_1, \frac{\sigma^2}{S_{xx}}\right)$$
The variance $\sigma^2 / S_{xx}$ makes intuitive sense: more noise ($\sigma^2$ large) or less spread in $x$ ($S_{xx}$ small) means more uncertainty in the slope estimate.
Step 3: Pivot to a $t$-statistic. We don't know $\sigma^2$, so we estimate it with the residual variance:
$$\hat{\sigma}^2 = \frac{1}{n-2} \sum_{i=1}^n (Y_i - \hat{Y}_i)^2 = \frac{\text{RSS}}{n-2}$$
Under the model assumptions, $\hat{\sigma}^2$ is independent of $\hat{\beta}_1$ and $(n-2)\hat{\sigma}^2/\sigma^2 \sim \chi^2_{n-2}$. Therefore:
$$t = \frac{\hat{\beta}_1 - \beta_1}{\hat{\sigma}/\sqrt{S_{xx}}} \sim t_{n-2}$$
Step 4: Confidence interval. Inverting the pivot:
$$\hat{\beta}_1 \pm t_{n-2, \alpha/2} \cdot \frac{\hat{\sigma}}{\sqrt{S_{xx}}}$$
This is the $(1-\alpha)$ confidence interval for $\beta_1$.
Assumptions required:
- Linearity: $E[Y_i \mid x_i] = \beta_0 + \beta_1 x_i$. If violated, the estimator is biased.
- Independence: The $\varepsilon_i$ are independent. Serial correlation (common in time series) invalidates the standard error formula -- the actual variance of $\hat{\beta}_1$ could be much larger than $\sigma^2/S_{xx}$.
- Homoscedasticity: $\text{Var}(\varepsilon_i) = \sigma^2$ for all $i$. Heteroscedasticity doesn't bias $\hat{\beta}_1$ but makes the standard error wrong. Use heteroscedasticity-robust (White) standard errors in practice.
- Normality: $\varepsilon_i \sim N(0, \sigma^2)$. This is needed for the exact $t$-distribution in finite samples. For large $n$, the CLT gives approximate normality of $\hat{\beta}_1$ even without normal errors.
Practical Considerations: In finance, assumptions 2 and 3 are almost always violated. Time series residuals are serially correlated and heteroscedastic (volatility clustering). Use Newey-West standard errors for serial correlation and White standard errors for heteroscedasticity. The confidence interval formula stays the same -- only the standard error computation changes.
Answer: The $(1-\alpha)$ confidence interval for $\beta_1$ is $\hat{\beta}_1 \pm t_{n-2, \alpha/2} \cdot \hat{\sigma}/\sqrt{S_{xx}}$, valid under linearity, independence, homoscedasticity, and normality (the last can be relaxed for large $n$ via CLT).
Intuition
The confidence interval for a regression coefficient is really just a noise-to-signal ratio in disguise. The 'signal' is how spread out your $x$-values are (captured by $S_{xx}$) -- more spread means the slope is easier to pin down. The 'noise' is the residual variance $\sigma^2$. The ratio $\sigma^2/S_{xx}$ is the variance of your slope estimate, and the confidence interval is just the estimate plus or minus a few standard errors.
In practice, the formula itself matters less than understanding when it breaks. In finance, the two biggest killers are serial correlation (which makes the effective sample size much smaller than $n$) and heteroscedasticity (which means some observations are much noisier than others). Both problems don't bias the point estimate but destroy the confidence interval's coverage. A 95% CI that actually covers 60% of the time is worse than useless -- it gives false confidence. That's why robust standard errors (Newey-West, White) are standard practice on any trading desk.