Why Use Adjusted R-Squared?
You are fitting a linear regression and your colleague suggests adding more predictors to improve the model. They point out that $R^2$ keeps increasing as you add variables. What is wrong with using $R^2$ as the sole criterion for model selection, and how does adjusted $R^2$ address this?
Hints
- Think about what happens to $R^2$ when you add a predictor that is pure random noise to a regression. Does it go up, down, or stay the same? Why?
- Compare the two formulas: $R^2 = 1 - \text{SSE}/\text{SST}$ vs. $R^2_{\text{adj}} = 1 - (\text{SSE}/(n-p-1))/(\text{SST}/(n-1))$. What extra quantity appears in the adjusted version?
- The penalty term is $n - p - 1$. As you add more predictors ($p$ increases), this shrinks, which inflates the SSE fraction. This means $R^2_{\text{adj}}$ can decrease when adding a predictor that does not reduce SSE enough to offset the lost degree of freedom.
Worked Solution
How to Think About It: $R^2$ measures the fraction of variance explained by the model. The problem is that it can only go up when you add a variable -- even if that variable is pure noise. This means an uncritical model builder can inflate $R^2$ arbitrarily by throwing in garbage predictors. Adjusted $R^2$ penalizes you for every extra parameter you add, so it only increases when a new variable earns its keep.
Key Insight: $R^2$ rewards complexity for free; adjusted $R^2$ makes complexity pay its own way through the degrees-of-freedom penalty.
The Method:
Recall the two formulas. Standard $R^2$: $$R^2 = 1 - \frac{\text{SSE}}{\text{SST}}$$ Adjusted $R^2$: $$R^2_{\text{adj}} = 1 - \frac{\text{SSE}/(n - p - 1)}{\text{SST}/(n-1)}$$
where $n$ is the number of observations and $p$ is the number of predictors.
The key difference: in $R^2_{\text{adj}}$, the residual variance is estimated as $\text{SSE}/(n-p-1)$ rather than raw $\text{SSE}$. As $p$ increases, the denominator $n-p-1$ shrinks, which inflates the SSE term and penalizes the $R^2_{\text{adj}}$ score. Adding a completely uninformative predictor reduces SSE by a tiny amount but costs one degree of freedom -- the net effect is a decrease in $R^2_{\text{adj}}$.
Formally, adding predictor $j$ to the model increases $R^2_{\text{adj}}$ if and only if the $t$-statistic on $\hat{\beta}_j$ satisfies $|t_j| > 1$, which is a much lower bar than statistical significance but at least rules out pure noise.
Practical Considerations: - Adjusted $R^2$ is still a in-sample metric and does not prevent overfitting as effectively as cross-validation. - For model selection among many predictors, prefer AIC, BIC, or out-of-sample validation. Adjusted $R^2$ is a quick check, not a complete solution. - In finance, adding lagged returns or correlated macro variables can inflate $R^2$ without adding genuine predictive power -- this is why out-of-sample $R^2$ is the standard in forecasting research.
Answer: $R^2$ never decreases when predictors are added, even if they are noise. Adjusted $R^2$ corrects for this by deflating the residual sum of squares by its degrees of freedom $(n-p-1)$, penalizing model complexity. It increases only when a new predictor reduces residual variance by more than the baseline noise level.
Intuition
The failure mode of $R^2$ is a specific instance of in-sample overfitting: you can always fit the training data better by adding parameters, but the improvements are illusory. Adjusted $R^2$ was one of the first attempts to penalize model complexity, predating more modern approaches like AIC, BIC, and cross-validation. All of these methods share the same philosophy: the benefit of a new parameter must exceed some threshold cost.
In quantitative finance, this issue is acute. A factor model with 50 factors will have a high $R^2$ on historical data almost by construction, but most of those factors are data-mined noise. The Fama-French and other parsimonious factor models are valuable precisely because they explain a lot of variation with very few parameters -- and because their factors have economic interpretations that survive out-of-sample. The lesson: always pair in-sample fit metrics with out-of-sample validation.