R-Squared Invariance Under Transformations
Consider the simple linear regression $Y = \alpha + \beta X + \epsilon$.
(a) If you multiply all $X$ values by a constant $c \neq 0$, what happens to $R^2$?
(b) If you add a constant $k$ to all $Y$ values, what happens to $R^2$?
(c) If you add a new predictor $Z$ that is perfectly collinear with $X$ (i.e., $Z = aX + b$ for constants $a, b$), can $R^2$ increase?
Hints
- In simple regression, $R^2 = r_{XY}^2$. What are the invariance properties of correlation?
- Covariance is linear in each argument, and variance is unaffected by adding a constant. Use these to show correlation is invariant under affine transformations.
- For part (c), think in terms of the column space of the design matrix. Does adding $Z = aX + b$ expand the span of $[\mathbf{1}, X]$?
Worked Solution
How to Think About It: In simple linear regression, $R^2 = r_{XY}^2$, the square of the Pearson correlation between $X$ and $Y$. Correlation has a well-known invariance property: it is unchanged by any affine transformation $X \to aX + b$ (as long as $a \neq 0$). Since $R^2$ is just the square of the correlation, the answer to all three parts is the same: $R^2$ does not change. The intuition is that correlation measures the strength of the linear relationship, not the scale or location of the data.
Quick Sanity Checks: - Multiplying $X$ by $c$ stretches the scatter plot horizontally but does not change how tightly the points cluster around the best-fit line. - Shifting $Y$ by $k$ moves the scatter plot vertically but does not change the spread or the fit quality. - Adding a perfectly collinear predictor gives the model no new information -- it is literally the same variable relabeled.
Formal Solution:
(a) Multiplying $X$ by $c \neq 0$: $R^2$ is unchanged.
Let $X' = cX$. The correlation between $X'$ and $Y$ is:
$$r_{X'Y} = \frac{\text{Cov}(cX, Y)}{\text{SD}(cX) \cdot \text{SD}(Y)} = \frac{c \cdot \text{Cov}(X, Y)}{|c| \cdot \text{SD}(X) \cdot \text{SD}(Y)} = \text{sign}(c) \cdot r_{XY}$$
Since $R^2 = r^2$, squaring removes the sign: $R^2$ is unchanged. Note that if $c < 0$, the sign of $\hat{\beta}$ flips, but $R^2$ does not care about direction.
(b) Adding $k$ to all $Y$: $R^2$ is unchanged.
Let $Y' = Y + k$. Then:
$$\text{Cov}(X, Y') = \text{Cov}(X, Y + k) = \text{Cov}(X, Y)$$
$$\text{SD}(Y') = \text{SD}(Y + k) = \text{SD}(Y)$$
So $r_{XY'} = r_{XY}$ and $R^2$ is unchanged. Adding a constant shifts the intercept $\hat{\alpha}$ but affects nothing else.
(c) Adding a perfectly collinear predictor: $R^2$ cannot increase.
If $Z = aX + b$, then $Z$ lies in the column space of $[\mathbf{1}, X]$. The design matrix $[\mathbf{1}, X, Z]$ has the same column space as $[\mathbf{1}, X]$:
$$\text{span}\{\mathbf{1}, X, Z\} = \text{span}\{\mathbf{1}, X\}$$
The OLS fitted values $\hat{Y}$ are the projection of $Y$ onto the column space, which is identical in both models. Therefore $R^2 = \|\hat{Y} - \bar{Y}\|^2 / \|Y - \bar{Y}\|^2$ is exactly the same.
In practice, the design matrix is rank-deficient, and standard software (R, Python, etc.) will either drop $Z$ or report a warning about perfect multicollinearity. The model is not identifiable -- there are infinitely many coefficient combinations $(\hat{\beta}_X, \hat{\beta}_Z)$ that produce the same fitted values.
Answer: $R^2$ is unchanged in all three cases. (a) Correlation is scale-invariant. (b) Correlation is shift-invariant. (c) A collinear predictor adds no new information to the column space.
Intuition
$R^2$ measures the fraction of variance in $Y$ explained by the linear relationship with $X$. This is a purely geometric quantity -- the angle between $Y$ (centered) and the column space of $X$ (centered). Rescaling $X$, shifting $Y$, or adding redundant predictors all leave this angle unchanged. The fitted values are the same, the residuals are the same, and $R^2$ is the same.
This invariance is practically important in regression modeling. It means you do not need to worry about units or centering when interpreting $R^2$ (though you should worry about them for interpreting coefficients). It also means that adding collinear features is useless for improving fit -- a mistake that trips up many junior analysts who throw in correlated features hoping to boost $R^2$. In multiple regression, $R^2$ can only increase with genuinely new information (a variable not in the current column space), which is why adjusted $R^2$ penalizes for additional predictors.