Fisher Information for Exponential Samples
Suppose $X_1, \dots, X_n \mid \lambda$ are IID $\text{Exponential}(\lambda)$ random variables. Find the Fisher Information $I(\lambda)$ for the rate parameter $\lambda$.
What is the value of the Fisher Information when $\lambda = 3$ and $n = 16$?
Hints
- Fisher Information for IID samples is additive -- compute it for a single observation and multiply by $n$.
- The exponential PDF is $f(x \mid \lambda) = \lambda e^{-\lambda x}$. Take the log, differentiate twice with respect to $\lambda$, and negate the expectation.
- The second derivative of $\log f$ is $-1/\lambda^2$, which is constant in $x$, so the expectation step is trivial.
Worked Solution
How to Think About It: Fisher Information measures how much information a sample carries about a parameter -- specifically, how sharply the log-likelihood peaks around the true value. For an exponential distribution with rate $\lambda$, the PDF is simple ($\lambda e^{-\lambda x}$), so the log-likelihood is clean and the second derivative is easy to compute. Since the observations are IID, total Fisher Information is just $n$ times the single-observation information.
Quick Estimate: For a single exponential observation, the MLE of $\lambda$ is $\hat{\lambda} = 1/X$, and by the Cramer-Rao bound, the variance of any unbiased estimator is at least $1/I(\lambda)$. Since $\text{Var}(\hat{\lambda})$ for the MLE scales like $\lambda^2/n$ (the exponential has coefficient of variation 1), we expect $I(\lambda) = n/\lambda^2$. For $\lambda = 3, n = 16$: $I = 16/9 \approx 1.78$.
Approach: Compute the single-observation Fisher Information using the second derivative of the log-likelihood, then multiply by $n$.
Formal Solution:
The PDF of a single observation is: $$f(x \mid \lambda) = \lambda e^{-\lambda x}, \quad x > 0$$
The support does not depend on $\lambda$, so we can use the standard formula: $$I_1(\lambda) = -E\left[\frac{\partial^2}{\partial \lambda^2} \log f(X \mid \lambda)\right]$$
Step 1 -- Log-likelihood for one observation: $$\log f(x \mid \lambda) = \log \lambda - \lambda x$$
Step 2 -- First derivative (score function): $$\frac{\partial}{\partial \lambda} \log f(x \mid \lambda) = \frac{1}{\lambda} - x$$
Step 3 -- Second derivative: $$\frac{\partial^2}{\partial \lambda^2} \log f(x \mid \lambda) = -\frac{1}{\lambda^2}$$
This is a constant (does not depend on $x$), so the expectation is trivial: $$I_1(\lambda) = -\left(-\frac{1}{\lambda^2}\right) = \frac{1}{\lambda^2}$$
Step 4 -- For $n$ IID observations, Fisher Information is additive: $$I(\lambda) = n \cdot I_1(\lambda) = \frac{n}{\lambda^2}$$
Step 5 -- Plug in $\lambda = 3$, $n = 16$: $$I(3) = \frac{16}{9}$$
Answer: $I(\lambda) = \dfrac{n}{\lambda^2}$. At $\lambda = 3$ and $n = 16$, $I(\lambda) = \dfrac{16}{9}$.
Intuition
Fisher Information tells you the precision with which you can estimate a parameter from data. For the exponential distribution, a nice feature is that the second derivative of the log-likelihood is a constant -- it does not depend on the observed data at all. This means the Fisher Information equals the observed information for every sample, which is unusual and makes the exponential distribution particularly clean for inference.
The formula $I(\lambda) = n/\lambda^2$ has a natural interpretation: larger $\lambda$ (shorter expected waiting times) means less information per observation, because the data is more concentrated near zero and there is less "spread" to learn from. More observations always help linearly. The Cramer-Rao lower bound says no unbiased estimator can have variance below $\lambda^2/n$, which is exactly achieved by the MLE in the large-sample limit.