MLE for a Bernoulli Parameter
You observe a sequence of independent coin flips: $\{1, 0, 0, 1, 0\}$, where 1 = heads and 0 = tails. The coin has unknown bias $\theta \in (0, 1)$.
Find the maximum likelihood estimate (MLE) of $\theta$. Derive it from first principles by writing out the likelihood function and maximizing it.
Hints
- The MLE maximizes $P(\text{data} \mid \theta)$ over $\theta$. For a coin with unknown bias, write out the probability of the full sequence.
- Take the log of the likelihood -- it turns a product into a sum, which is much easier to differentiate. The MLE is the same either way since $\log$ is monotone.
- Differentiate the log-likelihood $\ell(\theta) = (\sum x_i) \log \theta + (n - \sum x_i) \log(1-\theta)$ and set it to zero. The result is $\hat{\theta} = (\sum x_i)/n$.
Worked Solution
How to Think About It: The MLE is the value of $\theta$ that makes the observed data most probable. For a sequence of coin flips, that's just the fraction of heads -- but the derivation via calculus is worth knowing cold because it generalizes directly to logistic regression, Poisson models, and most of exponential family statistics.
Quick Estimate: We have 5 flips, 2 heads. Common sense says $\hat{\theta} = 2/5 = 0.4$. The MLE should confirm this.
Approach: Write the likelihood, take the log, differentiate, set to zero.
Formal Solution:
The PMF of a single Bernoulli observation is: $$p(x \mid \theta) = \theta^x (1 - \theta)^{1-x}, \quad x \in \{0, 1\}$$
For $n = 5$ IID observations, the likelihood function is: $$L(\theta) = \prod_{i=1}^{5} \theta^{x_i}(1-\theta)^{1-x_i} = \theta^{\sum x_i}(1-\theta)^{n - \sum x_i}$$
With the data $\{1, 0, 0, 1, 0\}$: $\sum x_i = 2$, $n = 5$.
$$L(\theta) = \theta^2 (1-\theta)^3$$
The log-likelihood is easier to differentiate: $$\ell(\theta) = 2 \log \theta + 3 \log(1 - \theta)$$
Set the derivative to zero: $$\frac{d\ell}{d\theta} = \frac{2}{\theta} - \frac{3}{1-\theta} = 0$$
$$2(1-\theta) = 3\theta \implies 2 = 5\theta \implies \theta = \frac{2}{5}$$
This is a maximum (the second derivative is negative for $\theta \in (0,1)$, or simply note $\ell$ is strictly concave on $(0,1)$).
In general, for $n$ flips with $\sum x_i = H$ heads: $$\hat{\theta}_{\text{MLE}} = \frac{H}{n} = \frac{\sum x_i}{n}$$
Answer: $\hat{\theta}_{\text{MLE}} = 2/5 = 0.4$.
Intuition
The Bernoulli MLE being the sample mean is almost obvious, but the derivation pattern -- write likelihood, take log, differentiate, solve -- is the backbone of statistical estimation. It generalizes: for any exponential family distribution, the MLE equates the sufficient statistic to its expectation under the model, which is exactly what happens here ($\hat{\theta} = \bar{x}$ is the moment condition $E[X] = \theta$).
In practice, knowing the MLE is the empirical frequency matters in market making. If you observe $H$ up-ticks and $n - H$ down-ticks, your best point estimate of the drift probability is $H/n$ -- no prior, no shrinkage, just the data. The Bayesian version (with a Beta prior) adds regularization toward your prior mean, which is useful when $n$ is small. But when $n$ is large, MLE and Bayes converge.