Expected Value: With vs Without Replacement
There are 10 coins in a bag: 3 pennies ($\$0.01$ each), 1 nickel ($\$0.05$), 4 dimes ($\$0.10$ each), and 2 quarters ($\$0.25$ each). You draw 3 coins one at a time.
- Compute the expected total value of your 3 draws if you sample with replacement.
- Compute the expected total value of your 3 draws if you sample without replacement.
- Which method gives a higher expected total? Does the variance differ between the two methods?
Hints
- Think about the marginal distribution of each draw. Does the position of a draw (first, second, third) affect which coin you are likely to get?
- Drawing without replacement is equivalent to choosing a random subset and then randomly ordering it. By symmetry, $P(X_i = c_j) = \frac{1}{10}$ for every draw position $i$.
- Since $E[X_i]$ is the same for all draw positions under both schemes, linearity of expectation gives the same total. The only difference is in the variance -- use the finite population correction factor $\frac{n-k}{n-1}$.
Worked Solution
How to Think About It: Before doing any math, think about what "expected value of the $i$-th draw" means. With replacement, every draw is identical -- same distribution, same expectation. Without replacement, draws are dependent, but here is the key: by symmetry, each position is equally likely to contain any particular coin. The second coin drawn without replacement is just as likely to be a quarter as the first one. So the marginal distribution of each draw is the same in both cases. The only thing that changes is the dependence structure -- and dependence affects variance, not expectation.
Quick Estimate: The average coin value is $(3 \times 0.01 + 1 \times 0.05 + 4 \times 0.10 + 2 \times 0.25)/10 = (0.03 + 0.05 + 0.40 + 0.50)/10 = 0.98/10 = \$0.098$. Drawing 3 coins, expected total $= 3 \times 0.098 = \$0.294$ regardless of replacement or not. Done -- neither method is better in expectation.
Approach: Prove that $E[X_i]$ is the same for every draw position under both schemes, then address the variance.
Formal Solution:
Label the 10 coins $c_1, c_2, \ldots, c_{10}$ with values $v_1, v_2, \ldots, v_{10}$.
With replacement: Each draw $X_i$ is an independent uniform sample from $\{c_1, \ldots, c_{10}\}$. So:
$E[X_i] = \frac{1}{10}\sum_{j=1}^{10} v_j = \frac{0.98}{10} = 0.098$
and $E[X_1 + X_2 + X_3] = 3 \times 0.098 = 0.294$.
Without replacement: Drawing 3 coins one at a time is equivalent to choosing a random permutation of 3 coins out of 10 and labeling them "first," "second," "third." By symmetry of the uniform distribution over permutations, every coin is equally likely to appear in any draw position:
$P(X_i = v_j) = \frac{1}{10} \quad \text{for all } i \in \{1,2,3\}, \; j \in \{1,\ldots,10\}$
So $E[X_i] = 0.098$ for each $i$, and the total expectation is again $0.294$.
Variance comparison: With replacement, the draws are independent, so $\text{Var}(\text{total}) = 3 \cdot \text{Var}(X)$. Without replacement, the draws are negatively correlated (if you draw a high-value coin first, there are fewer high-value coins left). This negative correlation reduces the variance:
$\text{Var}_{\text{without}}(\text{total}) = 3 \cdot \text{Var}(X) \cdot \frac{10 - 3}{10 - 1} = 3 \cdot \text{Var}(X) \cdot \frac{7}{9}$
The factor $\frac{n - k}{n - 1} = \frac{7}{9} \approx 0.778$ is the finite population correction. Since $\frac{7}{9} < 1$, sampling without replacement always has lower variance.
Answer: The expected total value is $\$0.294$ under both methods -- there is no difference in expectation. However, sampling without replacement has lower variance by a factor of $\frac{7}{9}$, because the negative correlation between draws concentrates outcomes closer to the mean.
Intuition
This problem tests one of the most useful symmetry arguments in probability: the marginal distribution of any single draw from a finite population is the same regardless of whether you sample with or without replacement. People often think the second draw without replacement is "different" because it depends on the first, but dependence is not the same as a changed marginal. Think of shuffling a deck of cards -- the probability that the 5th card is the ace of spades is exactly
In practice, this principle shows up constantly. When a trader draws a random sample from a portfolio to stress-test, the expected loss per position is the same whether positions are drawn with or without replacement. The difference is entirely in the variance and correlation structure. Without replacement gives you tighter estimates (lower variance) because of the negative correlation between draws -- once you have seen a high-loss position, it cannot appear again. This is why sampling without replacement is generally preferred for estimation tasks: same mean, less noise.