Numerically Stable Log-Sum-Exp and Softmax

Naive computation of $\log \sum_i e^{x_i}$ and $\text{softmax}(x)_j = e^{x_j} / \sum_i e^{x_i}$ blows up or underflows when entries of $x$ are large in magnitude. Your job is to implement numerically stable versions of both, and then build cross-entropy loss and its gradient on top. 1. Implement `log_sum_exp(x)` that computes $\log \sum_{i=1}^{n} e^{x_i}$ without overflow or underflow. 2. Implement `softmax(x)` that returns the vector $s$ with $s_j = e^{x_j} / \sum_i e^{x_i}$, also numerically stable. 3. Implement `cross_entropy_loss(logits, target)` where `target` is the index of the correct class. This should return $-\log s_{\text{target}}$ computed in a stable way. 4. Implement `cross_entropy_gradient(logits, target)` that returns $\partial L / \partial x$. 5. Write unit tests covering: - Normal inputs (small values) - Extreme inputs (entries on the order of $\pm 10^3$) - Inputs containing $-\infty$ - Inputs where all entries are equal - A single-element input **Constraints:** - Input is a 1-D array of floats with

Open the full interactive solver, hints, and worked solution →