Maximizing a Quadratic Form
Let $A$ be an $n \times n$ square matrix and $x \in \mathbb{R}^n$. You want to find the maximum of the quadratic form $x^\top A x$.
- Assume $A$ is symmetric and $\|x\|_2 = 1$. What is the maximum value of $x^\top A x$, and where is it achieved?
- What if $A$ is not symmetric? Show how to reduce this case to the symmetric one.
- What happens if you drop the unit-norm constraint and optimize over all $x \in \mathbb{R}^n$?
Hints
- On the unit sphere, think of coordinates in the eigenbasis of $A$ -- the quadratic form becomes a weighted sum of squares.
- For any skew-symmetric matrix $K$, $x^\top K x = 0$ for all $x$. Use this to reduce the non-symmetric case to the symmetric one.
- Write $x = \sum_i c_i v_i$ in the orthonormal eigenbasis and apply $\sum_i c_i^2 = 1$ together with $\sum_i c_i^2 \lambda_i \leq \lambda_{\max} \sum_i c_i^2$.
Worked Solution
How to Think About It: A quadratic form $x^\top A x$ on the unit sphere is a classic eigenvalue problem. The key insight is that any symmetric matrix can be diagonalized in an orthonormal basis -- so in that basis the quadratic form becomes a weighted sum of squares of the coordinates, and the maximum is just the largest weight, i.e., the largest eigenvalue. Before writing any formulas, you should know: the answer is $\lambda_{\max}$, achieved at the corresponding eigenvector. If $A$ is not symmetric, the skew-symmetric part $\frac{A - A^\top}{2}$ contributes zero to any quadratic form (since $x^\top B x = 0$ for any skew-symmetric $B$), so only the symmetric part matters.
Quick Sanity Check: Take $A = \text{diag}(3, 1)$ and $x = (\cos\theta, \sin\theta)^\top$. Then $x^\top A x = 3\cos^2\theta + \sin^2\theta$, maximized at $\theta = 0$ with value $3 = \lambda_{\max}$. That is exactly what the theorem says.
Formal Solution:
Part 1 -- Symmetric case. Since $A$ is symmetric, by the spectral theorem it has an orthonormal eigenbasis $\{v_1, \ldots, v_n\}$ with $Av_i = \lambda_i v_i$ and eigenvalues $\lambda_1 \geq \lambda_2 \geq \cdots \geq \lambda_n$. Write $x = \sum_{i=1}^n c_i v_i$. The unit-norm constraint gives $\sum_i c_i^2 = 1$. Then:
$$x^\top A x = \left(\sum_i c_i v_i\right)^\top A \left(\sum_j c_j v_j\right) = \sum_i c_i^2 \lambda_i \leq \lambda_1 \sum_i c_i^2 = \lambda_1$$
Equality holds when $c_1 = 1$ and $c_i = 0$ for $i > 1$, i.e., $x = v_1$. So $\max_{\|x\|=1} x^\top A x = \lambda_{\max}$, achieved at the eigenvector corresponding to the largest eigenvalue. This ratio $\frac{x^\top A x}{x^\top x}$ is called the Rayleigh quotient.
Part 2 -- Non-symmetric case. Decompose $A = S + K$ where $S = \frac{A + A^\top}{2}$ is symmetric and $K = \frac{A - A^\top}{2}$ is skew-symmetric. For any $x$:
$$x^\top K x = (x^\top K x)^\top = x^\top K^\top x = -x^\top K x \implies x^\top K x = 0$$
So $x^\top A x = x^\top S x$ for all $x$. The quadratic form depends only on the symmetric part of $A$. Apply Part 1 to $S$.
Part 3 -- Unconstrained case. If $A$ (or its symmetric part) has any strictly positive eigenvalue $\lambda > 0$ with eigenvector $v$, then $x = tv$ gives $x^\top A x = t^2 \lambda \to \infty$ as $t \to \infty$. No finite maximum exists. A finite maximum over all of $\mathbb{R}^n$ only exists if $A$ is negative semi-definite (all eigenvalues $\leq 0$), in which case the maximum is $0$, achieved at $x = 0$.
Answer: - Symmetric $A$, $\|x\| = 1$: $\max x^\top A x = \lambda_{\max}(A)$, achieved at the corresponding eigenvector. - Non-symmetric $A$: replace $A$ with $\frac{A+A^\top}{2}$ and apply the above. - Unconstrained: no finite maximum unless $A$ is negative semi-definite.
Intuition
The Rayleigh quotient is one of those results that appears constantly in quant work -- it shows up in PCA (the first principal component is the eigenvector of the covariance matrix with the largest eigenvalue), in portfolio optimization (the max-Sharpe direction under certain setups), and in numerical linear algebra. The punchline is simple: on the unit sphere, a quadratic form is just a weighted average of the eigenvalues, with weights given by how much of $x$ points along each eigendirection. The maximum is achieved by pointing entirely along the direction of the largest eigenvalue.
The non-symmetric reduction is worth remembering as a trick: any matrix $A$ can be split into symmetric and skew-symmetric parts, and quadratic forms are completely blind to the skew-symmetric part. This means when you see $x^\top A x$, you can always symmetrize $A$ first without changing the problem -- a useful simplification in optimization and stability analysis.