Singular Value Decomposition: Rotate, Stretch, Rotate

Every matrix is rotate, stretch, rotate

Eigendecomposition only cleanly applies to square matrices with enough independent eigenvectors, and even then the eigenvector basis usually isn't orthogonal. Singular Value Decomposition fixes both problems and works for any matrix, square or not: every matrix A factors as

A = U Σ Vᵀ

where U and V are orthogonal matrices (pure rotations/reflections — they preserve length and angles) and Σ is diagonal with non-negative entries sorted in decreasing order, the singular values. Geometrically this says every linear map, no matter how complicated, decomposes into exactly three simple steps: rotate (Vᵀ), stretch along the coordinate axes by the singular values (Σ), rotate again (U). Apply this to the unit circle: Vᵀ spins it in place (a circle doesn't care about rotation), Σ stretches it into an axis-aligned ellipse with semi-axes equal to the singular values, and U rotates that ellipse to its final orientation.

Singular values versus eigenvalues

For a general matrix, singular values and eigenvalues are different numbers answering different questions — eigenvalues describe scaling along special invariant directions (which may not exist as real numbers, or may not be orthogonal), while singular values describe the actual maximum and minimum stretching factor of the map, always real, always non-negative, always exist for any matrix. The connection: the singular values of A are the square roots of the eigenvalues of AᵀA (which is always square, symmetric and positive semi-definite, so its eigenvalues are guaranteed real and non-negative). For a symmetric positive-definite matrix specifically, singular values and eigenvalues coincide exactly and U = V, which is why SVD and eigendecomposition only look different for general matrices, not for that special case.

Reading off the largest singular value

The largest singular value σ1 is the operator norm of the matrix: the maximum factor by which A can stretch any unit vector, achieved exactly along the direction of the first right-singular vector (a column of V) and landing along the first left-singular vector (a column of U). The smallest singular value tells you how close the matrix is to being singular (non-invertible) — a very small smallest singular value relative to the largest means the matrix is nearly rank-deficient, numerically fragile to invert, and the ratio σmax/σmin is exactly the matrix's condition number, a standard measure of how much small input errors get amplified when solving a linear system with that matrix.

for A = [[a,b],[c,d]]:
  compute AᵀA (symmetric 2×2)
  eigenvalues of AᵀA = σ1², σ2²
  σ1 ≥ σ2 ≥ 0 are the singular values
  V = eigenvectors of AᵀA,  U = A·V / σ  (normalised)

Why SVD is the workhorse behind low-rank approximation

Because Σ's singular values are sorted, truncating the decomposition — keeping only the top k singular values and their corresponding rows/columns of U and V, zeroing the rest — gives the best possible rank-k approximation of A in a precise, provable sense (the Eckart-Young theorem: no other rank-k matrix is closer to A in the standard matrix norm). This single fact underlies image compression (keep the largest singular values of an image treated as a matrix, discard the rest, and much of the visual information survives), recommender systems (approximate a huge sparse user-item matrix with a low-rank SVD to infer missing preferences), noise reduction (small singular values often correspond to noise, not signal), and the numerical core of PCA (the principal components of a dataset are exactly the right-singular vectors of the mean-centred data matrix).

Why real numerical software prefers SVD over eigendecomposition

SVD is numerically more stable to compute than eigendecomposition for general matrices, always exists (even for singular or non-square matrices, where eigendecomposition may not even be defined), and directly exposes rank, null space and conditioning — which is why library routines for solving least-squares problems, computing matrix rank, or inverting ill-conditioned systems reach for SVD by default rather than an eigenvalue routine, even when the matrix happens to be square.

Frequently asked questions

Try it live

Everything above runs in your browser — open SVD Visualizer and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.