PCA and SVD: Finding the Axes Your Data Actually Lives On

The idea: rotate onto uncorrelated axes

Raw data axes are usually correlated and partly redundant — height and weight both move together, for instance. Principal Component Analysis finds a new orthogonal basis, ordered by how much variance each new axis explains, where the first component points along the single direction of maximum spread in the data, the second is perpendicular to the first and captures the next largest share of spread, and so on. This is a pure rotation of the coordinate system — the data itself never changes — that concentrates as much variance as possible into as few axes as possible.

Covariance matrix and its eigenvectors

The classic recipe: center the data by subtracting the mean of each column, then compute the covariance matrix from the centered data matrix X. Because a covariance matrix is always symmetric, the spectral theorem guarantees it has real eigenvalues and mutually orthogonal eigenvectors (see this site's eigenvectors article) — and those eigenvectors are the principal component directions, with each eigenvalue equal to the variance explained along its own component. Sorting eigenvalues from largest to smallest ranks the components by importance.

C = (1 / (n - 1)) * X-transpose * X       (X: centered n x d data matrix)
C v_i = lambda_i v_i                      (eigen-decomposition)
variance explained by component i = lambda_i / sum(lambda_j)

Why SVD instead of eigen-decomposing the covariance matrix directly

The singular value decomposition factorizes the centered data matrix directly as X = U Sigma V-transpose. The columns of V turn out to be exactly the eigenvectors of the covariance matrix, and the singular values relate to the eigenvalues by lambda_i = sigma_i^2 / (n-1). Computing the SVD of X directly is numerically more stable than first forming X-transpose X and then eigen-decomposing that, because squaring the data (as building the covariance matrix does) also squares its condition number and amplifies floating-point rounding error. That is the practical reason production implementations such as NumPy and scikit-learn compute PCA via SVD rather than via an explicit covariance-matrix eigen-decomposition.

X = U Sigma V-transpose
principal components = columns of V
scores (projected data) = X V = U Sigma

Dimensionality reduction and reconstruction error

Keeping only the top k components projects d-dimensional data down into k dimensions while retaining the maximum possible variance achievable by any k-dimensional linear projection — a guarantee known as the Eckart-Young theorem, which states that a truncated SVD is the best possible low-rank approximation in the least-squares sense. The reconstruction error from dropping the remaining components equals the sum of the discarded eigenvalues, which gives a direct, principled way to pick k: keep enough components that their cumulative variance reaches a target, commonly 95%.

k = 0; cumulative = 0
while cumulative / total_variance < 0.95:
  k += 1
  cumulative += lambda[k]

What PCA is and isn't

PCA is a linear, variance-based method — it is not the same as automatically finding the features most predictive of an outcome, and it is sensitive to the scale of each variable, since a feature measured on a larger numeric scale will dominate the variance unless the data is standardized first. Nonlinear structure, such as data lying on a curved manifold, is not captured well by a small number of straight principal axes, which is why nonlinear alternatives like kernel PCA, t-SNE or UMAP exist for visualizing that kind of data.

Frequently asked questions

Try it live

Everything above runs in your browser — open PCA & SVD Visualiser and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.