PCA: Rotating Data Onto Its Principal Components

Rotating the cloud until the spread lines up with the axes

Principal Component Analysis looks for the rotation of a dataset that makes its structure as simple as possible to describe. Take a scatter of points in two or more correlated dimensions — height and weight, or the pixel intensities of a face — and there is almost always a direction along which the points are most spread out. PCA finds that direction first, then the next-most-spread direction perpendicular to it, and so on, producing a new set of axes called principal components that are uncorrelated with each other by construction.

The key quantity is the covariance matrix of the data. For centred data (mean subtracted from every dimension), the covariance matrix C = (1/n) XᵀX summarises how every pair of dimensions varies together. Its diagonal holds the variance of each original dimension; its off-diagonal entries hold the covariances that make the data cloud tilted rather than axis-aligned.

Eigenvectors are the axes, eigenvalues are the variance

PCA is, mathematically, an eigendecomposition of the covariance matrix. Each eigenvector of C points along one principal component, and its paired eigenvalue is exactly the variance of the data projected onto that direction.

C v_i = λ_i v_i

v_i  — i-th eigenvector (principal component direction, unit length)
λ_i  — i-th eigenvalue  (variance explained along that direction)

Sorting the eigenvalues from largest to smallest orders the components from most to least explanatory. The first principal component points along the direction of maximum variance; the second is the direction of maximum remaining variance subject to being perpendicular to the first, and so on, so that every component after the first mops up whatever variance its predecessors could not capture.

Dropping dimensions without losing much information

Because the eigenvalues tell you exactly how much variance each axis carries, PCA gives a principled way to compress data: keep the top k components and discard the rest, and you keep the fraction (λ₁ + ... + λ_k) / (λ₁ + ... + λ_d) of the total variance. A dataset of correlated features often has most of its variance concentrated in the first two or three components — that's why PCA plots of high-dimensional data (gene expression, image patches, survey responses) frequently show a meaningful 2D or 3D picture even though the original data had dozens of dimensions.

In practice this is computed either by eigendecomposition of the covariance matrix directly, or — more numerically stable for real data — by Singular Value Decomposition (SVD) of the centred data matrix X = UΣVᵀ, where the columns of V are the principal components and the singular values in Σ are the square roots of the eigenvalues. SVD avoids ever forming XᵀX explicitly, which matters because that matrix multiplication can amplify floating-point error.

What PCA is not

PCA only ever finds linear combinations of the original features, so it cannot unroll a curved manifold — a Swiss-roll shaped cloud stays tangled no matter how you rotate it, and nonlinear methods like t-SNE or UMAP exist precisely to handle that case. It is also sensitive to scale: a feature measured in kilometres will dominate a feature measured in millimetres purely because of units, unless you standardise each dimension to unit variance first. And PCA maximises variance, not necessarily interpretability or class separation — a direction with huge variance can still be useless for telling two categories apart, which is why supervised alternatives like Linear Discriminant Analysis exist for classification tasks.

Frequently asked questions

Try it live

Everything above runs in your browser — open PCA and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.