PCA and Dimensionality Reduction: Finding the Directions That Matter
How Principal Component Analysis rotates a dataset onto new axes ranked by how much variance they capture, letting high-dimensional data be compressed, visualized and denoised.
The problem of too many dimensions
Real-world datasets often have far more columns than a person can reason about at once. A customer dataset might track dozens of behavioural and demographic variables; a genomics dataset might record the expression level of thousands of genes per sample; an image is, at its most literal, a dataset with one dimension per pixel. High dimensionality causes practical problems beyond just being hard to look at: many of these variables are correlated with each other (redundant), some distance-based algorithms behave poorly as dimensionality grows (a phenomenon often called the curse of dimensionality, where data points all start to look roughly equidistant from each other), and models trained on very high-dimensional data with limited examples are prone to overfitting.
Dimensionality reduction is the family of techniques for compressing this kind of data down to fewer variables while preserving as much of the meaningful structure as possible. Principal Component Analysis, or PCA, developed in essentially its modern form by Harold Hotelling in 1933 (building on earlier work by Karl Pearson in 1901), is the oldest and still most widely used of these techniques, prized for being simple, fast, and grounded in linear algebra that produces exact, interpretable results rather than an approximate or stochastic one.
Variance as a proxy for information
PCA is built on a specific, useful assumption: the directions in the data along which values vary the most are the directions carrying the most information. Imagine a dataset with two correlated features, say a person's height and weight, plotted as a scatter of points. The cloud of points is not spread evenly in every direction; it tends to stretch out diagonally, because taller people tend to be heavier. There is far more spread along that diagonal direction than there is perpendicular to it.
PCA formalises this intuition mathematically. It looks for the single direction, called the first principal component, along which the data's variance (spread) is maximised. It then looks for a second direction, perpendicular (orthogonal) to the first, along which the remaining variance is maximised, and so on, each new principal component orthogonal to all the ones before it. For a dataset with n original features, PCA can find up to n such principal components, but the key practical payoff is that in most real datasets, a small number of the earliest components capture the great majority of the total variance, because real features tend to be correlated with each other rather than independent. Keeping only the first two or three components and discarding the rest often preserves 80, 90 or more percent of the dataset's variance while reducing dozens or hundreds of original columns down to a handful of new ones.
How PCA actually computes the components
Under the hood, PCA is a piece of linear algebra applied to the dataset's covariance matrix, which records how every pair of original features varies together. Concretely, the standard procedure centres the data (subtracting each feature's mean so the whole dataset is centred at the origin), computes the covariance matrix of the centred features, and then finds that matrix's eigenvectors and eigenvalues. Each eigenvector points in one of the directions of maximal variance described above, and its corresponding eigenvalue tells you exactly how much variance lies along that direction. Sorting the eigenvectors by eigenvalue, largest first, gives you the principal components in order of importance: the first eigenvector (with the largest eigenvalue) is the first principal component, and so on.
To actually transform the data into this new coordinate system, each original data point is projected onto the chosen eigenvectors — a simple matrix multiplication that produces a new set of coordinates, one per retained component, for every original point. This transformed data is what gets used downstream, whether for visualization, as compressed input features to another model, or for storage. Because principal components are just linear combinations of the original features (a weighted sum of, say, height, weight and age), the transformation is fully reversible up to the information discarded by dropping later components: you can reconstruct an approximation of the original data from the retained components, and the reconstruction error is directly related to how much variance was left behind.
Choosing how many components to keep, and what gets lost
The standard tool for deciding how many components to retain is the scree plot, which shows the proportion of total variance explained by each successive component, typically as a bar chart with a cumulative line overlaid. In practice, analysts look either for an "elbow" where the marginal variance explained by additional components drops off sharply, or simply pick the smallest number of components whose cumulative explained variance crosses a target threshold, commonly 90 or 95 percent. Both approaches trade off compression against fidelity: fewer components mean more compression and easier visualization, but also more information discarded.
What is discarded matters. PCA's components are ranked purely by variance, with no regard for whether that variance happens to be useful for a downstream task like classification. It is entirely possible, though not typical, for an important but low-variance signal to be thrown away while a high-variance but task-irrelevant direction is kept, which is why PCA is described as an unsupervised technique: it never looks at labels, only at the spread of the features themselves. It is also worth being explicit that the resulting components are new synthetic axes, not any of the original features; the first principal component of a customer dataset is not "income" or "age" but some specific weighted mixture of many original variables, which is why PCA components are often described as less interpretable than the raw features, even though they are mathematically precise and derived deterministically rather than learned by an opaque neural network.
Where PCA gets used in practice
The most immediate and visual use of PCA is exploratory visualization: projecting a dataset with dozens of features down to its first two or three principal components lets you plot it on an ordinary chart and visually inspect whether natural clusters or structure exist, something impossible to eyeball directly in the original high-dimensional space. This is exactly the kind of transformation that benefits from an interactive 3D view, since rotating a three-component PCA projection lets you see clustering structure from different angles that a single static 2D scatterplot would hide or distort.
PCA is also used as a preprocessing step before other algorithms: reducing a dataset's dimensionality before feeding it to a distance-based method like k-nearest neighbours or k-means clustering can both speed up computation and mitigate the curse of dimensionality; reducing dimensionality before training a model on a dataset with far more features than examples can reduce overfitting; and because later principal components tend to capture noise rather than signal (since noise is typically low-variance and uncorrelated), dropping them can act as a simple form of denoising, which is one reason PCA has historically been used in image compression and in preprocessing for facial recognition systems, where the technique applied to face images is sometimes specifically called eigenfaces. It is worth noting PCA assumes linear relationships between features; when the meaningful structure in data is genuinely non-linear, techniques like t-SNE or UMAP, though more computationally expensive and less interpretable, often reveal structure that PCA's straight-line projections miss.
Frequently Asked Questions
Do I need to scale my features before running PCA?
Almost always, yes. PCA finds directions of maximum variance, and variance is directly affected by the units and scale of each feature. A feature measured in thousands (like annual income) will dominate the variance calculation and swamp a feature measured in small decimals (like a ratio between 0 and 1), even if the smaller-scale feature is just as informative. Standardizing each feature to zero mean and unit variance before running PCA ensures every original feature gets a fair chance to contribute to the principal components based on its actual informativeness rather than its arbitrary measurement scale.
Is PCA the same thing as feature selection?
No, and this distinction matters. Feature selection chooses a subset of the original, still-interpretable features and discards the rest (for example, keeping height and weight but dropping shoe size). PCA instead creates entirely new features, each a mathematical combination of all the original ones, and none of the original features survive individually in the output. Feature selection preserves interpretability; PCA sacrifices interpretability for often more efficient compression, since a single principal component can capture variance that would otherwise require several original features.
How much variance should the retained components explain before I stop adding more?
There is no universal answer, but a common rule of thumb targets somewhere between 90 and 99 percent of cumulative explained variance, depending on how much compression versus fidelity the downstream task needs. Exploratory visualization often deliberately keeps only two or three components regardless of how little variance they explain, purely because that is what can be plotted, while preprocessing before a predictive model might retain many more components to avoid discarding potentially useful signal.
Can PCA be used on categorical data?
Not directly. PCA operates on the covariance structure of numerical variables, so categorical variables need to be converted to a numerical form first, commonly through one-hot encoding. However, PCA on one-hot encoded categorical data often behaves poorly because the resulting variance structure does not reflect meaningful similarity the way it does for continuous numerical features. For datasets that are primarily categorical, related techniques like Multiple Correspondence Analysis (MCA) are usually a better-suited alternative to plain PCA.