PCA, t-SNE and UMAP: Three Very Different Ways to Flatten High-Dimensional Data

The problem: too many axes to look at

Real datasets often live in dozens, hundreds or thousands of dimensions, and human vision only works in two or three. Dimensionality reduction finds a low-dimensional coordinate system that preserves as much of the meaningful structure of the high-dimensional data as possible, so that clusters, gradients and outliers that are invisible in a table of numbers become visible in a scatter plot. The three techniques compared here — PCA, t-SNE and UMAP — solve that problem with fundamentally different mathematics and fundamentally different notions of what "preserve the structure" even means.

PCA: the best possible linear shadow

Principal Component Analysis asks a purely linear question: which direction in the data captures the most variance? Centre the data, compute its covariance matrix Σ, and diagonalise it:

Σ = (1/n) XᵀX                      (X mean-centred, rows = samples)
Σ vi = λi vi                       eigenvectors vi = principal components
                                    eigenvalues λi  = variance along vi

project onto the top k eigenvectors → the k-dimensional view that
                                       preserves the most total variance
                                       of any linear projection

Equivalently, PCA falls straight out of the singular value decomposition X = UΣVᵀ: the right singular vectors V are the principal components, and the singular values in Σ are proportional to the square roots of the eigenvalues above. Because it is a rigid linear projection — literally rotating the data and then dropping the low-variance axes — PCA is fast (a single eigendecomposition, no iteration), deterministic, and invertible in the sense that you always know exactly how much variance you threw away (the ratio of dropped to total eigenvalues). Its weakness is exactly its strength: a linear projection cannot unroll a curved manifold. Data shaped like a Swiss roll or two interleaved spirals will be crushed into an uninformative blob no matter which linear direction you pick, because the interesting structure is intrinsically non-linear.

t-SNE: preserve neighbours, not distances

t-distributed Stochastic Neighbor Embedding, introduced by van der Maaten and Hinton in 2008, gives up on preserving distances globally and instead tries to preserve who is whose neighbour. In the high-dimensional space it converts pairwise distances into conditional probabilities that point j is a neighbour of point i, using a Gaussian centred on i whose bandwidth is tuned per-point by a target perplexity (loosely, an effective neighbour count — typical values 5 to 50):

pj|i ∝ exp( −||xi − xj||² / 2σi² )      per-point σi set by target perplexity
pij  = (pj|i + pi|j) / 2n               symmetrised joint probability

in the low-dimensional map, use a heavier-tailed Student-t distribution:
qij ∝ (1 + ||yi − yj||²)⁻¹

minimise  KL(P ‖ Q) = Σ pij log(pij / qij)   by gradient descent on the yi

The heavy-tailed Student-t in the low-dimensional space (instead of another Gaussian) is the key trick, and it solves what van der Maaten and Hinton called the crowding problem: a high-dimensional space simply has room for far more equidistant neighbours than a 2D plane does, so a symmetric Gaussian embedding is forced to either crush moderate distances together or use unreasonably large distances for far-apart points. The t-distribution's fatter tails let moderately dissimilar points sit comfortably far apart in the map without needing very close points to also spread out, which is what gives t-SNE its trademark tight, well-separated clusters. The price is that only local structure is trustworthy: cluster sizes, the distances between clusters and even how many points end up in a cluster can be artifacts of the perplexity and the random initialisation, not the true geometry of the data — a classic pitfall is reading meaning into how far apart two t-SNE clusters are, which the algorithm does not try to preserve at all.

UMAP: topology instead of probability

Uniform Manifold Approximation and Projection (McInnes, Healy & Melville, 2018) starts from a different premise, rooted in topological data analysis: assume the high-dimensional data lies on (or near) a lower-dimensional manifold, and try to reconstruct a fuzzy topological representation of that manifold — for each point, a fuzzy simplicial set built from its k nearest neighbours, where edge weights fall off with distance the same way t-SNE's probabilities do, but each point's local distance scale is calibrated to its own nearest-neighbour distance rather than a global perplexity target. It then optimises a low-dimensional layout to have the most similar fuzzy topology, minimising a cross-entropy between the high- and low-dimensional fuzzy sets, using an efficient stochastic gradient descent with negative sampling similar in spirit to word2vec.

In practice UMAP tends to run noticeably faster than t-SNE on large datasets, and its theoretical grounding gives it a genuine edge at preserving more global structure — the relative positions of clusters carry somewhat more information than in t-SNE — though it still should not be read as preserving true distances. Both methods share the same core caveat: they are non-linear and stochastic, so the axes of the output plot carry no direct physical meaning (unlike PCA, where each axis is a specific linear combination of the original features with a known variance), and re-running with a different random seed or parameter can visibly rearrange the layout even though the underlying neighbour relationships are similar.

Which one, and when

PCA      fast, deterministic, linear, axes are interpretable        →  first pass,
                                                                          preprocessing,
                                                                          linear structure
t-SNE    slow(er), stochastic, excellent local clusters, no global      →  visual exploration
         distance meaning, sensitive to perplexity                         of cluster structure
UMAP     faster than t-SNE, more global structure preserved,            →  large datasets,
         theoretically grounded, still not distance-preserving             pipeline/production use

A common and effective workflow chains them: run PCA first to cut, say, 500 noisy dimensions down to 30-50 that still capture most of the variance (which also denoises the input and speeds up what follows enormously), and only then hand that reduced representation to t-SNE or UMAP for the final non-linear 2D or 3D embedding used for visualisation.

Frequently asked questions

Try it live

Everything above runs in your browser — open Dimensionality Reduction and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.