t-SNE: Squeezing High Dimensions Into a Picture You Can Read

The problem: a picture from a space you cannot see

Real datasets — word embeddings, gene expression profiles, hidden layer activations of a neural network — often live in hundreds or thousands of dimensions. Humans cannot look at that directly. t-Distributed Stochastic Neighbour Embedding (t-SNE), introduced by Laurens van der Maaten and Geoffrey Hinton in 2008, finds a 2D or 3D layout of points whose neighbourhood structure mirrors the original high-dimensional structure as closely as possible — points that were close in the original space end up close in the picture, and points that were far apart end up far apart, at least locally.

Step one: turn distances into probabilities

t-SNE first converts high-dimensional Euclidean distances into a probability distribution over pairs: for each point i, a Gaussian centred on i assigns a conditional probability p(j|i) that i would pick j as its neighbour, higher for nearby points and rapidly falling to near-zero for distant ones. These are symmetrised into a single joint distribution pᵢⱼ over all pairs in the input space.

high-dim similarity (Gaussian):
  p(j|i) = exp(−||xi−xj||² / 2σi²) / Σ_k≠i exp(−||xi−xk||² / 2σi²)

low-dim similarity (Student-t, 1 degree of freedom):
  q(i,j) = (1 + ||yi−yj||²)^−1 / Σ_k≠l (1 + ||yk−yl||²)^−1

The width σᵢ of each point's Gaussian is not fixed globally — it is tuned individually so that the resulting distribution has a fixed perplexity, a smooth stand-in for "effective number of neighbours." Points in dense regions get a narrower Gaussian, points in sparse regions get a wider one, so every point ends up considering a comparable effective neighbourhood regardless of local density.

Step two: the crowding problem and why the tails must be heavy

If the low-dimensional similarities also used a Gaussian, the algorithm would run into the crowding problem: high-dimensional space has room for a point to have many moderately-distant neighbours simultaneously, but 2D space runs out of room for that much fast, forcing all those moderate distances to compress and pile up near the centre of each cluster. Van der Maaten and Hinton's fix was to give the low-dimensional map a Student-t distribution with one degree of freedom (equivalent to a Cauchy distribution) instead of a Gaussian. Its heavier tails let moderately dissimilar points end up considerably far apart in the map without much probability cost, which frees up space and pushes clusters apart from one another, producing the sharply separated blobs t-SNE plots are known for.

Step three: minimise KL divergence by gradient descent

With both distributions defined, t-SNE searches for the 2D point positions y that make the low-dimensional distribution Q match the high-dimensional distribution P as closely as possible, measured by Kullback-Leibler divergence, and minimised by gradient descent:

Cost:  C = KL(P || Q) = Σᵢ Σⱼ pᵢⱼ · log(pᵢⱼ / qᵢⱼ)

Gradient (attractive term pulls similar points together,
repulsive term pushes dissimilar points apart):
  ∂C/∂yi = 4 Σⱼ (pᵢⱼ − qᵢⱼ)(yi − yj)(1 + ||yi−yj||²)⁻¹

Because KL divergence is asymmetric, it penalises placing genuinely nearby points far apart much more heavily than the reverse — so the optimisation prioritises preserving local structure at the expense of faithfully representing global distances, which is precisely the trade-off that makes t-SNE excellent at revealing clusters but unreliable for reading off exact distances between them.

Practical knobs, and how t-SNE differs from PCA

Perplexity (typically 5-50) and learning rate are the two settings that matter most, and both change the resulting picture noticeably — it is standard practice to run t-SNE at a few different perplexities before drawing conclusions. Unlike PCA, which finds a single global linear projection that maximises preserved variance and is deterministic and reversible, t-SNE is nonlinear, stochastic (different random initialisations give different layouts), and non-convex, so it can get stuck in different local minima on different runs — it is a visualisation tool for spotting structure, not a general-purpose dimensionality-reduction step to feed into further modelling.

Frequently asked questions

Try it live

Everything above runs in your browser — open t-SNE Visualiser and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.