Hierarchical Clustering and Dendrograms: Building a Tree of Customer Segments

A clustering method that doesn't ask for k upfront

K-means clustering forces an early commitment: you have to decide the number of groups before you've seen how the data actually clusters. Hierarchical clustering sidesteps that by building a complete nested structure of groupings, from every point in its own cluster up to all points in one cluster, and letting you decide afterwards where to cut. That structure is called a dendrogram, and it's the tool's defining output: a tree diagram where the leaves are individual data points and each internal branch marks a merge, drawn at a height proportional to how dissimilar the two things being merged were.

The variant most commonly used in practice is agglomerative, meaning it works bottom-up. It starts by treating every single data point as its own cluster, then repeatedly finds the two closest clusters and merges them into one, recording the distance at which the merge happened. It keeps doing this — cluster count shrinking by one at every step — until everything has merged into a single cluster containing the whole dataset. Applied to something like customer purchase and income data, the leaves of the resulting tree are individual customers, and as you move up the tree, you can watch small groups of similar customers merge into larger and larger segments, right up to the trivial single cluster that is everyone.

Linkage: how do you measure distance between two clusters?

Merging the two closest clusters is straightforward when a cluster is a single point, but once clusters contain multiple points, "distance between two clusters" needs a definition, and that definition is called the linkage method. It's the single choice that most changes the shape of the resulting tree.

• Single linkage: distance between two clusters is the distance between their single closest pair of points. This is sensitive to chains — a thin bridge of intermediate points can link two otherwise distinct groups into one long, stringy cluster, an effect called chaining. It's also the most sensitive of the common linkage methods to outliers, because one point sitting between two clusters is enough to trigger an early merge.
• Complete linkage: distance is the distance between the two clusters' farthest pair of points. This produces tighter, more compact, more evenly sized clusters than single linkage, because a merge only happens once every point in one cluster is reasonably close to every point in the other.
• Average linkage: distance is the average distance across all pairs of points between the two clusters. A middle ground between single and complete linkage.
• Ward's method: merges the pair of clusters that produces the smallest increase in total within-cluster variance. In practice this tends to produce the most balanced, interpretable clusters for general-purpose segmentation, and is often the default recommendation when there's no strong reason to pick something else.

Running the same dataset through single linkage and then Ward's method can produce visibly different trees. Single linkage is especially prone to producing one dominant cluster that has absorbed almost everything through a chain of near-neighbours, with a scattering of small clusters merging in very late — a warning sign that's obvious on the dendrogram (a few merges happening at heights far higher than the rest) but easy to miss if you only look at the final flat cluster labels.

Reading a dendrogram

A dendrogram encodes two things simultaneously: which items merged with which, and how dissimilar they were when they merged. The height of a branch is the key piece of information — a low merge height means the two clusters being joined were very similar, a tall merge height means they were quite different and the algorithm was, in a sense, forced to combine them because nothing better remained.

To choose a number of clusters from a dendrogram, look for the tallest vertical gaps between consecutive merge heights, then draw a horizontal cut through one of those gaps. The number of vertical lines the cut crosses is the number of clusters you get at that cut. A long gap before a merge is evidence that the clusters on either side were genuinely distinct, since it took a big jump in dissimilarity to finally combine them; a series of short gaps suggests the tree doesn't have a strongly preferred number of clusters and the split is a matter of choice rather than a clear structural fact.

With more than a few dozen data points, a full dendrogram becomes an unreadable wall of lines. The standard fix is truncation: only the last p merges are drawn, with each remaining leaf representing a collapsed sub-tree rather than an individual point (in SciPy this is the truncate_mode='lastp' option). This keeps the diagram legible while preserving exactly the part of the tree that matters for choosing a cut — the highest, most consequential merges — at the cost of no longer being able to trace an individual data point down to a specific leaf.

From tree to labels: cutting with fcluster

A dendrogram by itself is a visualisation, not a set of cluster assignments. To turn it into usable labels — the kind you'd attach back onto a customer table — you cut the tree, and SciPy's fcluster function does this in one of two ways: by specifying the number of clusters you want directly, or by specifying a distance threshold and letting the tree produce however many clusters exist below that height. Cutting a tree built from customer data with single linkage at two clusters versus three clusters can produce very different results depending on where the tallest gaps happen to sit; it's worth generating a couple of candidate cuts and comparing them rather than committing to the first one.

Once a cut produces flat cluster labels, they can be evaluated the same way k-means clusters are evaluated — most directly with the silhouette score, which measures how well each point fits inside its assigned cluster relative to the nearest neighbouring cluster. This gives an objective way to compare hierarchical clustering against k-means run on the same standardised data: if hierarchical clustering with Ward linkage produces a noticeably higher silhouette score at three clusters than k-means does at three clusters, that's genuine evidence the tree-based approach found more natural boundaries in this particular dataset, not just a different but equally valid split.

When a tree beats a fixed number of centroids

Hierarchical clustering carries a real computational cost — building the full tree from n points has roughly quadratic or worse time and memory complexity depending on the implementation, which makes it impractical on datasets with hundreds of thousands of rows in a way that k-means, with its linear-per-iteration cost, is not. So it isn't the default choice for large-scale, repeatedly-refreshed segmentation pipelines. Where it earns its cost is in exploratory analysis: because the dendrogram shows every possible cut simultaneously, an analyst doesn't have to commit to a cluster count before understanding whether two clusters, three clusters, or six clusters is the more natural read of the data. It's also useful precisely because it doesn't share k-means's assumption of round, similarly sized clusters, and different linkage choices give you a way to test how sensitive a segmentation is to that assumption — if Ward, average and single linkage all produce broadly similar groupings, that's a much stronger signal that the segments are real than if they only appear under one specific linkage method.