Mutual Information: What Correlation Misses

How much does knowing X tell you about Y?

Mutual information I(X;Y) answers a precise version of a question people ask informally all the time: how much does observing one random variable actually reduce your uncertainty about a different one? It is built directly on top of Shannon entropy, and it has a beautifully symmetric definition — you can compute it three completely different ways and always get the same number, which is itself a small proof that the concept is capturing something real rather than an artefact of how you set up the calculation.

I(X;Y) = H(X) - H(X|Y)      // uncertainty in X, minus what's left once you know Y
       = H(Y) - H(Y|X)      // symmetric — same value, computed the other direction
       = H(X) + H(Y) - H(X,Y)   // via the joint entropy of the pair together

// all three formulas give the SAME number: I(X;Y) = I(Y;X), always

The entropy Venn diagram

The cleanest intuition is a Venn diagram of two overlapping circles, one for H(X) and one for H(Y). The total area covered by the union is the joint entropy H(X,Y) — the uncertainty of the pair taken together. The part of the X circle not covered by the Y circle is the conditional entropy H(X|Y) — what remains unknown about X even after Y is fully known. And the overlap between the two circles is exactly the mutual information I(X;Y) — the uncertainty about X that observing Y eliminates, which by the symmetric definition above is identically the uncertainty about Y that observing X eliminates.

|-------- H(X) --------|
|                       |
|   H(X|Y)  |  I(X;Y)  |  H(Y|X)   |
|           |          |           |
                       |-------- H(Y) --------|

H(X,Y) = H(X|Y) + I(X;Y) + H(Y|X)     (the whole union, no double-counting)

Two extremes, worked by hand

If X and Y are completely independent — knowing one tells you nothing about the other — then by definition p(x,y) = p(x)p(y) for every pair, which forces H(X,Y) = H(X) + H(Y), and plugging into the formula above gives I(X;Y) = H(X) + H(Y) - H(X,Y) = 0. Mutual information of exactly zero is, in fact, one of the standard formal definitions of statistical independence.

If Y is a deterministic, invertible function of X — say Y is always exactly X, or always exactly not-X for a binary variable — then knowing X removes all uncertainty about Y, so H(Y|X) = 0, and the mutual information collapses to its maximum possible value for the pair: I(X;Y) = H(Y) - H(Y|X) = H(Y). For two variables with equal entropy H(X) = H(Y) = 1 bit (like two perfectly correlated fair coins), that means I(X;Y) = 1 full bit — every bit of one variable's uncertainty is explained by the other.

Why mutual information catches what correlation misses

Pearson correlation, the everyday go-to statistic, only detects linear relationships — it can be exactly zero between two variables that are perfectly, deterministically related, just nonlinearly. The textbook example is Y = X^2 for X drawn symmetrically around zero: correlation computes to 0 (the linear component genuinely cancels out over a symmetric range), yet Y is completely determined by X, with zero actual uncertainty left once X is known. Mutual information does not have this blind spot, because it is built from the full joint probability distribution p(x,y), not from a linear projection of it — I(X;Y) for that same Y = X^2 relationship comes out strongly positive, correctly reporting that Y is fully predictable from X, just not linearly.

KL divergence: the same machinery, seen as a distance

There is a second, equivalent way to define mutual information that reveals what it really is: the Kullback-Leibler divergence between the actual joint distribution p(x,y) and the hypothetical joint distribution you would get if X and Y really were independent, p(x)p(y):

I(X;Y) = D_KL( p(x,y)  ||  p(x)*p(y) )
       = sum over x,y of  p(x,y) * log2( p(x,y) / (p(x)*p(y)) )

// D_KL(P || Q) in general measures how different distribution P is
// from a reference distribution Q, in bits — always >= 0, and exactly
// 0 only when P and Q are identical

// so mutual information is literally: "how far is the real joint
// distribution from the world where X and Y don't affect each other at all" 

Because KL divergence is always non-negative and zero only when the two distributions are identical, this immediately re-proves that mutual information can never be negative — I(X;Y) >= 0 always, with equality exactly at independence — a fact that is not obvious from the entropy-subtraction definition alone but falls straight out of the KL-divergence view.

Where it actually gets used

Mutual information shows up wherever “how related are these two things, without assuming a linear model” is the real question. Feature selection in machine learning ranks candidate input features by their mutual information with the target label, catching non-linear predictive relationships that a correlation-based filter would miss entirely. Decision tree algorithms (ID3, C4.5) choose which feature to split on at each node by picking the one with the highest information gain — which is exactly the mutual information between that feature and the class label. In neuroscience and genomics, mutual information between a stimulus and a neural spike train, or between two genes' expression levels, is a standard tool precisely because biological relationships are routinely nonlinear and a linear correlation coefficient would systematically understate how coupled two signals really are.

Frequently asked questions

Try it live

Everything above runs in your browser — open Mutual Information and change the parameters while it is running. Nothing is installed, nothing is uploaded, the whole model lives in one tab.