Home▸Information Theory & Coding▸Mutual Information: Shared Information Between Variables

🔀 Mutual Information: Shared Information Between Variables

Edit a joint probability table live and watch H(X), H(Y), joint entropy H(X,Y) and mutual information I(X;Y) update in real time, visualized as an entropy Venn diagram of overlapping circles.

Information Theory & Coding2DAdvanced60 FPS

About Mutual Information

Mutual information I(X;Y) is the exact, information-theoretic answer to the question "how much does knowing Y tell me about X?" It is defined as I(X;Y) = H(X) + H(Y) − H(X,Y), where H(X) and H(Y) are the Shannon entropies of the two marginal distributions and H(X,Y) is the entropy of their joint distribution. Equivalently, I(X;Y) = Σ p(x,y)·log₂[p(x,y)/(p(x)p(y))], which is the Kullback-Leibler divergence between the true joint distribution and the product-of-marginals distribution that would hold if X and Y were independent. Because that divergence is zero exactly when the joint equals the product of marginals, mutual information is provably zero if and only if X and Y are statistically independent — and it grows as the variables become more dependent, capped at min(H(X),H(Y)) when one variable perfectly determines the other.

In this simulation you edit a real joint probability table P(X,Y) by dragging its cells; the marginals P(X) and P(Y) are recomputed live by summing rows and columns, and H(X), H(Y), the joint entropy H(X,Y), and I(X;Y) are recalculated from the definitions on every change. The two overlapping circles are a standard information-theory teaching device: circle areas represent H(X) and H(Y), and the overlap area represents I(X;Y) — a direct visual reading of the identity I(X;Y) = H(X) + H(Y) − H(X,Y). Mutual information underlies feature selection in machine learning (features are ranked by how much mutual information they carry about the target label), and neuroscience uses it to quantify how much a neuron's firing pattern reveals about a stimulus.

Frequently Asked Questions

What exactly does mutual information measure?

Mutual information I(X;Y) measures, in bits, how much the uncertainty about X shrinks once you learn the value of Y (and symmetrically, how much uncertainty about Y shrinks once you learn X). It is a genuine, well-defined quantity derived directly from entropy, not an ad-hoc correlation score, and it captures any kind of statistical dependence — linear or not.

Why is I(X;Y) exactly zero when X and Y are independent?

If X and Y are independent, then by definition p(x,y) = p(x)p(y) for every pair, so the joint entropy satisfies H(X,Y) = H(X) + H(Y). Substituting into I(X;Y) = H(X) + H(Y) − H(X,Y) gives exactly zero. Equivalently, the KL-divergence form Σ p(x,y)·log₂[p(x,y)/(p(x)p(y))] has every log term equal to log₂(1) = 0, so the sum is zero — this is a provable fact from the definition, not an approximation.

What does the overlapping-circles Venn diagram actually represent?

It is a standard, correct pedagogical visualization of the entropy decomposition. Circle A has area proportional to H(X) and circle B has area proportional to H(Y); their union has area proportional to the joint entropy H(X,Y), and their intersection has area proportional to the mutual information I(X;Y). This matches the algebra exactly: area(A) + area(B) − area(A∩B) = area(A∪B) mirrors H(X) + H(Y) − I(X;Y) = H(X,Y).

Can mutual information ever be negative?

No. Mutual information is always ≥ 0 for any joint distribution, a direct consequence of Jensen's inequality applied to the KL-divergence definition (KL-divergence between any two probability distributions is never negative). It equals zero only for independent variables and is strictly positive whenever any statistical dependence exists.

How is mutual information used in machine-learning feature selection?

Filter-based feature selection methods compute I(feature; target label) for every candidate feature and rank features by that score, keeping the ones that carry the most information about the label. Unlike a linear correlation coefficient, mutual information also detects non-linear and non-monotonic dependencies, which makes it a popular model-agnostic screening tool before training.

How does neuroscience use mutual information?

Neural coding studies compute the mutual information between a sensory stimulus and a neuron's spike train (or population response) to quantify, in bits, how much the response actually reveals about the stimulus. This gives an upper bound on how well an ideal decoder could reconstruct the stimulus from the neural activity, independent of any assumed decoding algorithm.

What is normalized mutual information and why use it?

Raw mutual information is measured in bits and its maximum value depends on the entropies of the two variables involved, which makes scores hard to compare across different variable pairs. Normalized mutual information divides I(X;Y) by a reference such as min(H(X),H(Y)), producing a dependency measure that always lies in [0,1]: 0 for independence and 1 when one variable fully determines the other.

How does mutual information relate to conditional entropy and KL divergence?

Mutual information has three equivalent forms: I(X;Y) = H(X) + H(Y) − H(X,Y) = H(X) − H(X|Y) = D_KL(P(X,Y) ‖ P(X)P(Y)). The middle form says I(X;Y) is exactly how much the conditional entropy H(X|Y) — the remaining uncertainty about X after observing Y — falls short of the unconditional entropy H(X). The KL-divergence form frames mutual information as literally measuring how far the real joint distribution is from the independence assumption.

⚙ Under the hood

Edit a joint probability table live and watch H(X), H(Y), joint entropy and mutual information update in real time, visualized as an entropy Venn diagram.

information-theorymutual-informationentropyjoint-distributionkl-divergenceprobability

2D · HTML5 Canvas 2D · 60 FPS target · runs fully client-side, no install