Shannon Entropy: Measuring Information

Shannon Entropy

Preset distribution

Number of symbols (custom): 8 Drag a probability bar in the chart to reshape the distribution — all bars renormalize to sum to 1. 20-questions demo

Statistics

Symbols N—

Entropy H (bits)—

Max entropy log₂N—

Redundancy—

Selected symbol—

p(selected)—

Info content −log₂p—

Questions asked—

Entropy H = −Σ p(x)·log₂p(x) is the average number of bits of genuine surprise a source produces per symbol. A fair coin (p=0.5, 0.5) has H=1 bit; a lopsided coin is more predictable, so each flip carries less new information. The "20 questions" demo below asks yes/no questions that roughly halve the remaining probability mass each time — the number of questions needed to pin down a symbol tracks −log₂p(symbol), which is exactly why Shannon's source coding theorem makes H the theoretical floor for average bits/symbol in any lossless code.