In 1948 Claude Shannon proved that no lossless code can, on average, represent
symbols from a source using fewer bits than the source's entropy:
H = −Σ pᵢ log₂ pᵢ. This is the hard floor beneath every
compression algorithm — ZIP, JPEG's entropy stage, video codecs, all of them.
⌈−log₂ pᵢ⌉. Frequent symbols get short codes, rare ones get long codes.H, rescaled onto the same axis as the code-length bars.H bits/symbol (entropy coding), the grey column fills at a fixed rate of ⌈log₂ N⌉ bits/symbol (naive fixed-length coding). The cyan column always wins the race.A perfectly uniform source (all symbols equally likely) has maximum entropy — it is already "random" and cannot be compressed at all. Skew the distribution toward one dominant symbol and entropy collapses toward zero: the source becomes almost perfectly predictable, and predictability is exactly what compression exploits.
An interactive 3D symbol-frequency histogram where reshaping a source's probability distribution computes the Shannon entropy live and shows it as the hard floor beneath any compression scheme.
Each pillar's height is a symbol's probability; the translucent overlay is the optimal code length Shannon's theorem predicts for it. The glowing ring marks entropy H — and the streaming bins below race entropy coding against naive fixed-length coding to show the bits actually saved.
Pick an alphabet size, drag the skew slider to make the distribution more lopsided, or hit Randomize for a fresh draw. Watch H fall as one symbol comes to dominate, and watch the entropy-coded bin fill slower than the fixed-length one.
A uniform distribution has maximum entropy and cannot be compressed at all — every symbol is equally surprising. Real text, images and audio are far from uniform, which is exactly why ZIP, JPEG and MP3 work.