What Is Information?
Before Shannon, "information" had no mathematical definition. Engineers knew that transmitting a message over a noisy telephone line was unreliable, but they had no precise way to quantify what was being transmitted or what was being lost. Shannon's fundamental insight was radical: information is not about meaning, semantics, or importance. It is about surprise — the degree to which a message reduces uncertainty.
Formally, the information content of an event with probability p is I = −log₂(p) bits. The choice of base 2 gives units of bits (binary digits). An event with probability 1/2 (a fair coin flip) carries exactly 1 bit of information. An event with probability 1/4 carries 2 bits. An event with probability 1/8 carries 3 bits. A certain event (p = 1) carries 0 bits — it tells you nothing you didn't already know. This definition is the only one consistent with three natural requirements: information is additive for independent events, is a continuous function of probability, and is maximized for uniform distributions.
Shannon Entropy
For a source that produces symbols from an alphabet with probabilities p₁, p₂, ..., pₙ, the Shannon entropy is: H = −Σᵢ pᵢ log₂(pᵢ) bits per symbol. This measures the average information content per symbol — equivalently, the average uncertainty before observing each symbol.
A fair coin has H = 1 bit. A coin biased to heads with probability 0.9 has H ≈ 0.469 bits — much less uncertainty, hence less information per flip. English text, with its highly predictable letter frequencies and grammatical structure, has an entropy of roughly 1.0–1.5 bits per character (Shannon estimated this through human-subject experiments). This means English text has enormous redundancy — roughly 75% of characters are predictable from context.
Shannon entropy is formally identical to the thermodynamic entropy in Boltzmann's formula, with k replaced by the logarithm base. This is not a coincidence: both measure the number of possible states (microstates in physics, messages in communication theory) compatible with known macroscopic constraints. The deep connection between information theory and thermodynamics — explored by Rolf Landauer, Charles Bennett, and others — reveals that erasing information has a minimum thermodynamic cost of kT ln(2) joules per bit. This "Landauer's principle" connects computation, information, and physics at a fundamental level.
Data Compression
Shannon's source coding theorem (1948) establishes the fundamental limit on lossless data compression: no algorithm can compress data below H bits per symbol on average, where H is the source entropy. Above H, perfect lossless compression is achievable (in principle, for long messages). This limit is both a floor below which no algorithm can go and a target that good algorithms approach.
Huffman coding (1952) assigns variable-length binary codes to symbols, with shorter codes for more frequent symbols — just as Morse code gives 'E' a single dot (the most common English letter). Huffman coding is optimal among symbol-by-symbol codes. Arithmetic coding operates on entire messages and can approach the entropy limit arbitrarily closely. The LZ77 and LZ78 algorithms (1977–78), used in zip, gzip, and PNG, exploit repeated patterns in data rather than symbol statistics, and form the basis of most modern general-purpose compressors. Contemporary compressors like Brotli (used in web browsers) and Zstandard combine statistical modeling with entropy coding to achieve compression within a few percent of the theoretical limit.
The Noisy Channel Theorem
Shannon's most astonishing result — the noisy channel coding theorem — demolished what engineers believed was a fundamental limitation. The intuition was: any real communication channel adds noise, errors corrupt messages, and the only way to reduce errors is to transmit more slowly. Shannon proved this intuition wrong.
For any channel with capacity C = B · log₂(1 + S/N) bits/second (where B is bandwidth in Hz and S/N is signal-to-noise power ratio), it is possible to transmit information at any rate R < C with an error probability that approaches zero as message length increases — simply by choosing the right error-correcting code. Transmission at rates above C is impossible regardless of the code. The channel capacity C is the fundamental information-theoretic limit.
The proof is non-constructive: Shannon showed that a random code would work with high probability, but did not say which code to use. Finding practical codes that approach the Shannon limit occupied engineers for the next five decades.
Explore the algorithms that encode, compress, and transmit information in the Huffman Coding simulation — see how variable-length codes and encoding relate to the underlying information-theoretic principles Shannon uncovered.
Error-Correcting Codes
Richard Hamming, working at Bell Labs in 1950, devised the first error-correcting codes. Hamming codes add redundant check bits to a message such that any single-bit error can be not merely detected but corrected. A (7,4) Hamming code sends 7 bits for every 4 data bits, allowing correction of any single-bit error. The scheme works by choosing codewords that are maximally far apart in "Hamming distance" (the number of bit positions where they differ).
Reed-Solomon codes (1960), now used on every CD, DVD, QR code, and in deep-space communication, treat the data as coefficients of a polynomial and transmit extra evaluation points. This allows correction of burst errors — a scratch on a CD may corrupt many consecutive bits, and Reed-Solomon can reconstruct the original data as long as enough evaluation points survive. Turbo codes (1993) and low-density parity-check (LDPC) codes can achieve transmission rates within a fraction of a percent of the Shannon limit — a feat considered practically impossible for decades. NASA's Deep Space Network uses LDPC codes on signals from the Voyager probes, now over 20 billion kilometers away.
Information Theory in Biology and Machine Learning
DNA can be analyzed as an information channel, with mutations acting as noise. The human genome encodes roughly 6.4 billion base pairs, corresponding to about 1.5 gigabytes of raw information — though the effective information content is lower due to repetitive sequences and redundant codons (the genetic code uses 64 codons for 20 amino acids plus stop signals). Information-theoretic analysis of genomic sequences helps identify functional regions: low-entropy regions (highly conserved sequences) often correspond to essential genes or regulatory elements.
In machine learning, Shannon entropy appears throughout. Cross-entropy loss — the standard training objective for classification networks — is exactly the Shannon entropy between the true label distribution and the model's predicted distribution. Minimizing cross-entropy is equivalent to maximum likelihood estimation. Mutual information I(X;Y) = H(X) − H(X|Y) measures how much knowing variable Y reduces uncertainty about X; it is used for feature selection, representation learning, and in information bottleneck methods that compress representations to retain only task-relevant information. The minimum description length (MDL) principle, due to Jorma Rissanen, frames statistical learning as a compression problem: the best model is the one that most compresses the data, naturally balancing goodness-of-fit against model complexity.