Information Theory Explained
Quantify uncertainty and information. Learn how entropy bounds compression and channel capacity limits reliable communication.
📚 Fundamentals
- Entropy H(X): average uncertainty
- KL divergence D(P||Q): mismatch penalty
- Mutual information I(X;Y): shared information
❓ Frequently Asked Questions
1) Bits?
Log base 2 measures information in bits.
Log base 2 measures information in bits.
2) Lossless compression limits?
Average code length ≥ entropy.
Average code length ≥ entropy.
3) Channel capacity?
Maximal reliable rate under noise.
Maximal reliable rate under noise.