What is information theory? How Shannon measured information

Information theory is the mathematics of measuring information and of storing and sending it efficiently and reliably. Its founding idea, set out by the engineer Claude Shannon in 1948, is that information is surprise: a message carries more information the less predictable it was, and the amount can be measured in bits. From that one idea come hard limits on how far data can be compressed and how fast it can cross a noisy channel.

In short

• The information in an outcome depends only on how unlikely it was: an outcome with probability p carries log₂(1/p) bits.
• Entropy is the average information per symbol from a source, and it is the floor for lossless compression.
• Every channel has a capacity, a top speed for error-free communication, and clever codes can get close to it but never past it.

What did Shannon mean by information?

Shannon set aside meaning altogether. To an engineer sending a message, what matters is not what it says but which of the possible messages it is. If you already know what someone will say, hearing it tells you nothing. If it could have been any of a million things, hearing which one tells you a lot.

His paper, A Mathematical Theory of Communication, appeared in the Bell System Technical Journal in July and October 1948 (Wikipedia). It pictured every communication system the same way: a source produces a message, a transmitter turns it into a signal, the signal crosses a channel where noise corrupts it, and a receiver tries to rebuild the message. A phone call, a radio broadcast, a hard drive and a strand of DNA all fit. Earlier engineers, Harry Nyquist and Ralph Hartley in the 1920s, had measured how much a line could carry; Shannon brought in probability and turned the subject into a theory.

The unit is the bit, the same word as a binary digit, which Shannon's paper put in print for the first time. One bit is the information in one fair yes-or-no choice. The two meanings are linked but not identical: a stored binary digit can carry less than one bit of information if its value was predictable, which bits vs bytes touches on from the storage side.

How do you measure information? Worked examples

The information in a single outcome with probability p is log₂(1/p) bits. Log₂ asks "2 to what power gives this number?", so an outcome with probability 1/2 carries 1 bit, 1/4 carries 2 bits and 1/8 carries 3 bits. Halve the probability and you add one bit of surprise.

A fair coin. Heads and tails each have probability 1/2, so each outcome carries log₂ 2 = 1 bit. On average, a toss carries 1 bit.

A loaded coin that lands heads 3 times in 4. Heads is unsurprising: log₂(1/0.75) ≈ 0.415 bits. Tails is a surprise: log₂(1/0.25) = 2 bits. The average, weighting each by how often it happens, is 0.75 × 0.415 + 0.25 × 2 ≈ 0.311 + 0.5 = 0.811 bits per toss. That average is the source's entropy, written H:

H = Σ p × log₂(1/p), added up over every possible outcome.

The loaded coin's entropy is below 1 bit because you can partly guess it. A coin that always lands heads has entropy 0: it tells you nothing. A fair six-sided die has entropy log₂ 6 ≈ 2.58 bits, and among all sources with a given number of outcomes, the one with every outcome equally likely has the highest entropy. Shannon borrowed the word from physics because his formula has the same form as one for thermodynamic entropy, and the note on entropy explains how the two connect.

Why is entropy the limit on compression?

Because entropy says how many bits per symbol a message really needs. Take a city's daily weather report, with four possible values:

Weather	Probability	Fixed-length code	Variable-length code	Bits used
Sunny	1/2	00	0	1
Cloudy	1/4	01	10	2
Rain	1/8	10	110	3
Snow	1/8	11	111	3

The fixed-length code always uses 2 bits. The variable-length code gives the common report a short codeword and the rare ones longer codewords, and no codeword is the start of another, so a string like 0110100 can be read back without gaps: sunny, rain, cloudy, sunny. Its average length is 1/2 × 1 + 1/4 × 2 + 1/8 × 3 + 1/8 × 3 = 1.75 bits per report.

Now compute the entropy: 1/2 × 1 + 1/4 × 2 + 1/8 × 3 + 1/8 × 3 = 1.75 bits. The clever code lands exactly on the entropy, and Shannon proved no lossless code can average less. That is his source coding theorem (Wikipedia). Here the code saves 12.5% over the fixed one; the more lopsided the probabilities, the bigger the saving. In 1952 the MIT student David Huffman published an algorithm that builds the shortest such code for any set of probabilities (Wikipedia). Modern formats from ZIP files to images build on these ideas, which the note on how compression works follows into practice.

English is highly predictable: after "q", "u" is almost certain. Estimates put ordinary English text at between 0.6 and 1.3 bits of entropy per character, well under the 8 bits a byte uses to store each letter, which is why text compresses so well.

What is channel capacity?

Every channel has noise: static on a radio, scratches on a disc, interference on a cable. Shannon's second great result, the noisy-channel coding theorem, says that each channel has a capacity C, a number of bits per second. Below C, there are codes that make the error rate as small as you like. Above it, no code can.

For a channel limited by bandwidth B (in hertz) and signal-to-noise ratio S/N, the Shannon–Hartley theorem gives C = B × log₂(1 + S/N). For a telephone-style line with 4,000 Hz of bandwidth and a signal 100 times stronger than the noise, C = 4,000 × log₂ 101 ≈ 26,600 bits per second (Wikipedia). Doubling the bandwidth doubles the capacity; doubling the signal power adds only about one bit per second for each hertz. That trade-off shapes every Wi-Fi standard and mobile network.

The surprise in the theorem was that perfect reliability doesn't require slowing to a crawl. Before Shannon, engineers assumed more noise meant repeating each message more and more. He showed that a fixed overhead, chosen well, is enough. He didn't say how to build such codes, and finding practical ones took more than four decades.

What did information theory make possible? A timeline

Year	Step	Where it is used
1948	Shannon's paper defines information, entropy and capacity	The foundation for what follows
1950	Richard Hamming invents codes that correct single-bit errors, first to fix reading errors in Bell Labs relay computers (Wikipedia)	Computer memory
1952	Huffman coding gives the shortest variable-length codes	Inside ZIP, JPEG and MP3
1960	Robert Gallager proposes low-density parity-check (LDPC) codes, little used until rediscovered in the 1990s (Wikipedia)	Wi-Fi (from 802.11n), 10-gigabit Ethernet, 5G data channels
1977 to 1978	Abraham Lempel and Jacob Ziv's compression methods, which need no table of probabilities in advance	ZIP, GIF and PNG files
1993	Turbo codes come within a fraction of a decibel of Shannon's limit	3G and 4G mobile networks
2009	Erdal Arıkan publishes polar codes, the first with an explicit construction proved to reach capacity on a broad class of channels	5G control channels

The same mathematics reaches beyond engineering. Machine learning trains models by minimizing cross-entropy, a measure of how surprised a model is by real data, which is how a large language model learns to predict the next word, and how natural language processing measures how well a model fits text.

The commonest misreading is to think information theory is about meaning or truth. A random string of letters has more Shannon information per character than a sentence, because it is less predictable. The theory measures how much must be sent to pick out a message, not whether the message is worth sending. Shannon said so on his paper's first page: messages often have meaning, but "these semantic aspects of communication are irrelevant to the engineering problem."

Check yourself

• Without looking back, explain why a coin that lands heads three times in four carries less than 1 bit per toss.
• Why can no lossless code for the weather reports average fewer than 1.75 bits?
• What does a channel's capacity promise, and what does it rule out?

Sources

• A Mathematical Theory of Communication — Claude E. Shannon, Bell System Technical Journal, 1948
• Information theory — Wikipedia
• Entropy (information theory) — Wikipedia
• Shannon–Hartley theorem — Wikipedia
• Huffman coding — Wikipedia
• Low-density parity-check code — Wikipedia