What is Bayes' theorem? Updating beliefs with evidence

Bayes' theorem is the rule for updating a probability in the light of new evidence. It says how likely a claim is after you see some evidence, given how likely the claim was before and how much more often that evidence turns up when the claim is true than when it is false. Its most famous lesson is that a positive result on an accurate test for a rare condition is often wrong.

In short

• Bayes' theorem combines two things: how likely a claim was to begin with (the base rate) and how strongly the new evidence points to it.
• When the base rate is low, even good evidence can leave the claim unlikely, because false alarms from the large majority outnumber true hits from the few.
• Counting imaginary people, "natural frequencies", makes the arithmetic easy enough to do in your head.

What does Bayes' theorem say?

In symbols, for a claim A and evidence B:

P(A | B) = P(B | A) × P(A) ÷ P(B)

Read the vertical bar as "given". P(A | B) is the probability of A given that B is observed, the thing you want. P(A) is the prior, the probability of A before the evidence. P(B | A) is the likelihood, how probable the evidence is if A is true. P(B) is the overall probability of seeing the evidence at all, from either cause. The result, P(A | B), is called the posterior.

The theorem follows in two lines from the basic rules of probability, which is why the Stanford Encyclopedia of Philosophy calls it a mathematical triviality whose insight lies at the heart of whole approaches to statistics and reasoning. It is named after Thomas Bayes, an English minister who worked it out for a special case. After Bayes died in 1761, his friend Richard Price edited his essay, which was read to the Royal Society on 23 December 1763 and printed in its Philosophical Transactions. Pierre-Simon Laplace reached the same result independently in 1774, apparently unaware of Bayes, and developed it into the general method used today (Wikipedia).

Why does a positive test sometimes mean only 8%? A worked example

Suppose a condition affects 1% of people, and a test for it is 90% accurate both ways: it correctly flags 90% of people who have the condition, and correctly clears 90% of people who don't. You test positive. What is the chance you have the condition?

Most people answer 90%. Count 1,000 imaginary people instead:

	Test positive	Test negative	Total
Have the condition (1%)	9 (90% of 10)	1	10
Don't have it (99%)	99 (10% of 990)	891	990
Total	108	892	1,000

Of the 108 people who test positive, only 9 have the condition. The chance a positive result is right is 9 ÷ 108, about 8.3%, not 90%. The test is good; the condition is rare. The 10% false-positive rate applies to 990 healthy people and produces 99 false alarms, which swamp the 9 true cases.

The theorem gives the same answer: P(condition | positive) = 0.9 × 0.01 ÷ 0.108 = 0.083. The table is just the theorem written as counts. Presenting risks this way, as counts of people rather than percentages, is called using natural frequencies, and studies find that both laypeople and doctors reason far more accurately with it (Wikipedia).

Illustrative arithmetic

These numbers are made up to show the reasoning. Real tests have different accuracies for positives and negatives, and a doctor orders a test because symptoms already raise the starting probability. This note explains a rule of probability; it isn't medical advice.

What happens when more evidence arrives?

Each posterior becomes the prior for the next piece of evidence. That is what updating means. The cleanest way to see it uses odds, the chances for against the chances against:

posterior odds = prior odds × likelihood ratio

The likelihood ratio is how many times more likely the evidence is if the claim is true than if it is false. For the test above, a positive result happens 90% of the time with the condition and 10% without, a ratio of 9.

1. Before any test, the odds are 1 to 99 (1% have it).
2. After one positive test, multiply by 9: odds of 9 to 99, which is 9 ÷ 108, about 8.3%. The same answer as the table.
3. After a second, independent positive test, multiply by 9 again: odds of 81 to 99, which is 81 ÷ 180, or 45%.
4. After a third, 729 to 99, about 88%.

The rule shows why strong claims need either strong evidence or a lot of it: each piece of evidence multiplies the odds, and a very low starting point takes several multiplications to overcome. It also shows that the order doesn't matter, only the total weight of evidence, provided the pieces really are independent. Two tests that make the same mistake for the same reason count only once.

What is the base rate fallacy?

The base rate fallacy is ignoring how common something is when judging evidence about it, as most people do with the test above. The psychologists Daniel Kahneman and Amos Tversky studied it in the 1970s, in work published from 1973, and argued that people judge by how well the evidence matches a picture, how "representative" it is, and neglect the base rate (Wikipedia).

It turns up well beyond medicine. A security system that flags 99% of intruders and wrongly flags 1% of innocent visitors will, at an airport where real threats are extremely rare, raise almost nothing but false alarms. A witness who is right 80% of the time about a taxi's color is less convincing than they sound if one color of taxi is rare in that city. Weighing evidence well means asking both how well it fits the claim and how likely the claim was to begin with. Its mirror image is confirmation bias: noticing only the evidence that fits a belief one already holds, which no amount of Bayesian arithmetic can correct if the evidence is chosen that way.

Where is Bayes' theorem used?

Spam filters. Early spam filters counted how often each word appeared in spam and in ordinary mail and combined those counts with the base rate of spam using Bayes' rule. The note on machine learning works through one with numbers.

Search for lost things. Bayesian search theory divides a search area into squares, gives each a prior probability of holding the target, and updates after every search. In 1968 the US Navy used the method to find the wreck of the submarine USS Scorpion, and it later played a key role in recovering the flight recorders of Air France Flight 447, lost in 2009 (Wikipedia). A failed search is evidence too. Say a square has a 40% chance of holding the wreck and a search there would spot it 80% of the time if it were there. After a search finds nothing, that square's probability falls to 0.4 × 0.2 ÷ (1 − 0.4 × 0.8) = 0.08 ÷ 0.68, about 12%, and every other square's probability rises by the same factor, 1 ÷ 0.68, about 1.47.

Puzzles. The Monty Hall problem is a Bayes problem in disguise: the host's choice of door is evidence, and how much it tells you depends on what he knows.

Still argued

Statisticians agree on the theorem, which is a piece of mathematics. What is argued is how far to rely on it in science. Bayesian statisticians treat probability as a degree of belief and update priors with data; frequentist statisticians avoid priors and judge methods by how often they would err over many repetitions, the logic behind the p-value. Critics of Bayesian methods say the choice of prior can smuggle in the answer; defenders reply that every method makes assumptions and Bayes at least states them. Many working statisticians use both.

Check yourself

• Without the table, explain why most positive results on a 90%-accurate test for a 1-in-100 condition are false alarms.
• What are the prior, the likelihood and the posterior in the medical example?
• Why does a second positive test raise the probability so much more than the first one did?

Sources

• Bayes' Theorem — Stanford Encyclopedia of Philosophy
• Base rate fallacy — Wikipedia
• Bayesian search theory — Wikipedia
• Bayes' theorem — Wikipedia