What is survivorship bias? The mistake of seeing winners

Survivorship bias is the error of drawing conclusions only from the people or things that made it through some filter, such as success, survival or simply still being around, while forgetting the ones that didn't. Because the failures are out of sight, the survivors look typical when they are not. It skews business advice, investment records, history and everyday judgments about what works.

In short

• The survivors you can see were selected, so what they have in common may have nothing to do with why they survived.
• The classic case is the Second World War statistician Abraham Wald, who worked out how to judge aircraft damage from only the planes that came back.
• The habit that counters it is to ask, every time, what the missing cases would look like and how many there were.

What is survivorship bias, in plain terms?

It is a kind of selection bias: the sample you are looking at was filtered, and the filter is related to the thing you are trying to judge. If you study only the winners, you can't tell what made them win, because you never see the losers who did the same things.

Two features make it hard to notice. The missing cases are usually invisible by nature: failed companies don't publish memoirs, closed funds drop out of the rankings, and planes that crashed don't land for inspection. And the survivors often tell a convincing story about why they made it, which fills the gap where the evidence should be.

What really happened with Wald and the bombers?

The story usually goes like this. During the Second World War, the military studied the bullet holes on bombers returning from missions and planned to add armor where the holes were thickest. The statistician Abraham Wald pointed out that those were the places where a plane could be hit and still come home. The armor should go where the returning planes had no holes, because planes hit there were the ones that never returned.

The core of that is Wald's insight, but the scene of officers proposing armor where the holes were comes from later retellings rather than from his technical reports, and the real work was more technical. Wald was a member of the Statistical Research Group at Columbia University, which advised the US military, and he wrote a series of memoranda on estimating how vulnerable each part of a plane was, using only the damage on aircraft that survived. The Center for Naval Analyses reprinted them in 1980 under the title A Method of Estimating Plane Vulnerability Based on Damage of Survivors (CNA, 1980). His contribution was a way to calculate, from the pattern of hits on the planes that returned, the probability that a hit in each area brought a plane down.

The diagram of a bomber covered in red dots that appears in almost every article on the subject is not from Wald. It is a modern illustration on Wikimedia Commons, described there as a "hypothetical damage pattern," loosely based on an "unillustrated report" by Wald (Wikimedia Commons). The picture is useful, but it isn't a historical document.

Where does survivorship bias show up in everyday life?

Business and startup advice. A founder who dropped out of college and built a large company is interviewed; the many who dropped out and failed are not. Advice drawn from the success stories ("take big risks", "never give up") may be equally true of people who failed, and you can't tell without them.

"They don't make them like they used to." The old buildings, furniture and cars still around today are the ones that were built well or cared for. Most of what was built at the same time was demolished, broken or scrapped long ago. Comparing today's average product with the past's survivors makes the past look better made than it was.

Investment track records. Funds that do badly are often closed or merged into others, and their records vanish from the lists investors compare. The funds left look better than the average fund actually did. A widely cited 1996 study of US mutual funds measured the resulting overstatement at about 0.9% a year in risk-adjusted return (Elton, Gruber and Blake, 1996, as summarized on Wikipedia). This explains a statistical effect; it isn't investment advice.

Health and history. Very old people asked for the secret of their long lives may name whisky or daily walks, but nobody interviews the whisky drinkers who died at 60. And the ancient texts, buildings and artworks that survive are a filtered sample of what existed, which shapes the modern picture of the past.

How much can it distort a number?

Here is a worked example with made-up numbers. A fund company launches 100 funds. Over 10 years, 30 of them do poorly and are closed. The 70 still open have averaged 7% a year, and the company advertises that figure.

Group	Funds	Average yearly return
Survivors, still open	70	7%
Closed along the way	30	2%
All funds launched	100	5.5%

The average across every fund launched is (70 × 7% + 30 × 2%) ÷ 100 = 5.5%. Someone who picked a fund at random at the start would have expected 5.5%, not 7%. The 1.5-point gap comes entirely from the 30 funds no longer on the list. Over 10 years, $1,000 compounding at 7% grows to about $1,967; at 5.5%, to about $1,708.

How do you spot the missing data?

A short checklist to run on any claim based on examples:

1. How was this group chosen? Was there a filter (success, survival, still existing, being famous) between the full set and the cases in front of you?
2. Who is missing? Picture the cases that didn't make it. Where would they be, and why can't you see them?
3. Did the missing ones do the same thing? If failures shared the trait that the survivors credit for their success, the trait explains nothing.
4. What's the denominator? "Ten people got rich doing this" means little without how many tried.

Still argued

How large survivorship bias is depends on the field and the data, and researchers argue over specific estimates, such as how much it inflates fund returns or historical stock-market returns across countries. There is no dispute that the bias exists. The arguments are about measurement, and about whether a particular data set already corrects for it.

Survivorship bias is a close cousin of other traps in reasoning. Like the Monty Hall problem, it turns on how information was filtered before you saw it. It feeds overconfidence of the kind described by the Dunning-Kruger effect when people learn only from success. And spotting it is a core skill of critical thinking, alongside rules of thumb like Occam's razor. Questions about what can be known from a partial sample go back to the beginnings of philosophy.

Check yourself

• In your own words, why did Wald think the armor belonged where the returning planes had no holes?
• Give an everyday example of survivorship bias that isn't in this note.
• What four questions help you spot missing data in a claim based on examples?

Sources

• A Reprint of "A Method of Estimating Plane Vulnerability Based on Damage of Survivors" by Abraham Wald — Center for Naval Analyses, 1980
• File:Survivorship-bias.svg — Wikimedia Commons, 2021
• Survivor Bias and Mutual Fund Performance — Edwin J. Elton, Martin J. Gruber and Christopher R. Blake, The Review of Financial Studies, 1996
• Survivorship bias — Wikipedia