What is a p-value? Statistical significance, explained

A p-value is the probability of getting a result at least as extreme as the one observed, if the null hypothesis, usually "there's no real effect, only chance", were true. A small p-value means the data would be surprising under that assumption. It does not tell you the probability that the effect is real, and it says nothing about how large or important the effect is. "Statistically significant" usually just means the p-value fell below 0.05.

In short

• A p-value answers one narrow question: if there were no effect, how often would chance alone produce data at least this extreme?
• "Significant" is a threshold, conventionally p < 0.05, not a verdict that a result is true or matters.
• Testing many things and reporting only the small p-values, called p-hacking, manufactures false findings and helped cause the replication crisis.

What does a p-value measure?

Every p-value starts from a null hypothesis, a precise statement of "nothing going on": the coin is fair, the drug does no better than a placebo, the two groups have the same average. The p-value assumes the null is true and asks how often, across many imaginary repeats of the same study, results at least as far from the null as the real ones would turn up by chance. The alternative is the claim the researcher suspects instead. Setting these up is covered in what a hypothesis is.

A p-value measures how surprising the data are if the null hypothesis is true, not how likely the null hypothesis is given the data. The two sound alike and are completely different, the way "the chance a person is Spanish, given they speak Spanish" differs from "the chance a person speaks Spanish, given they are Spanish". Turning one into the other needs a starting estimate of how plausible the hypothesis was in the first place, which is what Bayes' theorem does.

How do you calculate a p-value? A coin, step by step

A coin is tossed 100 times and lands heads 60 times. Is it biased?

1. State the null hypothesis. The coin is fair: each toss is heads with probability 0.5.
2. Work out what the null predicts. A fair coin gives 50 heads on average in 100 tosses. The typical spread, the standard deviation, is the square root of 100 × 0.5 × 0.5, which is 5 heads.
3. Measure how far off the result is. 60 heads is 10 above 50, or 2 standard deviations.
4. Decide what counts as "at least as extreme". A biased coin could favor tails just as well as heads, so a two-sided test counts results 10 or more from 50 in either direction: 60 or more heads, or 40 or fewer.
5. Add up the chances. Counting exactly with the binomial distribution, a fair coin gives 60 or more heads in 100 tosses with probability about 0.028. Doubling for the other side gives about 0.057.

So p ≈ 0.057. If the coin were fair, a result this lopsided would turn up about 1 time in 18. Using the bell-curve shortcut of the normal distribution gives almost the same answer once a small correction is made for counting whole heads.

Here is the catch that makes this example useful. At the usual 0.05 threshold, 60 heads is "not significant". But 61 heads gives a two-sided p of about 0.035, which is. One extra head turns a non-finding into a finding, though the evidence barely changed. And if the researcher had decided in advance to test only for a bias toward heads, the one-sided p for 60 heads would be 0.028, significant again. The threshold is a convention, and results just either side of it are much more alike than the labels suggest.

What does "statistically significant" mean, and what doesn't it?

A result is called statistically significant when its p-value falls below a threshold chosen in advance, called alpha, most often 0.05. The statistician Ronald Fisher popularized 0.05 in the 1920s as a convenient line for a result worth a second look, not as proof. Some fields use far stricter lines: particle physicists claim a discovery only at about five standard deviations, a p-value near 1 in 3.5 million.

The most common misreadings are worth setting side by side.

A p-value of 0.03 means…	True?
If there were no effect, data this extreme would turn up about 3% of the time	Yes
There is a 3% chance the result is due to chance alone	No: that is a statement about the hypothesis, which a p-value can't give
There is a 97% chance the effect is real	No
The effect is large or important	No: a tiny effect in a huge sample can give a tiny p-value
A repeat of the study will probably also give p < 0.05	Not necessarily: many results near 0.05 fail to repeat
p = 0.06 shows there is no effect	No: failing to reject the null isn't evidence that it's true

A significant result also says nothing about why two things are linked. It can confirm that a pattern is unlikely to be luck while leaving open whether it reflects a cause, the question correlation vs causation takes apart.

What did the American Statistical Association say about p-values?

On 7 March 2016, the American Statistical Association issued a statement on p-values, the first time the association's board had addressed the issue on behalf of statisticians, though they had been raising the same concerns for decades. It set out six principles:

1. P-values can indicate how incompatible the data are with a specified statistical model.
2. P-values do not measure the probability that the hypothesis is true, or the probability that the data were produced by chance alone.
3. Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a threshold.
4. Proper inference requires full reporting and transparency.
5. A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.
6. By itself, a p-value does not provide a good measure of evidence about a model or hypothesis.

How did p-hacking feed the replication crisis?

A threshold of 0.05 means that when there is truly no effect, about 1 test in 20 will still come out significant by chance. That is fine for one planned test. It breaks down when researchers run many. With 20 independent tests of effects that don't exist, the chance that at least one gives p < 0.05 is 1 − 0.95²⁰, about 64%.

P-hacking is getting there by small, often innocent-seeming choices: measuring several outcomes and reporting the one that worked, adding participants until the p-value dips under 0.05, dropping awkward data points as outliers, or splitting the sample into subgroups. Each choice is defensible on its own; together they make "significant" results easy to produce from noise. Journals that preferred positive results then published the ones that crossed the line.

When the Open Science Collaboration repeated 100 psychology studies in 2015, only about 36% of the 97 originally significant results were significant again (Wikipedia). What is psychology? tells that story. The main fixes target the choices, not the arithmetic: preregistering the hypothesis and analysis before collecting data, publishing studies on the strength of their design before the results are known, sharing data, and reporting effect sizes with confidence intervals alongside p-values.

Still argued

Statisticians disagree about what should replace the habit of p < 0.05. In 2018, 72 researchers led by Daniel Benjamin proposed lowering the threshold for new discoveries to 0.005. Another large group replied that any single threshold is arbitrary and that researchers should justify their own alpha for each study. Others argue for dropping significance testing in favor of effect sizes and intervals, or for Bayesian methods that state how much the data should shift belief. Nearly all agree that a p-value alone can't carry a conclusion.

Check yourself

• Without looking back, say what the p-value for 60 heads in 100 tosses measures, in one sentence a friend would understand.
• Why is "there's a 3% chance this result is due to chance" the wrong reading of p = 0.03?
• How can a researcher who never fakes any data still end up with a false significant finding?

Sources

• American Statistical Association releases statement on statistical significance and p-values — American Statistical Association, 2016
• Critical values and p values — NIST/SEMATECH e-Handbook of Statistical Methods
• P-value — Wikipedia
• Replication crisis — Wikipedia
• Data dredging — Wikipedia