What is standard deviation? Measuring spread, step by step

Standard deviation is a measure of spread: roughly, how far the values in a set of data typically sit from their mean. A small standard deviation means the values are bunched close to the average; a large one means they are scattered widely. It is the square root of the variance, the mean of the squared distances from the mean, and it comes out in the same units as the data itself.

In short

• Two data sets can share the same average and look nothing alike; standard deviation is the number that tells them apart.
• To compute it: find the mean, take each value's distance from it, square those, average the squares (that is the variance), then take the square root.
• For bell-shaped data, about 68% of values fall within 1 standard deviation of the mean and about 95% within 2.

Why isn't the average enough?

An average describes the center of the data and says nothing about how far the values stray from it. Take two groups of five students who sat the same test:

• Group A: 70, 75, 80, 85, 90
• Group B: 60, 65, 80, 95, 100

Both have a mean of 80, and both have a median of 80. Yet group A's scores are all within 10 points of the middle, while group B's run from 60 to 100. A teacher planning the next lesson would want to know that. The range, highest minus lowest, hints at it (20 points against 40), but the range depends on just two values, the most extreme ones. Standard deviation uses every value, which is why it became the standard measure of spread. For more on choosing the center itself, see mean, median and mode.

How do you calculate standard deviation? A worked example

Work through group A, step by step.

1. Find the mean. 70 + 75 + 80 + 85 + 90 = 400, and 400 ÷ 5 = 80.
2. Find each deviation, the value minus the mean.
3. Square each deviation. This makes every distance positive, so values below the mean don't cancel values above it.
4. Add the squares and divide by the number of values. That gives the variance.
5. Take the square root to get back to the original units, here test points.

Score	Deviation from 80	Squared deviation
70	−10	100
75	−5	25
80	0	0
85	+5	25
90	+10	100
Total	0	250

The deviations add to 0, as they always do: that is what being the mean means, and why the squaring step is needed. The variance is 250 ÷ 5 = 50 "points squared", a unit no one can picture. Its square root is the standard deviation: √50 ≈ 7.1 points.

Now group B. Its deviations are −20, −15, 0, +15 and +20; their squares are 400, 225, 0, 225 and 400, adding to 1,250. The variance is 1,250 ÷ 5 = 250, and the standard deviation is √250 ≈ 15.8 points. Same mean, more than twice the spread, and one number now says so.

In symbols, for a whole population of N values with mean μ (the Greek letter mu), the standard deviation is σ = √( Σ(x − μ)² ÷ N ). σ is the Greek letter sigma; Σ, capital sigma, means "add up over every value"; x stands for each value in turn.

Why do some formulas divide by n − 1?

Because there are two situations, and they need slightly different formulas. If the five scores are the whole group you care about, the population, divide by N, as above. If they are a sample drawn from a bigger population, say five students chosen to estimate how spread out the whole school's scores are, divide by n − 1 instead. The sample formula is s = √( Σ(x − x̄)² ÷ (n − 1) ), where x̄ is the sample's mean; the NIST/SEMATECH e-Handbook gives it this way.

The reason: a sample's own mean sits, by construction, as close as possible to the sample's values. Distances measured from it come out a little smaller, on average, than distances from the true population mean would. Dividing by one less than the count nudges the answer up to make up for it, a fix called Bessel's correction (Wikipedia). For group A, the sample version gives √(250 ÷ 4) = √62.5 ≈ 7.9 points instead of 7.1. The difference shrinks as samples grow: with 1,000 values, dividing by 999 rather than 1,000 barely matters.

This is the commonest source of confusion in practice. Spreadsheets offer both: STDEV.P for a population and STDEV.S for a sample, and calculators often show σ and s side by side. Using the wrong one on a small data set gives a noticeably different number.

What does a standard deviation tell you?

On its own, a standard deviation is a yardstick for distance from the mean. A score of 95 in group A is (95 − 80) ÷ 7.1 ≈ 2.1 standard deviations above the mean, which is far out; in group B the same score is less than 1 standard deviation above. This count of standard deviations is called a z-score, and it lets you compare values measured on different scales. IQ scores are built this way: they are set so the mean is 100 and the standard deviation 15, as the note on what IQ measures explains.

When data follow a bell shape, the yardstick becomes a map. About 68% of values lie within 1 standard deviation of the mean, about 95% within 2, and about 99.7% within 3, more precisely 68.27%, 95.45% and 99.73% (Wikipedia). This 68-95-99.7 rule belongs to the normal distribution, and it holds only roughly, or not at all, for data that aren't bell-shaped.

Where does standard deviation show up?

• Finance. An investment's volatility is the standard deviation of its returns. A fund whose yearly returns swing between −20% and +30% has a much larger standard deviation than one that moves between 3% and 6%, whatever their averages. Indexes like the S&P 500 are often described this way.
• Science. The error bars on a chart often show one standard deviation, or the related standard error, which is the standard deviation divided by the square root of the sample size. Deciding whether a difference is bigger than chance would produce, which a p-value reports, starts from these measures of spread.
• Manufacturing. Factories track the standard deviation of a part's size. "Six Sigma" quality programs take their name from σ: the goal is a process so consistent that the tolerance limits sit about six standard deviations from the target.

The term itself was introduced in writing by the statistician Karl Pearson in 1894, after he had used it in lectures; before that, Carl Friedrich Gauss had called a similar idea the "mean error" (Wikipedia).

What do people get wrong about standard deviation?

The most common slip is to read it as the average distance from the mean. It is close to that but not the same. Group A's values sit, on average, 6 points from the mean (10 + 5 + 0 + 5 + 10 = 30, and 30 ÷ 5 = 6), while its standard deviation is 7.1. Squaring gives large deviations extra weight, so the standard deviation is always at least as big as the mean absolute distance, and one outlier inflates it sharply. Squaring is used anyway because it has convenient mathematics: variances of independent quantities add up, which absolute distances don't, and that property runs through all of probability and statistics.

A second slip is assuming the 68-95-99.7 rule applies to any data. For incomes, wait times or anything with a long tail, far more values can lie beyond 2 or 3 standard deviations than the rule allows. Standard deviation itself is still a valid number for such data, just a less complete description.

Check yourself

• Without looking back, walk through the five steps for the scores 4, 6 and 8, and say why the deviations must add to zero.
• When would you divide by n − 1 instead of n, and why does it make the answer larger?
• Why is a standard deviation not quite the average distance from the mean?

Sources

• Measures of Scale, in the e-Handbook of Statistical Methods — NIST/SEMATECH
• Measures of the Spread of the Data, in Introductory Statistics 2e — OpenStax, Rice University
• Standard deviation — Wikipedia
• 68–95–99.7 rule — Wikipedia