What is deep learning? Layers, data and why it works

Deep learning is a kind of machine learning that uses neural networks with many layers stacked on top of each other. Each layer turns what the layer below found into something more abstract: in a picture, pixels become edges, edges become shapes, and shapes become a face or a cat. "Deep" means only that there are many layers. Deep learning is what powers image recognition, speech recognition, translation and chatbots.

In short

• A deep network learns its own features from examples, instead of a programmer writing rules for what to look for.
• It took off around 2012, when large labeled datasets, graphics chips and better training methods arrived together.
• It is very good at finding patterns in huge amounts of data, and it needs huge amounts of data and computing to do it.

What does "deep" mean in deep learning?

It refers to depth in layers, not depth of understanding. A neural network is made of simple units, loosely modeled on brain cells, arranged in layers. Each unit takes numbers from the layer before, multiplies each by a weight, adds them up and passes the result on. A network with a single hidden layer between input and output is shallow. One with many is deep: the 2012 network that started the boom had 8 layers that learned, and many networks since have dozens or hundreds.

Depth matters because each layer can build on the last. In 2013, two researchers at New York University, Matthew Zeiler and Rob Fergus, made pictures of what each layer of an image network responded to. The first layers lit up for edges and patches of color. Middle layers responded to corners, textures and repeated patterns. Higher layers responded to parts of objects, such as dog faces or birds' legs, and the top layer to whole objects, such as keyboards and dogs. Nobody told the network to look for dog faces. The layers discovered their own ladder of features, because those features helped it get the answers right.

That is the difference from earlier machine learning, where engineers spent years hand-designing features, such as edge detectors for images or sound patterns for speech, and the learning came only at the end. Deep learning learns the features too.

How does a deep network learn?

It learns by being wrong and adjusting, millions of times. Here is the process on a small, classic task: reading handwritten digits from images 28 pixels square.

1. Input. Each image is 784 numbers, one per pixel, saying how dark it is.
2. Layers. Suppose one hidden layer of 128 units and an output layer of 10 units, one for each digit from 0 to 9. Every input connects to every hidden unit, so that layer has 784 × 128 = 100,352 weights, plus 128 adjustable offsets called biases: 100,480 numbers. The output layer adds 128 × 10 weights and 10 biases: 1,290 more. The whole network has 101,770 numbers to learn.
3. Guess. At first the numbers are random, so for a picture of a 7 the network might give 7 a score of 11% and 3 a score of 14%.
4. Measure the error. The training data says the answer is 7. The difference between the network's scores and the right answer is turned into a single error number.
5. Adjust. An algorithm called backpropagation works out how much each of the 101,770 numbers contributed to the error, and nudges each one slightly in the direction that would have reduced it.
6. Repeat. Over tens of thousands of labeled images, seen many times, the nudges add up to a network that reads most digits correctly.

That network has one hidden layer, so it isn't deep. Make it deep and the count climbs fast. The image network that started the boom in 2012 had about 60 million numbers to learn, and the largest language models of the 2020s have hundreds of billions or more. The training method is the same one, scaled up. Backpropagation itself was popularized in a 1986 paper in Nature by David Rumelhart, Geoffrey Hinton and Ronald Williams.

Why did deep learning take off around 2012?

The ideas were decades old, but three things arrived together. The first was data: ImageNet, a collection of millions of labeled photos put together at Princeton and Stanford, gave networks enough examples to learn from. The second was hardware: graphics processing units (GPUs), built for video games, turned out to be very fast at the multiplications neural networks need. The third was better methods, small changes to how units fire and how training avoids memorizing the examples.

The turning point was a contest. In 2012, a network from the University of Toronto, later called AlexNet, entered the ImageNet image-recognition challenge. Its paper reports that its five guesses missed the right answer 15.3% of the time, against 26.2% for the next-best entry. It was trained on two GPUs. Within a few years nearly every competitive entry was a deep network.

Year	What happened
1958	Frank Rosenblatt builds the perceptron, a one-layer learning machine
1986	Backpropagation is popularized for training networks with hidden layers
1989	Yann LeCun's network learns to read handwritten zip codes
2012	AlexNet wins the ImageNet challenge by a wide margin
2015	Yann LeCun, Yoshua Bengio and Geoffrey Hinton publish their review of deep learning in Nature
2016	DeepMind's AlphaGo beats Lee Sedol, one of the world's best Go players
2017	Google researchers introduce the transformer, the design behind today's language models
2019	LeCun, Bengio and Hinton share the 2018 Turing Award, computing's top prize
2022	ChatGPT is released to the public on 30 November
2024	Nobel Prizes go to John Hopfield and Hinton in physics, and in chemistry partly to the makers of AlphaFold

Where is deep learning used?

Almost everywhere AI shows up. Phone cameras use it to recognize faces and sharpen photos. Voice assistants use it to turn speech into text. Translation services, spam filters and the recommendations on video and shopping sites run on it. In science, DeepMind's AlphaFold predicts the shapes of proteins from their sequences, work that shared the 2024 Nobel Prize in Chemistry.

The chatbots of the 2020s are deep learning too. The "T" in GPT stands for transformer, a 2017 design described in what GPT stands for, and a large language model is a very deep transformer trained to predict the next word across a vast amount of text. That is why it needs data centers full of chips to train.

What can't deep learning do well?

It needs a great deal of data and computing, and it fails in ways that are hard to predict. A network can be very accurate on examples like the ones it trained on and badly wrong on ones that differ in a way a person wouldn't notice. Its millions of numbers don't explain themselves, so it is hard to say why it gave a particular answer, a problem known as interpretability. And it learns whatever patterns are in its data, including errors and biases.

Still argued

Whether deep learning alone can lead to human-level intelligence is argued. Some researchers point out that scaling up data and computing has kept producing new abilities. Others, such as the cognitive scientist Gary Marcus, argue that pattern-learning lacks reasoning and an understanding of cause and effect, and will need to be combined with other methods. The record so far shows steady gains on tests, not a settled answer.

Deep learning is one branch of artificial intelligence, and right now the dominant one.

Check yourself

• Without looking back: what does "deep" refer to, and why does having many layers help?
• In your own words, how does a network use its errors to get better?
• Name the three things that came together around 2012, and why each one mattered.

Sources

• Deep learning — Yann LeCun, Yoshua Bengio and Geoffrey Hinton, Nature, 2015
• ImageNet Classification with Deep Convolutional Neural Networks — Alex Krizhevsky, Ilya Sutskever and Geoffrey Hinton, NeurIPS, 2012
• Visualizing and Understanding Convolutional Networks — Matthew Zeiler and Rob Fergus, 2013
• Learning representations by back-propagating errors — David Rumelhart, Geoffrey Hinton and Ronald Williams, Nature, 1986
• The Nobel Prize in Chemistry 2024, press release — Royal Swedish Academy of Sciences, 2024