What is a neural network? How AI loosely copies the brain

A neural network is a computer program made of many simple connected units, called artificial neurons, that each take in numbers, weigh them, add them up and pass a result on. Arranged in layers, these units can learn to recognize faces, translate languages or write text, by adjusting the weights on their connections until their answers match examples. The design was loosely inspired by the brain, but it works very differently from one.

In short

• One artificial neuron does almost nothing: it multiplies each input by a weight, adds them up, and decides how strongly to pass the result on.
• Stacking thousands or billions of them in layers lets a network turn raw data, such as pixels, into useful answers, such as "this is a cat".
• Training is repeated small nudges: compare the answer with the right one, then adjust every weight a little in the direction that shrinks the error.

What is an artificial neuron?

An artificial neuron is a small calculation with three parts. It receives several numbers as inputs. It multiplies each by a weight, which says how much that input matters, adds the results together with one more number called a bias, and then passes the total through an activation function, which decides how strongly the neuron "fires". The first version of this idea came from Warren McCulloch and Walter Pitts in 1943, who used an all-or-nothing threshold: fire if the total passes a set value, stay silent if not.

Here is one neuron deciding whether to suggest an umbrella. It has two inputs: whether the sky is cloudy (1 for yes, 0 for no) and the forecast chance of rain (0.8 means 80%). Its weights are 0.5 for clouds and 2.0 for the forecast, and its bias is −1.5.

1. Multiply each input by its weight: 1 × 0.5 = 0.5, and 0.8 × 2.0 = 1.6.
2. Add them and the bias: 0.5 + 1.6 − 1.5 = 0.6.
3. Apply the activation. With a simple threshold at zero, 0.6 is above it, so the neuron fires: take the umbrella. With the smoother "sigmoid" function that most early networks used, 0.6 becomes about 0.65, which can be read as "65% sure".

The weights carry everything the neuron knows. A weight of 2.0 on the forecast against 0.5 on the clouds means the forecast counts four times as much. Change the weights and the same neuron makes different decisions.

How do neurons make a network?

One neuron can only draw a simple line between yes and no. Power comes from connecting many in layers. An input layer takes in raw data: for a 28 × 28 pixel image of a handwritten digit, that is 784 numbers, one per pixel's brightness. One or more hidden layers in the middle each take the outputs of the layer before as their inputs. An output layer gives the answer, such as ten neurons, one for each digit from 0 to 9, with the strongest one being the network's guess.

Nobody tells the hidden layers what to look for; they settle on useful features during training. In image networks, early layers often end up responding to edges and patches of contrast, later ones to shapes such as loops and corners, and the last ones to whole digits or objects. A network with many hidden layers is called deep, which is where the name deep learning comes from. Modern networks are vast: OpenAI's GPT-3, described in 2020, had 175 billion weights, also called parameters.

How does a neural network learn?

By being wrong and adjusting, many millions of times. Training is the central method of machine learning, and for neural networks it runs in a loop:

1. Show the network an example and let it make a prediction.
2. Measure how wrong the prediction was, using a number called the loss.
3. Work out, for every weight, whether nudging it up or down would make the loss smaller. The method for doing this through many layers at once is called backpropagation; David Rumelhart, Geoffrey Hinton and Ronald Williams showed in 1986 that it lets networks with hidden layers learn.
4. Nudge every weight a small step in the helpful direction, then repeat with the next example.

Take the umbrella neuron, using the sigmoid, on a day when the sky is cloudy (1) but the forecast says no rain (0), and it rains anyway.

1. Its total is 1 × 0.5 + 0 × 2.0 − 1.5 = −1.0, which the sigmoid turns into 0.27: it is only 27% sure rain is coming.
2. The right answer was 1, so it was off by 0.27 − 1 = −0.73.
3. Each weight's nudge is the error times its input. The cloud input was 1, so its weight should rise; the forecast input was 0, so its weight played no part and stays put. The bias acts like an input that is always 1, so it rises too.
4. With a step size of 0.5, both the cloud weight and the bias move up by 0.5 × 0.73, about 0.37: the cloud weight goes from 0.5 to 0.87, and the bias from −1.5 to −1.13.
5. Next time, the same inputs give 0.87 − 1.13 = −0.27, which the sigmoid turns into 0.43.

One example moved the neuron from 27% to 43% sure, closer to the truth. Training a real network is this same step, repeated across billions of weights and millions of examples.

Is a neural network like a brain?

Only loosely. The name and the first ideas came from neuroscience, and both systems are made of many simple units whose connections carry strengths. Almost everything else differs.

	A brain's neurons	A network's artificial neurons
What a unit is	A living cell with branches, chemistry and electrical spikes	A line of arithmetic: multiply, add, apply a function
How signals travel	Timed pulses, in many directions, with feedback loops	Numbers passed layer to layer, usually in one direction at a time
How learning happens	Connections strengthen and weaken through many biological processes, still being worked out	Backpropagation adjusts every weight from a measured error, a method the brain isn't thought to use in this form
What it needs	Learns from relatively few examples, on about the power of a dim light bulb	Often needs huge datasets and large amounts of electricity

A network doesn't think like a person. It finds statistical patterns that link inputs to outputs, which can be remarkably useful and can also fail in ways a person wouldn't, such as misreading an image after a few pixels are changed. For how real neurons work, see how the brain works.

What are neural networks used for?

Most of the artificial intelligence in daily use since the 2010s runs on them: recognizing faces and speech, recommending videos, translating text, spotting tumors in scans and driving the chatbots of the 2020s. Those chatbots use a particular design called the transformer, which what GPT stands for explains. The idea is older than it looks: the history of AI shows neural networks falling in and out of favor for 70 years before computing power and data caught up with them.

Their limits come from the same design. They need lots of examples, they can be confidently wrong, and their reasoning is spread across billions of weights that are hard to inspect, which is why researchers call them black boxes.

Check yourself

• Without looking back, walk through what one artificial neuron does with its inputs, weights and bias.
• What do the hidden layers of a network do, and who decides what they look for?
• In your own words, how does training change a network, and why are the steps small?

Sources

• Explained: Neural networks — Larry Hardesty, MIT News, 2017
• What is a neural network? — IBM
• Learning representations by back-propagating errors — Rumelhart, Hinton and Williams, Nature, 1986
• Language Models are Few-Shot Learners — Brown et al., OpenAI, 2020