What is the singularity? The idea of runaway AI explained

The singularity, in technology, is the hypothetical point at which machines become more intelligent than people and start improving themselves faster than humans can follow, so that what comes after can't be predicted from what came before. The idea grew from the statistician I. J. Good's 1965 "intelligence explosion" and was named and popularized by Vernor Vinge and Ray Kurzweil. Whether it will happen at all is disputed. (In physics, a singularity is something else: the point at the center of a black hole where known laws break down.)

In short

• The core argument is a loop: a machine smarter than us would be better than us at designing machines, so it could build a smarter one, and so on.
• Its supporters disagree among themselves about when and how; dates have been offered since the 1960s and several have already passed.
• The main objections are that progress may get harder, not easier, at each step, and that intelligence may not be one thing that can be turned up like a dial.

Where does the idea come from?

The first hint is a remembered conversation. In 1958, the mathematician Stanislaw Ulam recalled talking with John von Neumann about "the ever accelerating progress of technology", which gave "the appearance of approaching some essential singularity in the history of the race." Von Neumann meant a point beyond which human affairs, as people knew them, could not continue.

The argument itself came from I. J. Good, a British statistician who had worked with Alan Turing as a codebreaker during the Second World War. In his 1965 article "Speculations Concerning the First Ultraintelligent Machine", he wrote: "Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion'." Thus, Good concluded, "the first ultraintelligent machine is the last invention that man need ever make," and he added a condition often left off: "provided that the machine is docile enough to tell us how to keep it under control."

The science fiction writer and mathematician Vernor Vinge gave it its name. He introduced the term "singularity" in a 1983 magazine article and made it famous with his 1993 essay "The Coming Technological Singularity", which opened: "Within thirty years, we will have the technological means to create superhuman intelligence. Shortly after, the human era will be ended." He borrowed the word from physics: like the center of a black hole, it is a point past which our models stop working.

The inventor Ray Kurzweil brought the idea to a mass audience with his 2005 book The Singularity Is Near. He argued that computing power and other technologies grow exponentially, by what he called the "law of accelerating returns", and predicted that AI would reach human-level intelligence by 2029 and that the singularity, when people merge with their machines, would arrive around 2045. He repeated both dates in a 2024 sequel, The Singularity Is Nearer.

What exactly do proponents claim?

There are really three claims, often run together. The philosopher David Chalmers, who set out the argument carefully in 2010, separates the steps:

1. There will be AI at least as intelligent as an average human, which links to the idea of artificial general intelligence.
2. Soon after, there will be AI more intelligent than the most intelligent human.
3. Soon after that, there will be superintelligence: AI as far beyond the brightest person as that person is beyond a mouse.

Each step is meant to follow from the one before, unless something stops it. Chalmers calls those possible stoppers "defeaters", such as a disaster, a lack of will, or a deliberate decision to halt. Proponents also differ on what the singularity looks like. Kurzweil's version is one of steadily accelerating change, ending in humans merging with their technology. Vinge's stresses the "event horizon": a future we can't see past. Good's is the explosion itself.

How fast would self-improvement have to be?

The argument stands or falls on one question: when an AI makes itself smarter, does the next improvement get easier or harder? Chalmers gives an example of the hard case: a 10% increase in intelligence that leads to a 5% increase, then 2.5%, and so on. Here is what three different patterns do over five rounds of self-improvement, starting from a score of 100:

Round	Gains halve each time (10%, 5%, 2.5%…)	Gains stay at 10%	Gains grow by half each time (10%, 15%, 22.5%…)
1	110	110	110
2	115.5	121	126.5
3	118.4	133.1	155.0
4	119.9	146.4	207.3
5	120.6	161.1	312.2
After 50 rounds	About 121, and stuck there	About 11,700	Runaway

With shrinking gains, the system levels off barely above where it started: no explosion. With steady gains, growth compounds into something enormous, given enough rounds. With growing gains, it runs away. The numbers are illustrative, not measured. Nobody knows which column describes real AI, and the answer may differ between hardware, software and the experiments that research needs.

What are the main objections?

That progress gets harder. In 2011, the Microsoft co-founder Paul Allen and the computer scientist Mark Greaves argued in MIT Technology Review that the "law of accelerating returns" is not a law of nature. They proposed a "complexity brake": the deeper science digs into a system like the brain, the more complexity it finds, so understanding slows rather than speeds up. Chalmers, who takes the argument seriously, calls diminishing returns "perhaps the most serious structural obstacle."

That intelligence isn't one dial. The argument treats intelligence as a single quantity that can be measured and increased. Critics answer that it is a bundle of different skills, and that being better at designing chips doesn't make a system better at everything else.

That thinking isn't enough. Even a very capable system has to run experiments, build factories and wait for results in the physical world, which doesn't speed up just because a mind does. Energy, materials and computing power all set limits.

That forecasts keep slipping. Chalmers notes that Good expected an ultraintelligent machine by 2000, and Vinge greater-than-human intelligence between 2005 and 2030. Several such dates have come and gone.

Still argued

Whether a singularity is likely, possible, or even coherent is unresolved. Proponents say the logic is simple and only the timing is uncertain, and point to fast recent progress in AI. Critics say each step of the argument hides an assumption about easy, compounding improvement that has never been shown. A further split runs through those who take it seriously: some expect it to end scarcity and disease, while others see it as the root of the risk discussed in whether AI is an existential risk. Strange offshoots of the idea, such as Roko's basilisk, come from the same debates.

The idea matters even if it never happens, because it shapes how many people in artificial intelligence think about what they are building. It is a different thing from the singularity at the heart of a black hole, though Vinge chose the word for the likeness.

Check yourself

• Without looking back, explain Good's "intelligence explosion" in two sentences.
• Why does it matter whether each round of self-improvement gets easier or harder?
• What is the "complexity brake", and which claim of Kurzweil's does it answer?

Sources

• The Singularity: A Philosophical Analysis — David J. Chalmers, Journal of Consciousness Studies, 2010
• Speculations Concerning the First Ultraintelligent Machine — I. J. Good, Advances in Computers, 1965
• The Coming Technological Singularity: How to Survive in the Post-Human Era — Vernor Vinge, 1993
• Paul Allen: The Singularity Isn't Near — Paul Allen and Mark Greaves, MIT Technology Review, 2011
• The Singularity Is Nearer — Wikipedia