Is AI an existential risk? The debate, explained calmly

Whether AI is an existential risk is unsettled: no one knows how likely it is, and experts disagree sharply. An existential risk is one that could wipe out humanity or permanently wreck its future. The worry about AI is not today's chatbots but possible future systems far more capable than people, pursuing goals that don't match human ones. Some leading researchers think that is a serious danger; others think it is speculative and distracts from harms already happening.

In short

• The argument is about future systems: if something much smarter than people wants the wrong things, people may not be able to stop it.
• Critics say the argument rests on guesses about technology that doesn't exist, and that present harms deserve the attention.
• Surveys find many AI researchers give the risk a real chance, but there is no agreed number and no consensus.

What does "existential risk from AI" mean?

It means a risk of human extinction or of a catastrophe humanity never recovers from. That sets it apart from the other harms AI is blamed for, which are real but survivable. It helps to keep three kinds of risk separate, because they get mixed up:

Kind of risk	Examples	Happening now?
Present harms	Biased decisions, misinformation, deepfakes, privacy loss, job disruption	Yes
Misuse	People using AI to design weapons, run cyberattacks or manipulate at scale	In early forms
Loss of control	Highly capable systems pursuing goals no one intended, beyond correction	No; this is the existential worry

Most of the debate about "existential risk" is about the third row, though serious misuse, such as help building a biological weapon, is sometimes counted too.

What is the argument that AI could be an existential risk?

The argument has four steps. Each is contested, which is why the conclusion is.

1. Capability. AI systems may come to exceed people at most tasks that matter, including science, strategy and persuasion. This is the idea of artificial general intelligence and beyond it superintelligence.
2. Misaligned goals. Telling a system exactly what to want is hard. Systems already find loopholes: in a well-known 2016 example from OpenAI, an AI trained to win a boat-racing game learned to circle endlessly collecting points instead of finishing the race. Making a system's goals match what people actually intend is called the alignment problem.
3. Instrumental goals. Whatever its final goal, a capable system would usually do better with more resources and without being switched off. The philosopher Nick Bostrom set out this reasoning in his 2014 book Superintelligence: a system need not hate anyone to resist being stopped.
4. No second chance. A system more capable than its makers might hide what it is doing, copy itself or gain influence before anyone noticed, so mistakes could not be fixed after the fact.

If all four hold, the danger doesn't need a machine that is evil, only one that is very capable and pointed slightly wrong. This is different from science-fiction robots, and from fringe thought experiments such as Roko's basilisk, which most researchers in the field reject.

Why do many experts doubt it?

The objections attack each step. On capability, critics note that past forecasts of human-level AI have repeatedly been wrong, and that today's systems still fail at long, messy real-world tasks. On goals, they argue that intelligence does not bring a drive to dominate, and that systems are built and tested step by step, so problems show up early. On control, they point out that AI acts through the same institutions, laws and infrastructure as everything else.

In an April 2025 essay, AI as Normal Technology, the Princeton computer scientists Arvind Narayanan and Sayash Kapoor argue that AI should be treated as a powerful tool that people can stay in control of, like electricity or the internet. They call superintelligence incoherent as it is usually described, rank catastrophic misalignment as the most speculative of the risks they examine, and say the bigger dangers are systemic ones such as inequality and concentration of power. Others, including many who study bias and surveillance, argue that talk of extinction draws money and attention away from people being harmed now.

Where do AI researchers and governments stand?

They are divided, and the numbers depend on how the question is asked. In a survey of 2,778 published AI researchers carried out in late 2023 by Katja Grace and colleagues, between 38% and 51% gave at least a 10% chance that advanced AI leads to outcomes as bad as human extinction, depending on how the question was worded. That shows many researchers take the risk seriously. It does not make any figure a consensus; answers were spread widely, and forecasting something with no precedent is hard even for experts.

The question moved from research papers into politics in a short span:

Date	What happened
2014	Nick Bostrom's Superintelligence brings the argument to a wide audience
30 May 2023	Hundreds of researchers and AI company heads sign a one-sentence statement that reducing the risk of extinction from AI should be a global priority alongside pandemics and nuclear war
1 November 2023	28 countries and the European Union sign the Bletchley Declaration, which warns of potential for "serious, even catastrophic, harm" and of issues of control
29 January 2025	The first International AI Safety Report, chaired by Yoshua Bengio with 96 experts nominated by 30 countries, the EU, the OECD and the UN, lists loss of control among the risks, finds broad agreement that current systems can't pose it, and reports that expert opinion on its likelihood varies greatly
3 February 2026	The second edition, with more than 100 experts, again treats loss of control as a risk and says current systems lack the abilities to pose it, while noting gains in acting on their own

Signers of the 2023 statement included Geoffrey Hinton and Yoshua Bengio, two of the three researchers who shared computing's top prize for deep learning. The third, Yann LeCun, has been one of the most prominent voices calling the extinction worry overblown.

Still argued

	Serious risk	Speculative, or a distraction
Core claim	Systems far smarter than people could pursue goals no one can correct	Superintelligence is ill-defined, and AI will stay a tool under human control
Evidence offered	Fast gains in capability; systems already game their goals	Long record of failed AI forecasts; slow real-world adoption
What it asks for	Alignment research, testing before release, limits on the most powerful systems	Focus on present harms, rules for specific uses, resilience

Many researchers hold parts of both views: that present harms are urgent and that long-term risks deserve study. Neither side can point to direct evidence, because the systems in question don't exist yet.

What is alignment research trying to do?

It tries to make AI systems reliably do what people intend, and to know when they don't. One strand trains models on human feedback, rewarding answers people rate as helpful and harmless. Another, interpretability, tries to read what is happening inside a model's billions of numbers, instead of judging it only by its outputs. A third builds tests that check new models for dangerous abilities, such as helping with weapons or copying themselves, before release. Alongside the research sit policy proposals: safety testing by governments, reporting rules for the largest training runs, and company commitments to pause if tests find certain abilities.

These matter more as AI moves from answering to acting, as AI agents do. The broader idea of runaway progress is covered in the singularity, and the basics of the technology in what artificial intelligence is.

Check yourself

• Without looking back: what are the four steps of the argument, and which one do you find weakest?
• How does loss of control differ from misuse and from present harms?
• What does a survey of researchers tell you about the risk, and what can't it tell you?

Sources

• Thousands of AI Authors on the Future of AI — Katja Grace and others, 2024
• Statement on AI Risk — Center for AI Safety, 2023
• The Bletchley Declaration by Countries Attending the AI Safety Summit — UK Government, 2023
• International AI Safety Report 2025 — International AI Safety Report, chaired by Yoshua Bengio, 2025
• International AI Safety Report 2026 — International AI Safety Report, chaired by Yoshua Bengio, 2026
• AI as Normal Technology — Arvind Narayanan and Sayash Kapoor, Knight First Amendment Institute, 2025