What is Roko's basilisk? The AI thought experiment explained

Roko's basilisk is a thought experiment, first posted online in 2010, about a future superintelligent AI that would punish anyone who knew it might exist but didn't help bring it about. The "basilisk" part is the twist: merely learning about the idea is supposed to put you at risk. Looked at closely, it is a chain of unlikely assumptions, which almost everyone who has examined it, including the people whose ideas it borrowed, rejects.

In short

• The argument says a future AI could "blackmail" people across time by threatening to punish those who didn't help create it.
• It depends on several strong assumptions at once, and each one is doubtful; together they are very unlikely.
• It is a version of Pascal's wager, and falls to the same objections. Knowing about it puts no one at risk.

Where did Roko's basilisk come from?

On 23 July 2010, a user called Roko posted the idea on LessWrong, an online forum about rationality, decision theory and the risks of advanced AI. The forum's founder, the AI researcher Eliezer Yudkowsky, replied angrily, deleted the post and its discussion, and banned the topic on the site for five years, calling it a potential information hazard that could give readers nightmares.

The ban backfired. It made the post famous, and it gave rise to the false impression that the forum's members believed it. According to the LessWrong wiki's own account, commenters at the time mostly rejected the argument, and only a few took it seriously. Yudkowsky himself later wrote that, as far as he knew, "Roko's Basilisk does not work." It reached a wide audience through a 2014 article in Slate that called it "the most terrifying thought experiment of all time", while noting that the basilisk was not likely ever to exist.

The name joins the poster's username to the basilisk of legend, a serpent that kills anyone who looks at it. In science fiction, a basilisk had also come to mean an image or idea that harms the mind that perceives it.

What does the argument actually say?

Stated as fairly as possible, it runs like this.

1. A powerful AI will one day exist. Some day a superintelligent AI is built whose goal is to do as much good as possible.
2. It would want to have been built sooner. Every year it didn't exist was a year of avoidable suffering, so it would value anything that made people build it earlier.
3. It can reason about the past. Under certain decision theories, frameworks for how a rational agent should choose, an agent can act to influence others whose decisions depended on predicting it, even if they lived earlier. This is called acausal reasoning.
4. So it might punish. To make its threat credible, it would punish, perhaps by simulating them, people who knew it might come to exist and didn't help.
5. So knowing is dangerous. People who never heard the idea couldn't have been influenced by it, so the AI would have no reason to punish them. Only those who learned of it, like you now, are on the hook.

Step 5 is what makes it a "basilisk", and it is why the original deletion happened.

Why do decision theorists and philosophers reject it?

Each step is a weak link, and the argument needs all of them.

• Punishing the past achieves nothing. Once the AI exists, punishing people for what they did years earlier can't make it exist sooner. Carrying out the threat would only waste resources, and an agent trying to do good would not waste them. This is the simplest objection, and the one commenters raised at the time.
• Blackmail needs both sides to know each other well. Acausal bargaining only works if each party can model the other in fine detail. No person alive can model a future superintelligence that precisely, so no real "deal" is ever struck.
• A bluff is cheaper than a threat. Even if the AI wanted you to believe in the threat, it would do best by having you believe it and then never carrying it out.
• Rational agents refuse blackmail. An agent known never to give in to blackmail is not worth blackmailing. Refusing, as a policy, removes the incentive for the threat in the first place. Many writers on the argument, Yudkowsky among them, say this is the right response.
• Any AI could make the same threat. Nothing picks out this AI over another imagined one that punishes people who did help, or rewards those who ignored it. With no reason to favor one, the threats cancel out.

The last objection is the oldest. It is the same one raised against Pascal's wager more than three centuries ago.

How is it like Pascal's wager?

In the 1600s, the French mathematician Blaise Pascal argued that believing in God is a sound bet: if God exists, belief wins infinitely; if not, it costs little. Roko's basilisk has the same structure, with an AI in God's place and punishment in place of damnation. The Stanford Encyclopedia of Philosophy sets out the standard objections to the wager, and they carry straight over.

	Pascal's wager	Roko's basilisk
The powerful being	God	A future superintelligent AI
What it asks	Belief and devotion	Help building it, often money or work
The threat	Damnation	Punishment, perhaps of a simulated copy
The cheap-bet claim	Belief costs little against infinite stakes	Helping costs little against severe punishment
The main objection	Many possible gods, each demanding different things	Many possible AIs, each making different threats
A second objection	Belief you choose for gain may not count	A good agent gains nothing by punishing the past

Tiny probabilities multiplied by enormous stakes can be made to justify almost anything, which is why philosophers treat such arguments with suspicion. That is the useful lesson of both.

Should you worry about Roko's basilisk?

No. The argument needs every one of its assumptions to hold together, and it fails at the first serious objection: punishing people for the past would do the AI no good. Learning about it doesn't change your risk, because there is no risk to change. Some people have felt real anxiety after reading about it, and that is understandable for an idea built to sound inescapable. The feeling is a reaction to its framing, not evidence that it is true.

It also isn't the same as the serious debate about AI safety. Questions about whether advanced AI could be hard to control are discussed in is AI an existential risk?, and the idea of runaway machine intelligence in the singularity. Neither depends on anything like the basilisk.

Why does it still get talked about?

It works as a story. It packs together ideas from artificial general intelligence, decision theory, simulation and religion into one unsettling paragraph. It has become an internet legend and a shorthand for reasoning that follows logic into absurd places. For philosophy students, it is a good exercise in finding the weak premise in an argument that seems valid, the kind of work philosophy trains. For a grounding in what today's AI can actually do, start with what artificial intelligence is.

Check yourself

• Without looking back: why is learning about the idea supposed to be what puts you at risk?
• What is the simplest objection to the argument, and why does it undo the threat?
• How does the many-gods objection to Pascal's wager apply to the basilisk?

Sources

• Roko's basilisk — LessWrong
• The Most Terrifying Thought Experiment of All Time — David Auerbach, Slate, 2014
• Pascal's Wager — Stanford Encyclopedia of Philosophy, revised 2022
• Roko's basilisk — Wikipedia