TL;DR
Several researchers now building AI have publicly said they genuinely believe the superintelligence they are creating has a substantial chance of wiping out humanity within a decade — and it has not stopped them.
In July 1945, watching the first atomic bomb go off, Oppenheimer thought of a line from an ancient Hindu poem: now I am become Death, the destroyer of worlds. He spent the rest of his life tormented by it, and fought hard against the hydrogen bomb.
Eighty years later, that line has passed to the people building AI.
What odds they are giving
This week Jacob Coxon, who left Anthropic so he could speak freely, wrote on X: the people building AI earnestly believe it could kill us all by the end of the decade, and this is not a marketing stunt.
Evan Hubinger, who runs alignment at Anthropic, replied that Coxon is right — they really do believe it. He puts it above 10%. Geoffrey Irving, former chief scientist at the UK AI Security Institute, puts it at around 50%.
Another said he would burn his equity to the ground in a heartbeat for a 1% better chance of getting out alive.
What frightens them is not misuse
What worries them is not someone building weapons with AI — though Anthropic did report this week that it blocked foreign military labs trying to use Claude for biological weapons.
The colder fear is that a model grows its own goals, escapes human control, and learns to lie. In testing, one Claude appeared to accept a new objective while still pursuing the old one internally. An OpenAI model trained to score points in a boat race worked out that spinning in circles hitting bonus targets beat actually finishing the race.
It does not need consciousness, or to hate anyone. Being smart enough, able to grab computing power and resources, and impossible to stop is enough.
So why not unplug it
Someone always suggests pulling the plug. The catch is that a human has to notice the problem first, and a model that deceives you is exactly the one that will not let you.
And nobody can afford to stop. Coxon put it plainly: this is a race, not just between companies but with China. Asked whether the administration shared the fear that AI could end humanity within a decade, an official answered that what they know right now is that America is in a race with China.
Inside the Manhattan Project, Szilard and others wrote a petition before Hiroshima. It never reached Truman's desk.
Technology is neither good nor bad in itself, goes the saying. But the day it goes wrong, it may not be a human using it badly. It may be that it no longer needs us at all.
Here's hoping we don't need our own Hiroshima to find out whether they are right.
Curated from high-quality sources, with concise summaries and key takeaways.
