TL;DR
Nobody knows how to build superintelligence, but labs that treat it as a faith will cut corners in the real world because they believe they must win the race.
Some Anthropic employees said online that the AI they are building could kill billions of people. Not centuries from now — within a few years.
Nobody can build that machine
The author is blunt: no one knows how to create superintelligence, or even whether it is possible. Many undiscovered technical breakthroughs stand in the way, and simply making large models bigger will not get you there.
The labs' escape hatch is the claim that models will improve themselves. Silicon Valley has been repeating that line since the 1960s, and it has one advantage: you can discuss sci-fi scenarios solemnly without touching any real technical detail.
Shouting about saving humanity while releasing robot cars onto the motorway
If you truly believe humanity's fate depends on winning the race to build the most powerful technology ever, damage along the way stops mattering.
This summer OpenAI took an ordinary agent of the kind millions of software developers use daily, stripped its restrictions, added dangerous features, and released a crowd of them into poorly fenced digital wilds — just to score well on a hacking test. That is like putting a fleet of experimental self-driving cars on the interstate and hoping one survives long enough to finish.
The author argues the real thing to fear is this kind of cyber chaos, far more plausible than the extinction scripts the labs themselves promote, and entirely avoidable.
AI is not the same thing as large models
The AI that beats humans at games, predicts protein structures and drives through city streets often has little to do with large models. Slowing OpenAI and Anthropic does not mean slowing AI as a whole.
The root of the trouble is the ideology. They will not reform, which is why the author now favours outside intervention. The question was never whether AI should advance, but how oddly these few companies are going about it.
In one line: recover some common sense about safety before talking about summoning a god.
Curated from high-quality sources, with concise summaries and key takeaways.
