TL;DR
Building a superintelligence that can improve itself endangers both human survival and ordinary people's control over their own future, so it should be halted by public decision rather than a handful of companies.
The author says he spent five years hoping not to write this piece. On 9/11 he had just started eighth grade. His mother picked him up, they pulled down every blind and watched the news, and he already knew his small world was gone for good.
Twenty-five years later, the thing running through his head is Flight 93.
On that plane, the passengers held a vote
The hijackers meant to fly into the Capitol. Once the passengers understood, the first thing they did was vote — to storm the cockpit and take their fate back into their own hands.
They fought with boiling water and a food cart, and the plane went down in a Pennsylvania field. A few seconds the other way and it could have ended differently.
The author's point: ordinary Americans have been dropped into the same situation, except this plane is parked in California.
He admits he has been wrong before
He once thought AI was just a tool — a bad writer, a poor researcher — and doubted it could keep getting stronger. Every time he settled into that view, a whole new way of scaling it appeared.
From pre-training, to human feedback, to tools and scratch pads, to reasoning models, and now to herding millions of AIs into a swarm that thinks together.
By one count, the messages passing through the swarm during a single problem would take about 10,000 years to read aloud. The run took 88 hours.
The people flying it are not alright
The harder problem is who is at the controls. The early AI safety crowd founded DeepMind, OpenAI and Anthropic — and pushed each other into becoming the fiercest competitors, driving the race toward a more dangerous finish.
Some genuinely believe they are building a machine god; some say they are racing China. The author asks: win, and then what?
He borrows a line: he would sooner be governed by the first two thousand names in the Boston phone directory than by the first two thousand employees of OpenAI.
The one line he wants to leave behind
Building a machine smarter than us, one that can upgrade itself, trips two wires at once: human survival, and ordinary people's grip on their own destiny.
He has gone to Washington to argue for banning both. It would not cost us the future — today's AI, merely refined, is still enough to cure cancer and reach Mars.
In one line: let the passengers raise their hands before the people in the cockpit decide where the plane is going.
Curated from high-quality sources, with concise summaries and key takeaways.
