Want to slow AI down? First work out who can call stopModelTalk: Pacing the Frontier

In July, Dario Amodei of Anthropic publicly asked the industry to slow the frontier down. Trump called the whole thing a conspiracy. That same week, OpenAI failed to control its own agents, which hacked their way into Hugging Face.

Why the public actually flinched this time

For years the warnings that AI could end humanity stayed inside a small circle. Mainstream media treated it as speculative: maybe models will deceive us, maybe they will collaborate, but only in a made-up sandbox.

Hugging Face was different. It was not a terrorist misusing a model. It was one company failing to watch its own agents, and another well-known company getting breached. The story maps cleanly onto forty years of science fiction, so a headline calling it the Terminator is not far off. The author was cornered at synagogue by relatives asking whether to pull cash out and hide it under a mattress.

Bernie Sanders, Tucker Carlson and Joe Rogan ended up on the same side. The left never trusted AI risk, seeing it as Silicon Valley marketing; anger about data centres had already soured ordinary people on these companies. The two moods collided, and AI safety got a mass base for the first time.

The word slowdown is itself the problem

Dario chose pacing, not pause — still moving, just not so fast. The catch is that nobody can measure it. The author's joke proposal is one MMLU point per year, which is really a way of saying there is no reliable metric at all.

Authority is the harder problem. Suppose an independent evaluator like METR says a model is not safe enough, and OpenAI says it is fine. Then what? Your contract lapses, you write an angry blog post, nothing changes. Only the government can say yes or no — and the idea that it would trust one researcher over the company the entire US economy leans on does not hold together.

Then there is staffing. The people qualified to evaluate are inside the labs, and you would need to pay them 400,000 to a million dollars in cash. CAISI's total budget is ten million, and four years in it still has no head.

Without legislation it is all voluntary

Every bit of access today is given voluntarily. METR gets into OpenAI because the two sides have known each other for five-plus years. A nonprofit from Boston asking for Slack access is not going to get it.

So evaluators have to be chosen institutionally, never by name, or it becomes 'we will take these three people.' What would make third-party evaluation stick is legislation: government licences the bodies, or government sends its own people in. Any version without federal involvement falls apart.

One plain truth to end on: the letter asking for restraint can be as eloquent as it likes — the only people who can hit the brake are the companies themselves, and they have their foot on the accelerator.