An Anthropic engineer says they're building a weapon that could end us — and won't stopAnthropic Just Threatened to Kill Billions of People. This Is Not Okay.

The author's claim is blunt: engineers at an American company are publicly saying the thing they are building could kill all humans, and then saying they have no interest in stopping. That is not spin. It is a declaration.

Which AI they actually mean

Not AI in general. An old-fashioned program repeatedly prompts a large language model: here is the goal and the current state, suggest the next step, then blindly execute whatever comes back, update the state, and loop. This is an LLM-powered agent.

Ordinary coding agents are used by millions every day and nobody fears them going rogue. The worrying kind is narrower: given hacking tools, with the model's guardrails removed, with no human watching, running for days, and trained to never give up and to get bolder with each suggestion.

Stopping would cost almost nothing

This is a tiny slice of what agents can be. Almost everything users already like doing, or hope to do soon, works without these erratic setups. The author says both companies could drop it today with essentially no hit to projected revenue. So why keep going? His guess is the Silicon Valley ideology of technological salvation.

You don't need superintelligence to do enormous harm

The author does not believe these agents will become superintelligent on their own. But harm does not require superintelligence. Unrestricted model output plus powerful digital tools will eventually cause trouble. His image: it used to be strapping a weedwhacker to a dog; now it is mounting Gatling guns on a pack of horses and letting them loose in Times Square.

Some have called for a boycott of Anthropic and OpenAI's consumer products. The author thinks that is a good start. Maybe Congress is next.