Former OpenAI researcher Nick Cammarata published a long post arguing generative AI models can in fact be understood, overturning the industry's core belief that neural networks are impenetrable.
Silicon Valley spent 24 hours obsessed with a 26,000-word blog post. Its author, Nick Cammarata, claims generative AI models are not black boxes—there is a clear path to understanding them. If true, the industry's basic assumptions would be upended.
The man who once proved they were unreadable now says they're legible
Cammarata earned his reputation at OpenAI helping establish that neural networks are nearly impossible to understand. Now at Goodfire, he argues the entire field has been wrong.
Neural networks are layers of neurons. Earlier work found one neuron can fire for unrelated things: cat faces, cat legs, car fronts. The assumption was that neurons are stuffed with arbitrary, mixed functions.
Cammarata now says that neuron was actually responding to V-shapes—cat faces have a V, car fronts have a V, even flames and jets of water. The network is not a tangled knot; it's clean.
A glimmer of hope, or another round of overpromising
The implication is direct: if we can read a model's internal mechanisms, we can stop it going haywire. OpenAI researchers called it the greatest optimism in years.
But DeepMind's Neel Nanda poured cold water: labelling every neuron's function is not the same as understanding the whole network. You know all your friends, but you can't predict whether they'll get along at your birthday party.
The post isn't peer-reviewed, and its main evidence comes from vision models, which are simpler than language models. And understanding the internals doesn't automatically make AI controlable.
The real black boxes may be OpenAI and Anthropic themselves. They say they don't fully know what they're building, yet keep pushing it further. Without regulation, only the companies can decide to stop.
Why it matters
If AI can be understood like a watch taken apart, the biggest fear of losing control eases by half; but it also removes the excuse of 'we didn't know it would do that'—the responsibility would fall squarely back on the builders.
Curated from high-quality sources, with concise summaries and key takeaways.
