Computer scientist Moritz Hardt argues that AI benchmarks' absolute scores have long lost their meaning, yet the rankings between models remain surprisingly reliable — though even that is now failing in the age of large language models.
An AI benchmark is like an exam: set a paper, see who scores highest. But there is a catch — the test-takers can practise on past papers.
A ruler with a warped scale
A benchmark should be a thermometer: you know if something is hot or cold at a glance. In practice, people train models directly on the test set, which is like doing the real exam papers before the exam.
Change the test set slightly, and a model scoring 90% can drop to 73%. The absolute number means almost nothing.
Rankings turned out to be reliable
The strange part is that if everyone uses the same training data, rankings stay very stable. The model with 90% usually still beats the one with 80% on a new dataset.
This is the main reason machine learning has escaped a scientific crisis so far — though nobody can fully explain why.
Large models broke the pact
With large language models, nobody knows what training data went in. Some models cram for the exam like high schoolers, and their scores look amazing.
After a new model comes out, scores on old benchmarks are often fantastic, while new benchmarks reveal much lower numbers. This is the accepted practice known as 'benchmaxxed'.
Hardt's suggestion: give all models 'altitude training' — let each of them train a little on benchmark-relevant data before testing, so the cramming is fair, and rankings become meaningful again.
Why it matters
Benchmarks are the main way the AI industry shows off capability to the outside world, so unreliable scores make 'surpassing human experts' claims in papers and product launches hard to trust. If Hardt's altitude training were adopted, comparisons between models would become meaningful again — but 'how good is it really' might get even harder to answer.
Curated from high-quality sources, with concise summaries and key takeaways.
