A new study finds AI needs far more tokens to process some languages than English, making non-English users pay more, get shorter context and wait longer.
Talk to AI in anything but English and you may quietly pay extra. A research coalition called CLEAR published a study last month and named the surcharge the 'tokenizer tax'.
One meaning, more pieces
Models do not read words; they read tokens, the small chunks text, images and audio get cut into. The more pieces, the more work.
Among the 20 most common online languages, the priciest used 1.5 to 5 times more tokens than the cheapest. Across 200 languages, the gap in one model reached 17 times.
It is not just the rare ones. Burmese, Tibetan, Khmer and other scripts the tokenizer rarely saw fare worst in nearly every model.
The bill is not the only cost
More tokens means slower replies and less room in the context window. The co-founder of an Indian AI fund put it this way: a good tutor remembers where a student went wrong, but in some languages that history eats the window in a few bites.
CLEAR did the arithmetic: a startup in Bengaluru pays a surcharge on every request, its rival in Austin does not.
Some point out token prices are falling fast, so the direct cost should ease. The real fix is for labs to design tokenizers that take these languages and scripts seriously.
Why it matters
AI is sold as a leveller across languages, but the tokenizer quietly turns everyone outside English into a second-class user — not a natural market preference, but a design choice about who was left out of the room.
Curated from high-quality sources, with concise summaries and key takeaways.
