A mechanism built in 2014 to help machines translate English into French became the core of large language models, yet Llion Jones, who worked on the Transformer, now says it is far from all of intelligence.
In 2014, an intern at a Montreal lab was working on English-to-French machine translation. Models of the time forced a whole sentence into fixed memory before translating, so long sentences came out wrong.
The intern, Dzmitry Bahdanau, had the model look back across the source sentence for each word it produced, picking out the most relevant words. That was attention, and long sentences came out right too.
A single pronoun opens the problem
Pronouns are the hard part. In 'the animal didn't cross the street because it was too tired', 'it' means the animal, so French needs the masculine il. In 'because it was too wide', 'it' means the street, so French needs the feminine elle.
Attention lets each word gather information from the others in the sentence. Through trillions of corrections in training, the model taught itself links like 'animals get tired, streets can be wide', and nobody spelled them out.
In 2017 a Google team pushed attention further in the Transformer, and large language models such as ChatGPT followed.
Predicting the next word of a translation is just a subtask of predicting the next word, period. A model that reads the whole internet and keeps guessing words has to learn translation, arithmetic and common sense. Sakana AI's Llion Jones argues this may be the closest definition of intelligence we have.
The inventors are not convinced
But Jones now thinks attention is not all you need. The brain has never read the whole internet and burns far less energy, yet it converses fluently.
Human thought is not purely linguistic either, drawing on senses, experience and memory. Jones says: 'The current paradigm won't solve this efficiency problem. You seem to have to jump to something very, very different.'
Machines have already translated 'it' correctly, which is impressive; whether they understand it is still argued over by the people who built them.
Why it matters
How close AI is to intelligence depends on where its bottleneck sits: if it only needs more data and compute, scaling is the road; if, as Jones argues, it needs a different paradigm, then faster scaling does not mean faster progress toward intelligence.
Curated from high-quality sources, with concise summaries and key takeaways.
