I trained a small transformer in 1.5hrs and it beats many LLMs

In 1.5 hours on a single RTX 5090, the author trained a small transformer from scratch and scored 44% on ARC-AGI-1 for 67 cents, beating many large models. Top researchers took notice.

Solving hard puzzles with the fewest samples

ARC is a reasoning benchmark of puzzles where each one follows a different rule. Humans find them easy but AI has long struggled. The author's bet: sample efficiency — learning new things from very few examples — is AI's biggest problem.

His approach is simple: convert inputs and outputs to token sequences, train a small transformer from scratch at test time, with a few tricks like per-task embeddings and 3D positional encodings. It is like teaching the model to learn on the job rather than memorizing data.

Cheap yet shockingly effective

He adopted a modern architecture and optimizer, and cut much of the data augmentation — which helped scores. A quirk: he no longer trains on input tokens, only outputs, and test loss is worse, yet scores are better. No idea why.

Ablations show the biggest win comes from good representations (3D RoPE + task embeddings); removing either drops the score to 25%. So cramming in more data is not the only way.

Pushing back against critics

Some called his method 'training on test set' and thus cheating. The author says that is false: it is transductive reasoning, and the output labels are still hidden. To him, learning from evaluation puzzles is the spirit of metalearning.

He also argues the ARC benchmark should ban offline training to force on-the-fly learning, making it a true test of sample efficiency. He is confident he can reach 65% with his approach, no magic or giant datasets needed.