Where will robot training data come from?How to Train Your Robot

The breakthroughs in AI language models came from vast amounts of training data, but robots have no ready-made data to consume and are stuck in a data gap.

Language models have advanced so quickly in recent years because they were fed enormous amounts of text scraped from the internet. GPT-3 trained on roughly 300 billion tokens; the total stock of quality public text data is estimated at around 300 trillion tokens.

Robots don't have that meal

Robots need action data: someone demonstrates a task like pouring water or folding a shirt, the robot's joint angles and camera feeds are recorded, and the model is trained to imitate. This is behavioural cloning.

The dominant collection method is teleoperation, with a human controlling the robot. The data quality is high, but it's expensive and the teleoperator has no sense of touch, so much of the data is unusable. Letting robots gather their own data in the real world sounds appealing, but few robots are doing real work today and the data would be highly repetitive.

Simulation is cheap but not faithful

Another approach is simulation: build a virtual world and train thousands of robots in it at once, cheaply and quickly. Figure taught its humanoid to walk this way.

But simulation has a fatal flaw: it's imperfect. Locomotion is fairly easy to simulate; grasping soft or deformable objects is not. OpenAI trained a robotic hand to solve a Rubik's Cube with the equivalent of 13,000 simulated years, and still hit only 20% success on the hardest scrambles.

Recording people as textbooks

Some companies record humans directly: volunteers wear head-mounted cameras while doing chores, and the first-person video is mapped onto robot actions. Figure and Shift are doing this, paying people to do housework while recording it. Much cheaper, but human motion doesn't map perfectly onto robot capabilities.

A 100,000-year gap

By one estimate, existing robot training data amounts to a few billion text tokens, while GPT-4 was trained on the equivalent of 685 million hours of text. That's a gap of roughly a thousandfold. At current robot data prices of $60–150 per hour, closing the gap would cost tens of billions to over a hundred billion dollars.

Whether we actually need that much is an open question. Maybe general-purpose AI models need only a little robot data to become capable controllers; maybe real-world manipulation is fundamentally harder than text and requires far more.