Ai2 opens ToyBlocks robot sim with 85K tabletop demos

Ai2's ToyBlocks arrives as robotics researchers rethink how robots learn. Most robots are trained today by people steering them through a task again and again, which is slow and costly; getting 10 times more examples means paying for far more hours of human labor. Florian Shkurti, an Ai2 team member and visiting research scientist from the University of Toronto, said of this traditional approach: "We don't think it's scalable." ToyBlocks instead creates computer simulations of the real lab setup Ai2 has in Seattle, and the software automatically generates thousands of different practice runs so the AI model learns from simulation before researchers test how those skills carry over to the real arm.

The approach gained momentum this year. Until recently, many roboticists doubted that simulation data was high quality or abundant enough to be effective. That changed, said Russ Tedrake, a robotics professor at MIT and co-founder and CEO of Walden Robotics, because powerful new AI models from OpenAI and Anthropic made it possible to build detailed simulated scenes on request. "Longtime skeptics are now suddenly hiring simulation engineers," he said. Ai2, long known for open AI research, is newer in robotics; last year it hired Dieter Fox from Nvidia's Seattle robotics lab, and its team of about 15 started as just Fox and two others. Fox said the institute is "way too small" to advance the field alone and needs help from the scientific community.

Ai2's own tests show the challenge. A leading robot AI model from San Francisco startup Physical Intelligence, given 1,000 simulated examples of stacking a green block on a red one, succeeded almost every time on the same patch of table but pretty much failed on another part. "For a person, this would be trivial," Fox said, but the robot had learned to stack blocks in one spot, not to stack blocks. ToyBlocks is a place to test whether more data fixes this, though Fox added that quantity isn't everything: "What really matters is the right kind of diversity in that data."