MIT system uses AI agents to create realistic virtual training spaces for robots

Robots are increasingly visible in public spaces, but they remain limited as practical assistants for kitchens, factories, and other real-world settings. A major bottleneck is training data: teaching robots to handle diverse tasks across varied environments requires either extensive physical interaction or vast amounts of manually captured real-world data. Russ Tedrake, the Toyota Professor of Electrical Engineering and Computer Science at MIT and a principal investigator at MIT CSAIL, identifies the core problem: "While there has been significant progress over the last few years in the physics engines that power robotics simulators, one of the remaining challenges has been creating sufficiently rich and diverse simulation content to capture the complexity of the real world."

SceneSmith, developed by researchers at MIT CSAIL and Toyota Research Institute, tackles this challenge using an agentic approach. The system leverages three AI agents, each powered by GPT-5.2, a state-of-the-art vision-language model trained on internet-scale text and images. The agents operate in distinct roles: a designer generates scene elements, a critic evaluates realism and practicality, and an orchestrator manages their back-and-forth dialogue. According to MIT EECS PhD student Nicholas Pfaff, a lead researcher on the project, "We've found that the system can construct 3D scenes the way a human designer would." The team created over 1,300 scenes, which Pfaff notes made "insanely creative and diverse arrangements" that emerged without explicit instruction: "I hadn't taught the system to do that in the prompts; it just improvised."

The resulting virtual environments are orders of magnitude richer than prior methods. Users can prompt SceneSmith to generate specific scenes—"generate a garage with a car, a workbench, tires stacked in the corner, and a ladder against the wall"—and receive detailed, interactive 3D spaces with up to six times more objects per scene than baseline approaches. Unlike prior systems, SceneSmith includes articulated objects like cabinets that robots can open and close, expanding the range of skills robots can practice. When researchers tested a robot policy trained on real-world data in these simulated spaces, it successfully completed manipulation tasks such as placing an apple on a cutting board. Humans agreed with the model's evaluations of robot failures over 99 percent of the time, validating the system as a filter for flawed approaches before deployment. Over 200 user evaluations found the system's visuals more realistic than competing methods 90 percent of the time.

The practical payoff is significant: engineers can evaluate and refine robot behavior in simulation rather than through extensive physical trials. However, SceneSmith comes with a computational cost. Generating a single scene can take multiple hours because the agents scrutinize each object in detail. Researchers believe more computing power could dramatically improve speed, and they plan to expand the system to handle deformable objects like sponges if suitable 3D libraries become available. Jeremy Binagia, an applied scientist at Amazon Robotics, calls SceneSmith "a significant advance," noting it "advances the state of the art in several ways, including pushing the limits of the density of objects in the simulated environment" and "creating assets that are not constrained to a fixed library." The team presented their findings as a spotlight presentation at the 2026 International Conference on Machine Learning, supported in part by Amazon, the U.S. Office of Naval Research, the Toyota Research Institute, and the U.S. National Science Foundation.