MIT system uses AI agents to create realistic virtual training spaces for robots

The core challenge roboticists face is that robots learn best through experience, but physically teaching machines across different settings is time-consuming and labor-intensive. Simulation has long been seen as a solution, but prior physics engines have struggled to create sufficiently realistic and diverse virtual environments that capture the complexity of the real world. SceneSmith addresses this gap by leveraging AI agents powered by vision-language models—systems trained on vast amounts of text and images from the internet—to generate rich, detailed indoor spaces.

The three-agent design is elegant in how it distributes creative and quality-control tasks. The designer generates elements, the critic ensures practicality (for instance, flagging when a bathtub doesn't belong in a living room), and the orchestrator manages their collaboration. Over 200 user evaluations found the system's visuals more realistic than competing approaches 90 percent of the time, and it adhered more closely to text prompts. Critically, when researchers tested a robot policy trained entirely on real-world data in SceneSmith's generated environments—without the policy ever having seen such scenes—it successfully completed tasks like moving an apple from a bowl to a cutting board, suggesting the simulations are faithful enough to transfer real-world-trained skills.