AI Agents Build Virtual Playgrounds to Train Robots More Efficiently

Robots are appearing on streets and workplaces more frequently, yet they still fall short of the versatile assistants needed for complex tasks like cooking or factory work. A major obstacle is data: robots, like humans, learn best through experience, but physically teaching them every action in diverse real-world environments is labor-intensive and time-consuming.

“One natural idea is to use simulation as a training ground,” says Russ Tedrake, a professor at MIT and principal investigator at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL). “One of the remaining challenges has been creating sufficiently rich and diverse simulation content to capture the complexity of the real world.”

Three AI Agents Collaborate

Researchers from MIT CSAIL and the Toyota Research Institute have developed a system called SceneSmith that overcomes this hurdle. It uses three AI agents—semi-autonomous programs that complete specific tasks—to generate lifelike 3D indoor scenes such as restaurants, bedrooms, and hotels. Each agent draws on a vision-language model (VLM), specifically the advanced GPT-5.2, trained on vast text and image data to understand spatial layouts.

The first agent, the “designer,” generates scene elements. The second, the “critic,” checks realism. The third, the “orchestrator,” manages their back-and-forth and decides when the design is complete. Their collaboration produces a scene that can be loaded directly into physics simulation software.

“We’ve found that the system can construct 3D scenes the way a human designer would,” says MIT PhD student Nicholas Pfaff, lead author of a paper on the work. “It made insanely creative and diverse arrangements. I hadn’t taught the system to do that; it just improvised.”

Testing Realism in Virtual Worlds

The virtual environments are rich with objects—up to six times more per scene than prior methods—enabling robots to practice skills such as placing a cup in a sink or moving a soda can. Researchers tested robot action plans in SceneSmith’s digital worlds, generating 100 unique spaces. A VLM agent evaluated the robot’s attempts, and humans agreed with its verdicts over 99 percent of the time, allowing flawed approaches to be weeded out before real-world deployment.

To verify realism, the team dropped a pretrained robot policy—trained on real-world data and never exposed to SceneSmith—into the generated environments. When told to “take the apple from the bowl and place it onto the cutting board,” the simulated robot succeeded. If the scenes had not closely matched the real settings the policy learned from, the task would have failed.

Additional tests involving teleoperated robots opening cabinets and navigating between rooms confirmed that the environments hold up under sustained physical interaction.

User Preference and Practical Advantages

In surveys of over 200 users, SceneSmith’s visuals were deemed more realistic than those of alternative systems more than 90 percent of the time. Users also noted that the system followed prompts more closely than other approaches, generating exactly the virtual playgrounds needed for training.

The system can even produce individual 3D objects with physical properties like mass and friction from a simple text prompt, such as “a rolling serving cart.” However, this detailed process can take multiple hours per scene due to the agents’ careful creation and scrutiny. With more computing power, efficiency could improve dramatically.

“SceneSmith provides an agentic framework for generating simulation-ready indoor environments just from a simple text prompt,” says Jeremy Binagia, an applied scientist at Amazon Robotics not involved in the research. “It pushes the limits of object density, ensures physical accuracy, and creates assets not constrained to a fixed library.”

Future Improvements

The researchers hope to expand SceneSmith to include deformable objects like sponges, pending the availability of extensive 3D libraries. The team, which includes MIT and Toyota Research Institute members, presented their findings as a spotlight at last week’s International Conference on Machine Learning, supported by Amazon, the U.S. Office of Naval Research, the Toyota Research Institute, and the U.S. National Science Foundation.

The source for this article is https://news.mit.edu/2026/ai-agents-create-virtual-playgrounds-to-help-robots-get-crucial-training-data-0713.