VR-collected robot demos match NVIDIA's 200+ dataset in simulation tests

The team at Sim XR undertook a technical reproduction of NVIDIA's published robotics pipeline, which chains simulation training (Isaac Lab), demonstration collection (LeRobot), vision-language-action model fine-tuning (VLA), and evaluation in the Arena benchmark. Their objective was to test whether a smaller set of 50 remotely collected VR demonstrations could achieve comparable performance to NVIDIA's larger dataset when training a policy for the Unitree G1 robot to pick up an apple.

In matched simulation evaluation on the original apple task, the checkpoint trained on 208 released demonstrations scored 93/100, while the checkpoint trained on 50 VR-collected demonstrations scored 84/100. This 9-point gap suggested that VR-collected data could serve as a viable substitute for larger demonstration sets. The team then tested robustness by repositioning the task to a different layout in the workspace. Both the existing policies—whether trained on the large dataset or the smaller VR set—failed entirely, scoring 0/20. After collecting targeted new VR demonstrations for the relocated layout, the team achieved a best result of 74/100, demonstrating that task adaptation required fresh demonstration data.

The team went further and tested whether the learned apple-picking policy would transfer to a different manipulation task: moving a mustard bottle into a bowl. The released apple checkpoint, trained on 208 demonstrations of apple picking, scored 0/30 on this new task. When the team fine-tuned a new checkpoint on 50 VR demonstrations specific to the mustard-bottle task, it reached 27/30. The team emphasized that all results were simulation-only and did not constitute real-robot or sim-to-real validation. The key finding, in their assessment, was that policies remained unexpectedly sensitive to both layout shifts and task variations, implying that while VR collection may reduce total demonstration volume, fine-tuning on new data remains necessary when conditions change materially.

The experiment stems from NVIDIA's published workflow for fine-tuning vision-language-action (VLA) models on robotics tasks using the Isaac Lab simulation environment and LeRobot framework. The Sim XR team set out to test whether smaller, remotely collected VR datasets could replace the larger demonstration sets NVIDIA had used, a practical question for teams without access to extensive in-person data collection. Their initial result—84/100 on the apple task with 50 VR demonstrations versus 93/100 with 208 released demonstrations—suggests the gap is modest, opening a potentially lower-cost path to policy training.

However, the team's further findings reveal important constraints. The policies showed high sensitivity to spatial layout changes; when the apple task was relocated, performance dropped to zero despite successful training on the original layout. Similarly, the learned apple policy did not generalize to a different manipulation task (mustard bottle to bowl), requiring task-specific fine-tuning. These results highlight that while VR demonstration collection may reduce data volume, the underlying policies remain brittle to changes in environment and task structure—a limitation the team notes persists even with the larger baseline dataset.

No comments yet. Be the first to share your thoughts!