Google Robotics Team Shares Engineering Fixes for Teaching Robots via Human Demonstration

The Robotics Team at Google is developing Vision-Language-Action (VLA) models—AI systems that understand images, language, and robotic motion—to create general-purpose robot foundation models. To train these models efficiently, they adopted Universal Manipulation Interface (UMI), a data collection framework co-developed by Stanford University and the Toyota Research Institute. UMI lets humans teach robots by demonstrating tasks with a handheld device that mimics a robot's gripper; the system records the human's hand pose and movements, then trains the robot to replicate the behavior.

The original UMI method estimated hand pose from action camera images and IMU (motion sensor) data, but this approach struggled when the background lacked rich visual features or the hand moved in ways that confused the tracker. To improve reliability, Google's team used VR trackers—standard positional tracking hardware from virtual reality systems—to measure hand pose with high precision. This solved the data collection problem but revealed a deeper issue when they tried to deploy the learned policy onto their custom robot.

The robot uses QDD motors, a type of actuator that allows precise torque control, and is based on the open-source Openarm platform. Unlike off-the-shelf industrial robots, custom-built robots accumulate small assembly errors: tolerances in part dimensions, slight misalignments during assembly. These errors cascaded into the robot's joint calibration. In a normal teleoperation setup (human directly controlling the robot), this zero-point offset matters less because it exists during both training and deployment, so errors cancel out. But when using UMI—where data comes from a handheld device and policy runs on a different physical platform—the offset became a critical problem, causing the robot's end-effector to miss target positions by several centimeters.

To fix this, the team implemented three engineering solutions. First, they used the XYZ 4-point method, a standard industrial robotics calibration technique: the robot approaches a fixed reference point (usually a sharp pin) from at least four different arm postures. By recording which joint angles align with the same physical point in space, they back-calculated the true joint offsets and corrected them. This required precise measurement, so they 3D-printed calibration tools to hold the reference point steady.

Second, they implemented gravity compensation in the robot's control system. QDD motors are naturally low-friction and allow transparent torque control, but to unlock this capability, the controller must actively cancel gravity throughout the arm's movement. Without this, a proportional-derivative (PD) controller would have to fight gravity, requiring high control gains that make the arm stiff and rigid. With gravity compensation, the controller only tracks errors, allowing much lower gains. This is crucial for the manual calibration process itself: during the XYZ 4-point method, a human operator must guide the robot's end-effector by hand to align with the reference point, and a compliant arm is far easier to guide than a stiff one.

Third, they recalibrated the camera. The team swapped the wrist camera's standard lens for a wider field-of-view lens to capture more of the task environment. But even a small lens change causes the image center (principal point) to shift and introduces new lens distortion. Using OpenCV, they performed intrinsic camera calibration—a standard computer vision procedure that determines the camera's exact optical parameters. They also calibrated the spatial relationship between the handheld device's end-effector and the VR tracker to ensure perfect alignment between the demonstration space and the robot's action space.

When testing the learned policy, the team used a top-down camera to observe both data collection (with a human hand visible) and robot execution (no hand). Domain shift—when training and deployment look visually different—can degrade performance, and they initially worried that the visible human hand during training would confuse the robot. Surprisingly, the robot completed both a pick-and-place task (requiring sub-centimeter precision) and a T-shirt folding task without special image masking, suggesting the learned model generalized across this visual gap better than expected. The team concludes that behind the impressive public videos of physical AI are unglamorous but essential engineering details: precise measurement, compliant control, and accurate sensing. These details are often invisible in published research but are critical for making robot learning work on real, non-standard hardware.