Brain waves join robot training data arsenal

In a warehouse in San Leandro, California, Andrew Ceja, one of Encord's pilots (the company's term for robotic trainers), carefully pulled wooden blocks from a Jenga tower while wearing a headset equipped with a camera that tracks his eye movement. But this headset included an additional innovation: sensors measuring his brain waves as he worked. Encord, a data-tooling company founded to help machine-vision applications annotate data, partnered with Zander Labs, a German neuroscience startup, to test whether brain-activity measurement could enhance training data for robots.

The trial emerged from a fundamental recognition: the robotics industry faces an acute shortage of real-world physical training data. As Vineeth Velmurugan, Encord's head of robot learning and a veteran of OpenAI's robot lab and warehouse-automation firm Berkshire Grey, explained, "The data simply does not exist." When Encord's robotics customers began pursuing end-to-end learning for manipulation, the company realized it would have to manufacture training data rather than simply manage it. Velmurugan estimates that robots will require a training data set roughly five times the size of YouTube's entire video corpus to achieve the kind of breakthrough that LLMs reached—a scale that explains why data generation has become its own business.

Zander's contribution centers on mental-state inference. According to Lucas Gehrke, a Zander neuroscientist supervising the work, the amount of brain activity used during a given task offers clues about when models need to deploy their highest-effort computation. Encord's plan is to build an initial brain-wave-tagged data set, run it through customer robotics models, and evaluate whether it actually improves performance before scaling up. In parallel, the San Leandro facility pursues other data modalities: pilots operate paired robotic arms (leader-follower rigs) to generate data for specific manipulation tasks—pouring coffee, stacking poker chips, plugging and unplugging ethernet cables from server backs—while another emerging modality uses forearm sensors to detect electrical muscle signals, allowing the company to infer 3D hand position when video alone cannot capture it. Encord annotates all its data sets with dense physical descriptions ("right hand tightens bolt") to aid LLM-based models in understanding task content. Velmurugan's analysis suggests this kind of annotation is worth 100 times as much as unannotated egocentric data for training specific tasks, and costs only 20 times more to produce. Yet that 20-fold cost multiplier, while economically favorable compared to the value gain, remains a substantial barrier—fundamentally different from the near-zero cost of scraping internet text for LLM training. Both pilots Ceja and Infante came from Scale, another AI data annotation firm, and the dozen or so workers at the facility form part of a burgeoning workforce manufacturing the building blocks for physical AI models.

The constraint limiting physical AI has shifted from model architecture to the sheer scarcity of real-world training data. Unlike large language models, which were built on text scraped from the internet at near-zero cost, robots require hands-on, physically grounded training data that must be deliberately manufactured. Encord's pivot from simply managing customer data to producing it reflects this new reality: the company realized that as robotics firms attempted end-to-end learning for manipulation tasks, "the data simply does not exist." This economic shift—from collection to manufacturing—changes the entire equation for physical AI development.

Encord's experiment with brain-wave measurement represents an attempt to increase the informational density of each training sample. By capturing not just video but also the pilot's mental states (error detection, intent, surprise), the company hopes to create a richer signal for model training. Velmurugan's claim that such dense annotation is worth 100 times as much as raw video, while costing only 20 times more, suggests a genuine efficiency gain—but "20 times more" remains a substantial cost barrier. The company's multi-modal approach—combining egocentric video, remote-operated robotic arms, muscle electrical signals, and now brain waves—reflects a broader bet that the next competitive advantage in robotics will belong to whoever can manufacture the highest-fidelity training data most efficiently.

No comments yet. Be the first to share your thoughts!