Researcher uses OpenClaw AI agent to control a physical robot arm, demonstrating that AI-powered coding can simplify robot training and control

The author connected OpenClaw (an AI coding agent) to a LeRobot 101 arm from HuggingFace's open-source robotics project. With help from OpenClaw and Codex, the author wrote a program that made the arm identify and grip a red ball, then trained a model to pick up objects.

Researchers from UC Berkeley, Nvidia, Carnegie Mellon University, and Stanford developed CaP-X, a benchmark to measure how well coding models can control robots. Their results show Gemini performs best for robot programming—possibly because Google DeepMind trained it to be multimodal (able to understand both text and images from the physical world). They also created CaP-Gym (a simulation and real-robot environment) and CaP-Agent0 (a framework that boosts coding-model performance on robot tasks).

Ken Goldberg, a roboticist at UC Berkeley, says AI-powered coding "has the potential to bridge the gap between conventional engineering methods, which are reliable but don't generalize, and contemporary vision-language-action models, which generalize but are not yet reliable." Spencer Huang notes that making robots controllable by spoken or typed commands is the "critical unlock for robots in society."