Yann LeCun: LLMs alone can't grasp a cup — world models are the missing layer

LeCun's contrast is not a claim that language models understand nothing about physics. Models can explain phenomena and reason with equations. What is missing, it argues, is a guarantee that such knowledge transfers into stable action on a real table, where position, weight, friction, joint angle, grip force, obstacles, and sensor noise all act at once. The proposed answer is a world model that predicts the next state from the current state and action, written as s_{t+1} = f(s_t, a_t), and often held as a distribution because the next state is not a single point. That form is not new; model predictive control has long used it in control engineering.

The JEPA line is presented as a way to avoid predicting every pixel. To walk past a person you need their position, heading, and distance to obstacles, not the pattern on their clothes. V-JEPA 2-AC adds an action-conditioned stage on top of a frozen encoder, trained on unlabeled DROID robot video, and the paper describes the planning as model predictive control in embedding space. The reported scope is short-horizon reaching, grasping, and placing, not general table clearing, and prediction errors can still select unsafe actions.

Hierarchy is the other half of the argument. A walking trip to the station involves choosing a destination and route at the top, avoiding people and steps in the middle, and issuing muscle commands at the bottom. Minimal code returns None when no action has zero cost, signaling that a higher-level plan such as emptying the cup or moving the neighboring plate is needed. It also notes that factory and logistics data must record which action produced which result for f(s_t, a_t) to be learnable, and that evaluation for physical systems needs success rate, safety, and robustness in unfamiliar situations alongside question-answering accuracy.