Runway's GWM Worlds 2 lets you prompt real-time worlds

Runway's path to GWM Worlds 2 did not start with worlds at all. The company began as a way to make open-source models usable for artists, then spent its early years on a segmentation tool called Green Screen that was used in Everything Everywhere All at Once. The pivot to generative video came in 2022, when Runway signed a deal for a cluster of a thousand A100s and released Gen-1 in January 2023.

The engineering challenges behind real-time worlds are substantial. Kahlow described two: making the model generate frame by frame rather than a whole clip at once, and making generation fast enough to play in real time. Runway addressed this by fine-tuning its foundational audio-video model, post-training for autoregressive generation, and using distillation methods. Germanidis said a model might go from around 50 denoising steps to four, with some quality loss but potentially comparable results.

Whether GWM Worlds 2 becomes more than a research preview hinges on several open problems. Error accumulation in autoregressive models, managing GPU memory during infinite generations, and the lack of perfect memory are all acknowledged by the Runway team. The harder test may be counterfactual generation: Germanidis noted that football training data contains more successful goals than failed attempts, so the model might render success more convincingly. Until that gap closes, the fully self-generated, real-time games that world models promise will likely remain a goal rather than a product.