Apple researchers propose efficiency boost for diffusion language models

What happened

Apple researchers published a technical paper describing Residual Context Diffusion (RCD), a method that recycles information from tokens discarded during the decoding process of diffusion language models (a type of AI that generates text in parallel rather than one token at a time). The technique improved accuracy by 5–10 points across benchmarks and reduced computational steps by up to 4–5x on challenging math problems, requiring only ∼1 billion tokens to convert existing models.

Why it matters

Diffusion language models promise faster inference than traditional autoregressive models, but current designs waste computation by discarding tokens that still contain useful context. RCD recovers that wasted computation efficiently, which could help make these alternative language models more practical for real-world deployment without adding significant overhead.

What to watch

On the most difficult AIME math tasks, RCD nearly doubled baseline accuracy. The method uses a two-stage training pipeline designed to avoid memory bottlenecks, suggesting it may be applicable across a wide range of existing diffusion models.