Apple's DACA-GRPO lifts GRPO gains by 36.3pp

Diffusion large language models have been positioned as an alternative to autoregressive models, but existing reinforcement learning methods for them treat all denoising steps as equally important and rely on biased, high-variance likelihood estimates. The Apple-led team pinpointed two specific weaknesses: the lack of temporal credit assignment across the denoising trajectory, and the systematic bias of mean-field likelihood estimates used for policy optimization.

DACA-GRPO responds with two complementary mechanisms. Denoising Progress Scores extract per-token importance weights from intermediate predictions at no extra forward cost, while Stratified Masking Likelihood partitions token positions into strata so each token is predicted with most of the sequence as context, reducing the mean-field bias. Because the method is designed as a plug-and-play addition, it can sit on top of existing GRPO-style trainers without replacing them.

The reported results span seven benchmarks covering mathematical reasoning, code generation, constraint satisfaction, and constrained generation. The wide range of gains — from 5.6pp on math reasoning to 36.3pp on constraint satisfaction — suggests the benefits may be task-dependent, and it remains to be seen how these improvements translate to other evaluation settings.