Apple researchers propose learned policies for diffusion language model sampling

Key takeaway

• Apple researchers have developed a method to automatically train sampling policies for diffusion language models using reinforcement learning, replacing manually tuned heuristics.

• The approach uses a lightweight transformer-based policy to decide which tokens to unmask during generation, matching or exceeding the performance of hand-crafted strategies while avoiding the need for manual tuning and scaling issues that plague existing methods.

3 Key Points

FAQ

How does the new approach differ from existing sampling methods for diffusion language models?

Existing methods use heuristic strategies like confidence thresholding, which require manual tuning and degrade with larger block sizes. The new work proposes training sampling procedures using reinforcement learning with a lightweight policy based on a single-layer transformer that maps token confidences to unmasking decisions.

Where does this approach perform best?

The trained policies match the performance of state-of-the-art heuristics when combined with semi-autoregressive (block) generation, while outperforming them in the full-diffusion setting.