Apple builds first tri-modal Masked Diffusion Models at 3B scale

Diffusion models have mainly been used for continuous data like images, but discrete diffusion models have recently emerged as strong alternatives to autoregressive language models. Previous work typically initialized and finetuned a base unimodal model for bi-modal generation. This new study diverges from that pattern, pretraining a tri-modal Masked Diffusion Model from scratch on text, image-text, and audio-text data.

The researchers also contribute practical insights on training. Their batch-size analysis proposes a stochastic differential equation (SDE) based reparameterization, which decouples the physical batch size, chosen based on compute constraints such as GPU saturation and wall-clock time, from the logical batch size, chosen to balance the variance of gradients during stochastic optimization.

The work is positioned as the largest systematic open study of multimodal discrete diffusion models conducted to date, and the authors note it provides valuable insights into scaling behaviors across multiple modalities. A preliminary model demonstrates the capabilities of the unified design at 3B model scale with 6.4T tokens.