CFG and cross-attention: how text steers AI image generation

The explainer is built as a follow-up to an earlier introduction to diffusion models, which described text-to-image as "text conditioning plus CFG" in one line; this piece opens that box. Its central division of labour is between understanding and drawing: CLIP's encoder, trained contrastively on roughly 400 million image-text pairs, is grounded in images but shallow as language, while T5's bidirectional encoder is a pure language model that never sees images. SD3/Flux use both — CLIP×2 plus T5-XXL — so their weaknesses offset each other.

The piece traces the denoiser's evolution from a CNN-based U-Net with inserted Transformer blocks toward DiT and MMDiT, which join text and image tokens in a single attention.

The final stage frames CFG as an answer to a specific problem: plain conditional diffusion follows prompts only weakly. Unlike classifier guidance, which needs a separately trained classifier and is limited to class labels, CFG drops the condition during training so one network handles both conditional and unconditional prediction, then amplifies their difference at generation. Negative prompts occupy the same unconditional slot, steering away from unwanted terms.