NVIDIA, Hugging Face enable scaled fine-tuning of video and image diffusion models

NVIDIA and Hugging Face have announced a collaboration that brings production-grade distributed diffusion training to any Diffusers-format model on the Hugging Face Hub. At the center is NVIDIA's NeMo Automodel, an open-source PyTorch DTensor-native training library designed around two principles: Hugging Face native integration (users point pretrained_model_name_or_path at any Diffusers model ID and begin training) and one-program-any-scale training (parallelism becomes a configuration choice, not a code rewrite).

NeMo Automodel currently supports flow-matching models only, using flow matching as the training objective with latent-space training via pre-encoded VAE outputs and multiresolution bucketed dataloading to accelerate throughput. The integration ships with ready-to-use fine-tuning recipes for FLUX.1-dev, Qwen-Image, Wan 2.1, Wan 2.2, and HunyuanVideo. The practical gains include no checkpoint conversion (pretrained weights work out of the box), fast path to new model support (a new diffusion model in Diffusers can be enabled in NeMo Automodel with a small, contained code addition), and full and parameter-efficient fine-tuning (both full fine-tuning and LoRA are supported). The system also offers scalable training with sharding schemes such as FSDP2, tensor, context, and pipeline parallelism, multi-node orchestration via SLURM (Kubernetes coming), and multiresolution bucketing.

The typical workflow involves three steps: pre-encoding the dataset (streaming images from Hugging Face and distributing preprocessing across GPUs to produce cached VAE latents and text embeddings), launching training with an existing YAML configuration and command-line overrides (no new config files required), and generating from the fine-tuned checkpoint. The showcase example fine-tunes FLUX.1-dev on the 78-card Rider–Waite tarot dataset. After 200 optimizer steps on 8× NVIDIA H100 80GB GPUs, the triggered astronaut prompts (using the learned "trtcrd" trigger token) acquire a cream, red, and black vintage palette, heavy ink contours, flat color fields, aged-paper tones, and allegorical card composition, while the untriggered astronaut remains photographic, demonstrating that the learned effect is substantially associated with the trigger token rather than globally replacing the base model.

Performance measurements collected on one node with 8× NVIDIA H100 80GB GPUs show that FLUX.1-dev full fine-tuning (512×512 images) achieves a step time of 0.902 ± 0.039 s and 35.51 ± 1.55 images/s, while LoRA with r64 achieves 0.894 ± 0.008 s and 53.73 ± 0.48 images/s. Qwen-Image full fine-tuning achieves 0.974 ± 0.075 s step time and 41.21 ± 3.06 images/s. For video (512×512×49 frames), Wan 2.1 14B full fine-tuning reaches 3.798 ± 0.017 s step time and 2.107 ± 0.006 clips/s, while LoRA achieves 7.585 ± 0.014 s and 2.110 ± 0.000 clips/s. HunyuanVideo 1.5 full fine-tuning achieves 5.926 ± 0.046 s step time and 1.350 ± 0.010 clips/s, while LoRA reaches 5.575 ± 0.006 s and 1.433 ± 0.006 clips/s. The integration is fully open source under Apache 2.0, with installation available via Docker container, pip3, or from source on GitHub; the integration is documented in the Diffusers training guide. The collaboration also plans to release Pythonic recipe APIs in the coming future.

The integration addresses a longstanding friction point in diffusion model training: scaling from research to production requires memory-efficient techniques (FSDP2, tensor parallelism, multiresolution bucketing, latent caching) that were previously scattered across custom training scripts or unavailable entirely. By embedding these capabilities into NeMo Automodel and linking it to Diffusers—the de facto standard interface for open-source diffusion models—NVIDIA and Hugging Face lower the barrier for domain specialization. Researchers can now fine-tune models like FLUX.1-dev (12B) or HunyuanVideo (13B) on single nodes using LoRA or on larger clusters with full fine-tuning, all without model rewrites. The showcase example (fine-tuning FLUX.1-dev on a 78-image Rider–Waite tarot dataset) demonstrates that even small datasets can acquire learned styles—the fine-tuned model's generated images exhibit a cream, red, and black vintage palette with ink contours and allegorical card composition when prompted with the learned trigger token, while the base model remains photographic. Because the recipe stack (FSDP2, checkpointing, generation) carries over unchanged, onboarding a new diffusion model into NeMo Automodel requires only a data preprocessing handler and a model adapter, not a full custom training script. This design choice makes it practical to keep pace with the rapid release cycle of new diffusion models.

No comments yet. Be the first to share your thoughts!