Knowledge distillation made cheap enough for single GPU

Knowledge distillation has become the standard way to compress large language models for deployment, but the process itself has remained prohibitively expensive. The distillation step traditionally requires keeping both the teacher model (which can have trillions of parameters) and the student model in memory simultaneously, computing a probability distribution over the entire vocabulary for every token position in the sequence. For models like gpt-oss-120b with a 201,088-token vocabulary at 32K sequence length and batch size 4, this creates a single teacher-probability tensor of roughly 50GB, and a total training iteration that can spike to 250GB of VRAM—more than even the largest single GPUs can hold. This has meant distillation campaigns could only be run at scale with specialized infrastructure: hundreds of GPUs, careful tensor-parallelism strategies, and significant engineering overhead.

The two optimizations address this directly. First, offline distillation eliminates the need to recompute the teacher at every training step; instead, the teacher's top-100 most likely tokens per position are cached once, and the student trains against that cache. This removes the teacher from memory entirely during training and eliminates redundant computation. Second, the fused chunked KL loss avoids ever materializing the full vocabulary-size × sequence-length matrix; instead of building the entire comparison grid before computing the loss, it processes the sequence in chunks, computing each chunk end-to-end and discarding it before moving to the next, recomputing during backpropagation as needed. Together, these changes reduce peak memory consumption from 250GB to 128GB or less in the original example, and achieve far more dramatic reductions at longer context lengths—at 256K tokens, memory use drops to 11.6 GiB versus 134.2 GiB for the next-best method.