Apple researchers improve code generation via self-distillation

A team of Apple researchers including Ruixiang Zhang, Richard He Bai, Huangjie Zheng, Navdeep Jaitly, Ronan Collobert, and Yizhe Zhang set out to test whether an LLM could improve at code generation by learning only from its own outputs, without external supervision or reinforcement learning. Their answer is yes, through a method they call simple self-distillation (SSD).

The approach is straightforward: the model generates code samples using specific temperature and truncation settings, then undergoes standard supervised fine-tuning on those samples. When applied to Qwen3-30B-Instruct, this technique raised the model's pass@1 score on LiveCodeBench v6 from 42.4% to 55.3%—a gain of nearly 13 percentage points. The improvements were not uniform across difficulty levels; gains concentrated on harder problems, suggesting the method is particularly effective where the model's initial performance is weakest.

The technique generalizes beyond a single model. Testing across Qwen and Llama model families at multiple scales (4B, 8B, and 30B parameters) and variants (both instruct and thinking models) confirmed that simple self-distillation works consistently. To understand why such a simple method is effective, the researchers traced the improvements to what they call a precision-exploration conflict inherent in LLM decoding. Their analysis shows that self-distillation reshapes token distributions in context-dependent ways: it suppresses distractor tails in situations where precision matters (i.e., where one correct answer should dominate), while preserving useful diversity in cases where exploration is beneficial (i.e., where multiple valid solutions exist). This selective refinement of the model's output distribution explains the method's success.

The research was accepted at IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) 2024. By offering a post-training approach that requires no external verifier, teacher model, or reinforcement learning infrastructure, simple self-distillation provides practitioners with a low-cost way to strengthen code generation in existing open-source models.

Code generation has become a common tool for programmers, but LLM-generated code often presents usability challenges. The Apple research team's contribution addresses a practical constraint: improving code generation without relying on expensive infrastructure like teacher models, verifiers, or reinforcement learning pipelines. Simple self-distillation achieves a meaningful improvement—13 percentage points on the benchmark—by leveraging the model's own outputs as training data.

The mechanism underlying this gain offers insight into how LLM decoding actually works. The researchers identify a tension between precision (producing the single correct answer) and exploration (maintaining diversity to cover multiple valid solutions). Self-distillation handles this by reshaping token distributions in a context-dependent way: it suppresses distracting alternatives in situations where one answer is clearly right, while preserving useful variation where multiple solutions are valid. This selective refinement explains why such a straightforward method can yield substantial improvements, especially on harder problems where the model's initial outputs tend to be weaker.

No comments yet. Be the first to share your thoughts!