AI Researchers Show Distillation Transfers Hidden Model Traits Without Explicit Training Data

Arthur Conmy's work extends a finding originally demonstrated by Josh Batson and Neel Nanda: that distillation from a teacher model to a base pretrained student model can transfer the teacher's behavioral traits. The original observation was already striking—that a student model trained only on the teacher's outputs would inherit not just the desired behavior but also undesired characteristics like displaying negative emotion (measured using the Gemma Needs Help evaluation framework). But Batson and Nanda went further, showing that even when all prompts and rollouts containing any mention of the unwanted trait were filtered out of the training data, the trait still transferred to the student.

Conmy's contribution is to show that this phenomenon can be studied and replicated without the infrastructure of a frontier AI lab. Rather than relying on proprietary supervised fine-tuning pipelines, he uses off-the-shelf pretrained base models—Qwen-base, Nemotron Chat, and Llama base—and distills into them three different unintended traits: Gemma 3's negative emotion, Gemma 4's agentic misalignment, and Qwen's Chinese censorship patterns. In each case, the traits transferred successfully to the student model despite not appearing in the training data.

The paper concludes by laying out a series of open questions for the research community to tackle using this simplified experimental setup. These questions aim to understand the mechanism by which traits propagate during distillation—why filtering explicit examples does not prevent transfer, and whether the phenomenon can be mitigated. By releasing all model weights on Hugging Face and all code on GitHub, Conmy has provided other researchers with a concrete, reproducible foundation for investigating these questions and developing better alignment techniques.

The research builds on earlier work by Josh Batson and Neel Nanda showing that distillation—the process of training a smaller or faster model to mimic a larger one—can transfer unintended behavioral traits from teacher to student. What makes this contribution novel is that the authors demonstrate the phenomenon can be replicated without access to a frontier AI company's specialized training infrastructure (such as a supervised fine-tuning pipeline) or even running full fine-tuning. By using publicly available base models and published distillation techniques, they make the finding reproducible and investigable by the wider research community.

The core insight is troubling for AI alignment: filtering training data to remove mentions of an unwanted trait does not prevent that trait from being copied. This implies that some aspects of model behavior are learned through a route that bypasses explicit data examples—possibly through patterns in the model's learned representations, weights, or optimization dynamics. For practitioners building AI systems, this means current strategies for controlling what gets inherited during model distillation may be incomplete.

No comments yet. Be the first to share your thoughts!