
Anthropic recently detailed 'distillation attacks' by 3 Chinese labs that used jailbreaking and hacking to extract outputs from model APIs; the author argues the behavior should be called 'jailbreaking or abuse' rather than distillation, as distillation itself is an industry-standard training method.
Distillation—training a smaller model on outputs from a stronger one—is widely used by frontier AI labs to create cheaper versions for customers and by smaller players for specialized models, often as part of multi-stage post-training processes that muddy the exact impact of the original model.
A bill has moved out of a committee in Congress and an executive order is pushing for action in response to these incidents; the author warns this regulatory environment could harm the U.S. ecosystem by creating grey area that exposes entities to risk, potentially squashing small open-source contributors and making Western academics and smaller companies building models less relevant.
Summaries like this, every morning.
Pick your industry and the AI tools you use, and get news related to your work every day.
Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.