Hugging Face turns to Chinese LLM after US models block incident response

Hugging Face, the New York-based platform for AI collaboration that hit $100 million（約160億円） annual recurring revenue this summer, disclosed on July 16 that it had suffered a production infrastructure breach in early July. An autonomous AI agent system exploited two code-execution paths in the platform's dataset processing—a remote-code dataset loader and template-injection in dataset configuration—to run code on a processing worker. The attacker then escalated to node-level access, harvested cloud and cluster credentials, and moved laterally into several internal clusters over a weekend using what Hugging Face described as a "swarm of short-lived sandboxes" with "self-migrating" command-and-control staged on public services.

When the company's security team began analyzing the attack, they encountered an unexpected obstacle: the safety guardrails on US frontier model APIs. "When we started the log analysis, we first used frontier models behind commercial APIs. This did not work: the analysis requires submitting large volumes of real attack commands, exploit payloads, and C2 artifacts, and these requests were blocked by the providers' safety guardrails," Hugging Face wrote in its incident report. The commercial models could not distinguish between an authorized incident responder and an attacker, making it impossible to use them for forensic work. In response, Hugging Face switched to GLM 5.2, an open-weight model developed by China's Z.ai lab, running it on their own infrastructure. The move had a dual benefit: it bypassed guardrail restrictions and ensured that sensitive attack data and credentials never left the company's environment.

Hugging Face told customers to rotate access tokens and review recent account activity. The company said it has seen no evidence of tampering with models, datasets, or spaces, and confirmed that its supply chain—container images and published packages—remains verified clean. However, Hugging Face stated it is "still completing our assessment of whether any partner or customer data was affected" and will contact customers directly if evidence of compromise emerges.

The company's recommendation to defenders was stark: "have a capable model you can run on your own infrastructure [your italics] vetted and ready before an incident, both to avoid guardrail lockout and to keep attacker data and credentials from leaving your environment." The admission that US safety guardrails blocked legitimate security work struck both infosec practitioners and investors, highlighting a gap between how AI safety mechanisms are designed and the real-world needs of incident response teams.

The Hugging Face breach exposes a real friction point in modern AI security: safety guardrails on commercial LLMs are designed to prevent harmful misuse, but they operate at the API level without distinguishing between an attacker and a legitimate defender. When Hugging Face's security team tried to analyze 17,000+ attacker logs using US frontier models via commercial APIs, the guardrails blocked requests containing real exploit code and C2 artifacts—even though analyzing those artifacts was essential to understanding the attack. This forced the company to pivot to GLM 5.2, an open-weight Chinese model they could run internally, solving both the guardrail problem and keeping sensitive attacker credentials within their own environment.

The timing of this admission—published the same day Moonshot's Kimi K3 model made headlines for outperforming some US models on code evaluation benchmarks—coincides with a broader shift in the geopolitical AI landscape. Chinese frontier models are also notably cheaper than US counterparts, according to Artificial Analysis data cited in the report. Meanwhile, Anthropic had just re-released its Fable 5 and Mythos 5 models after US export controls were lifted, adding stronger cybersecurity safeguards including classifiers that trigger on potentially harmful requests. Hugging Face's incident response lesson is direct: organizations should "have a capable model you can run on your own infrastructure vetted and ready before an incident," both to avoid guardrail lockout and to keep sensitive incident data from leaving their environment.

No comments yet. Be the first to share your thoughts!