Hugging Face says AI agent hacked it; used its own AI to fight back

On a weekend, an autonomous AI agent system conducted a multi-stage attack on Hugging Face's production infrastructure, exploiting weaknesses in the platform's data processing pipeline. The attack began with a malicious dataset that triggered two code execution paths: a remote code dataset loader and template injection in dataset configuration. Once inside, the attacker escalated privileges to node level, harvested cloud and cluster credentials, and moved laterally across multiple internal clusters. Hugging Face says the entire campaign was orchestrated by an autonomous agent framework built on an agentic security research harness—the company does not know which language model powered the attack—and the system executed many thousands of individual actions through a swarm of short-lived sandboxes while using self-migrating command-and-control infrastructure running on public services.

Hugging Face detected the breach through an AI-powered anomaly detection pipeline that runs LLM-based triage on security telemetry. To understand the attack's scope, the company faced a challenge: it had recorded over 17,000 attacker actions and needed to reconstruct the timeline, extract indicators of compromise, map affected credentials, and distinguish real damage from deception. The company deployed LLM-driven analysis agents to handle this work, completing in hours tasks that would normally have consumed days of manual analysis.

However, when Hugging Face's security team initially attempted to analyze the attack logs using frontier models behind commercial APIs, they hit a critical obstacle. The providers' safety guardrails blocked the requests because the systems could not differentiate an incident responder from an attacker when presented with real attack data—exploit payloads, C2 artifacts, and actual attack commands all tripped the filters. Forced to find an alternative, Hugging Face turned to the open-weight model GLM 5.2 running on its own infrastructure. This solution offered two advantages: no attacker data left Hugging Face's environment, and none of the referenced credentials ever left the company's own servers. "We do not know which model powered the attacker's agents, whether a jailbroken hosted model or an unrestricted open-weight one; either way, the attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried," the company wrote.

Hugging Face shut down the exploited code execution paths, revoked the attacker's access, rebuilt compromised nodes, and rotated affected credentials. The company tightened access controls and improved detection systems. Public models, datasets, and Spaces were not tampered with, and the software supply chain was not affected, though whether partner or customer data was compromised remains under investigation. Hugging Face is working with external cybersecurity forensics experts and has reported the incident to law enforcement. As a precaution, the company recommends all users rotate their access tokens and review recent account activity.

Hugging Face's breach represents a milestone in the evolution of AI-driven threats: the company explicitly classifies the incident as the "agentic attacker" scenario the industry has long predicted. An autonomous agent framework orchestrated thousands of individual actions through short-lived sandboxes and self-migrating command-and-control infrastructure on public services. The attacker's operational model—executing many thousands of actions at machine speed across multiple stages—demonstrates that autonomous, AI-driven attack tools have moved from theoretical threat to demonstrated reality.

The incident also reveals a critical paradox for defenders. Hugging Face relied on its own AI tools—an LLM-powered anomaly detection pipeline and LLM-driven analysis agents—to compress days of manual investigation into hours. Yet when the company first attempted to use commercial hosted AI models from major providers for the same forensic work, safety guardrails intended to prevent misuse blocked the requests. The guardrails could not distinguish legitimate incident response from malicious activity when presented with authentic attack data (exploit payloads, command-and-control artifacts, and attack commands). This forced Hugging Face to fall back on an open-weight model (GLM 5.2) running on its own infrastructure, where no such restrictions applied. The company argues this gap—where commercial safety measures inadvertently handicap defenders—is something the industry should prepare for. That said, Hugging Face is one of the largest platforms for open-source AI models and has a business incentive to promote open-weight models as essential for security work, so its conclusion carries an element of self-interest.

No comments yet. Be the first to share your thoughts!