AI guardrails blocking legitimate security researchers from defending networks

What happened

AI companies including Anthropic and OpenAI have imposed strict guardrails on their models to prevent malicious use, but these safeguards are now preventing legitimate cybersecurity researchers—who probe systems to find vulnerabilities before criminals do—from using the same tools effectively. In June, the U.S. government imposed export control restrictions on Anthropic's Mythos and Fable models over jailbreak concerns, though those controls have since been partially lifted.

Why it matters

Offensive security researchers say the guardrails create an impossible paradox: the same prompts needed to test whether a bug is exploitable (defensive work) are blocked by AI companies as potentially dangerous. Chris Anley of NCC Group compared it to a hammer—"You can't build a house without a hammer. It's definitely a tool but it's also irreducibly a weapon as well." The blanket restrictions treat security experts like threats, forcing them toward less regulated alternatives.

What to watch

Researchers are increasingly turning to unvetted open-source Chinese models like GLM that run locally with no restrictions, according to Chris Thompson of RemoteThreat. Thompson warned that defenders risk losing the "AI race" if guardrails push legitimate researchers away from U.S.-governed systems, and called for AI labs to "open up their programs, provide responsible access, and hold those who abuse their tools accountable" instead of tightening restrictions further.