Gray Swan, a startup co-founded by OpenAI safety board member Zico Kolter and CMU professor Matt Fredrikson, is tackling a new class of AI security risks—vulnerabilities in language models themselves—rather than traditional cybersecurity problems.

What happened

Gray Swan has raised funding and is positioning itself as a provider of AI safety and security tools. The company grew out of over a decade of research by Kolter and Fredrikson at Carnegie Mellon into vulnerabilities in deep learning systems. Their work includes the Gray Swan Arena (a red-teaming platform for testing model robustness), Shade (an adversarial red-teaming tool that Anthropic used to evaluate robustness against prompt injection attacks), and Cygnal (a guardrails product for policy enforcement).

What to watch

The body frames AI security as an emerging domain where specialized red-teaming models can now outperform humans at breaking AI systems, and where frontier models do not automatically become safer as they scale. The founders suggest that AI security may become embedded in insurance and compliance frameworks, and that major prompt-injection breaches may be inevitable—events that are visible in advance but difficult to prevent.