Cisco AI Defense tops multilingual safety test with 0.845 F1 score

Cisco AI Defense achieved the highest F1 score (0.845) in an independent benchmark conducted by ML6 on 80,000 Dutch-language prompts, testing guardrails against prompt injection, policy bypass, ambiguous instructions, and realistic enterprise interactions. The test placed Cisco's solution at the top of a cohort of providers, demonstrating strong performance on a language and attack surface rarely emphasized in AI safety benchmarks.

The core innovation is Cisco's constitutional taxonomy: a set of precise, machine-enforced operational specifications per technique that serve as a single source of truth for classification, model training, and customer-facing explanations. This approach reduces inter-model disagreement by up to 57× compared to traditional paragraph-level definitions. Critically, because each specification is machine-enforced rather than left to human interpretation, it applies with equal precision in French, Japanese, or Arabic—a property essential for enterprises operating globally. The taxonomy also distinguishes intent from content: it can identify a probed-and-refused attack (harmful intent without harmful output) separately from model misbehavior on a benign request (harmful content without adversarial intent), a distinction that Cisco research shows is essential in production, where the same surface language can carry very different meanings depending on conversational context.

Cisco's evaluation on an augmented multilingual dataset derived from LMSYS Chat-1M and WildChat—with approximately 5,800–5,900 conversations per language and roughly 14% adversarial examples—showed F1 scores ranging from 0.796 (Arabic) to 0.860 (Portuguese), a tight spread across nine typologically diverse languages. The system achieved 0.843 recall and 0.847 precision simultaneously, avoiding the precision collapse seen in a competing guardrail that reached only 0.453 F1 despite 0.327 recall and 0.737 precision. False-positive rates were stable across languages at 2.3–5.8%, indicating that the constitutional taxonomy produces consistent signal rather than silently trading precision for recall as users switch languages. Operating thresholds are configurable without retraining, allowing organizations to tune the precision-recall tradeoff to their specific risk profile.

Production performance is engineered for enterprise constraints. Cisco AI Defense operates at p90 = 40 ms and p99 = 250 ms per request, adding imperceptible overhead that is compatible with real-time conversational SLAs across chatbots, copilots, and agentic pipelines. This speed is critical because a guardrail that cannot keep pace with traffic will not stay in the critical path; security that adds seconds per request gets disabled or bypassed. Cisco research across 15 frontier models found that every model tested shows meaningful multi-turn vulnerability, with attack success rates bearing no consistent relationship to single-turn benchmarks. This finding has shifted the security perimeter: instead of relying on model-level defenses, Cisco AI Defense validates inputs and outputs in production, classifying the intent and active direction of each conversation, not just the surface content of each message. Guardrails are tailored to the specific vulnerabilities of each model and application, and applied at the point where AI behavior is actually shaped—the live exchange between user, model, data, and tools.

Enterprise AI safety has historically relied on single-language, single-turn benchmarks that do not match production reality. The ML6 benchmark and Cisco's evaluation highlight a critical gap: Cisco research across 15 frontier models found that every model tested shows meaningful multi-turn vulnerability, with attack success rates that bear no consistent relationship to single-turn benchmarks. Adversaries do not attack in isolation; they iterate, reframe refusals, and escalate gradually across turns. This shift moves the security perimeter outside the model itself, into the live conversation between user, model, data, and tools.

Cisco's constitutional taxonomy addresses this by establishing precise operational specifications per technique, rather than relying on paragraph-level safety labels that invite semantic ambiguity. Because the definition is machine-enforced, it translates reliably across typologically diverse languages—from Latin-script European languages to Arabic and Japanese. The F1 range of 0.796 to 0.860 across nine languages demonstrates this consistency; by contrast, a competing guardrail solution in the benchmark achieved 0.453 F1, collapsed by false alarms (0.737 precision despite 0.327 recall). Cisco achieved 0.843 recall and 0.847 precision simultaneously, a balance that protects without disabling legitimate use.

Production deployments have additional constraints: response-time SLAs, latency-sensitive agentic pipelines, and continuous model evolution. Cisco AI Defense is tuned to add imperceptible overhead (p90 = 40 ms, p99 = 250 ms per request) while supporting runtime protection across every language, model, and deployment framework an enterprise runs, independent of vendor or framework choice.

No comments yet. Be the first to share your thoughts!