Researchers introduce a theoretical framework formalizing attacks and defenses in large language model alignment, demonstrating a provably optimal attack strategy that outperforms existing adversarial prompting methods.

Researchers developed a game-theoretic framework that models interaction between an attacker and a defender of large language models (AI systems that understand and generate text). The framework designs a theoretical best-response attack strategy and derives a provably optimal defense strategy.

The attack strategy is closely related to many existing adversarial prompting methods (carefully designed inputs intended to circumvent safety defenses). The framework characterizes the resulting game's equilibria and reveals inherent advantages for the attacker.

Empirical evaluation of a practical instantiation of the theoretically optimal attack showed stronger performance relative to existing adversarial prompting approaches across diverse settings encompassing different LLMs and benchmarks.