garak 0.17.1 vs PyRIT: 85% attack success on mock app, 0% when guarded

garak and PyRIT answer different questions rather than competing products. garak, which its 2024 paper compares to the network scanner Nmap, is built to sweep many prebuilt probes across a target and surface known weakness patterns; the author counted 42 probe modules and 191 individual probes in garak 0.17.1. PyRIT, by contrast, is assembled from parts — targets, converters, scorers, and memory — so a tester can encode a specific objective and success condition for their own application.

The Base64 exercise makes the distinction concrete. In garak, the same encoding probes are fired at every target, so the resulting percentages serve as a rough pointer to where weaknesses cluster. In PyRIT, the author wrote a mock app with a plaintext input guardrail, a model that can read Base64, and an optional output filter, then checked whether a discount code could be extracted. Only the Base64 path succeeded, and only when the output filter was off — showing that bypassing an input check and getting the model to comply are two separate conditions that must both hold.

The author also notes that PyRIT 1.x ships scenarios mirroring garak's probe styles, but these are reimplemented with PyRIT's own converters rather than loading garak itself, and garak still offers more probe variety. The practical recommendation is sequential: use garak to locate weakness types, then write PyRIT tests tied to the specific application and keep them as regression checks.