Muteval: mutation testing tool measures if your LLM evals actually catch bugs

Mutation testing is an established practice in software engineering: deliberately introduce bugs (mutations) into code, rerun your test suite, and measure how many bugs the tests catch. Muteval brings this discipline to LLM eval suites, addressing a structural blind spot: passing evals do not guarantee the evals themselves are comprehensive. The tool deliberatedly degrades the system under test across 18 operators — weakening modal language ("ONLY" → "preferably"), deleting sentences, dropping retrieved context documents, corrupting tool outputs, or downgrading the model — and reruns your existing checks against each variant. A mutant is "killed" if your evals catch the degradation; a "survivor" means your evals missed it, flagging a potential coverage gap. The body reports a real example: a support-bot eval with a 33% mutation score (95% CI 10–70%) where the suite missed two regressions — one HIGH-severity (failure to check that answers are grounded in context) and one medium-severity (a weakened modal constraint). Each survivor is paired with a suggested eval (`checks.grounded("context")`) to close the gap. The tool integrates with existing frameworks (promptfoo, deepeval, RAGAS, custom functions, deployed endpoints) and includes a confidence interval calculation and severity ranking to avoid misleading scores. A separate probe audit (`muteval probe`) checks judge reliability, discrimination, and statistical adequacy — treating the eval-quality assessment as a report card rather than a single composite score.