iFixAI open-source tool grades AI agents on business impact in 120 seconds

iFixAI is a third-party auditing platform shipped as open source that evaluates AI agents by running 45 diagnostic inspections and delivering a letter grade with a five-pillar scorecard. The tool organizes those 45 tests into five core pillars—Fabrication (unsourced claims, missing audit trails, tool misuse), Manipulation (privilege escalation, policy violation, prompt injection, poisoned retrieval), Deception (sandbagging, secret side-stepping), Miscalibration (overconfidence, wrong confidence calibration), and Drift (behavior drift from intent or training)—plus eleven premium categories. A single run completes in under 120 seconds and outputs JSON and Markdown reports plus a rich terminal scorecard.

iFixAI runs three ways with the same underlying diagnostic. The guided CLI wizard (recommended for first-time users) prompts the operator to pick the provider (OpenAI, Anthropic, Gemini, OpenRouter, etc.), model, judge, and test suite via arrow keys, then saves the configuration to ifixai.yaml, so subsequent runs need no flags. The explicit-flag CLI mode is fully scriptable for CI and automation: `ifixai run --provider anthropic --api-key "$ANTHROPIC_API_KEY" --fixture ./my-fixture.yaml`. The agent skill (one-time setup via `uvx ifixai install --agents cursor`) scaffolds a native `/ifixai-skill` command into any of seven IDEs—Claude Code, Codex, Cursor, VS Code, Windsurf, Cline, Continue, Gemini, Zed, and Continue—so the agent discovers its own config, names the cost before any billing, and walks the operator through the scorecard interactively.

The test suite can be set to smoke (3 tests, pipeline check only), strategic (8 tests, quick high-risk read), core (32 tests, the graded five-pillar scorecard), extended (13 tests, frontier risk outside the grade), or all (45 tests, the default). For a bare model (the simplest case), a single-judge grade with Claude Sonnet costs roughly $12–18 per full suite. A two-judge ensemble—Gemini 2.5 Pro and GPT-5.4-mini on OpenRouter—costs ~$10–14 combined and is more affordable than Sonnet alone while adding cross-vendor robustness; ties break conservatively (fail > partial > pass, so no one vendor decides the grade). A citable grade requires two API keys from different providers: the system under test's key (passed explicitly via `--api-key`) and the judge's key (auto-paired from the environment). The SUT never grades itself; if only one key is available, the run executes but is flagged `--eval-mode self` (smoke-test only, non-citable). Full mode with a multi-judge ensemble requires a hand-built fixture in YAML; the CLI defaults to the bundled default fixture, which ships with seeded defects so users can see what failures look like.

iFixAI treats the agent as a black box reached through a thin adapter. For agents serving an OpenAI-compatible HTTP endpoint, `--provider http --endpoint <agent-url> --grounding sut` measures the governance the agent already enforces (system prompt, tools, retrieval, guardrails included). For agents elsewhere, implementing one method—`ChatProvider.send_message`—and optional capability hooks (list_tools, get_audit_trail, authorize_tool, retrieve_sources) allows iFixAI to score more inspections; incomplete adapters result in `insufficient_evidence` marks (reported but not counted for or against the grade). The tool supports OpenAI, Anthropic, Gemini, OpenRouter, Azure, Bedrock, Hugging Face, HTTP, and LangChain adapters; installation is `pip install "ifixai[openai]"` or the matching provider extra. Configuration is stored in ifixai.yaml (git-ignored by default) with the API key as an environment-variable name, never the secret itself.