ProofRun: cryptographic proof AI agents actually ran tests

ProofRun is an open-source command-line tool that cryptographically binds test and build check results to the exact code state, solving a trust problem in AI-assisted development. When an AI coding agent claims "all tests pass," it may be truthful, but it could also be inferring based on an earlier run or guessing—there is no way to tell the difference from the words alone. ProofRun closes that gap by making the claim itself verifiable: it runs the real command (e.g., pytest) in a subprocess, reads the actual exit code, and stores the result tied to a fingerprint of the current code.

The fingerprint is the key mechanism. Every result is bound to the git commit hash plus a SHA-256 hash of all uncommitted changes (staged or not, tracked or not). If so much as a single byte changes—a space, a new line, anything—the result automatically becomes STALE. The tool recomputes the fingerprint every time proofrun status is called and compares it against the stored receipt. This means developers never have to remember to ask "does this PASS still count?" The system answers automatically and correctly.

ProofRun supports four statuses: PASS (the command succeeded), FAIL (the command failed), STALE (the result is outdated because code changed), and NOT RUN (the check has never been executed). Notably, there is no fifth status for "probably fine" or "I think it passed." Each status comes from observed execution or documented absence. The tool makes no LLM calls anywhere—it does not use AI to verify AI. Instead, it starts a real subprocess and reads its exit code; that is the entire verification mechanism. It is fully offline with zero network calls and zero telemetry.

Configuration is minimal. Users create a .proofrun.yml file declaring named checks, each with a command (given as an argv array, not a shell string) and a required flag. For example, a test check might declare command: [pytest] with required: true, and a build check might declare command: [npm, run, build]. When proofrun run test -- pytest is executed, ProofRun runs pytest for real and binds the result. When code changes and proofrun status is called, the tool reports STALE for any check whose fingerprint no longer matches. The --strict flag makes proofrun status exit with a non-zero code if any required check is not PASS, making it usable as a pre-commit hook or CI gate.

ProofRun was built by Claude Code (an AI agent) under human direction, then underwent several rounds of independent, read-only adversarial review before the first release. That review found a vulnerability: ProofRun's command comparison could be tricked by a misquoted shell argument, allowing a check to silently run zero tests and still report PASS. The developers documented the full repro and the exact fix in docs/case-study.md, and every fix was verified against a real reproduction before being accepted—not just reviewed for plausibility. This scrutiny is intentional: a tool designed to hold AI agents accountable cannot exist if it cannot survive the same standards applied to itself.

A GitHub Action (yebiguo/proofrun@v1) is available for pull requests. It checks out the exact PR head commit (never trusting a synthetic merge-preview), clears any receipt.json that came in on the PR branch, downloads a checksum-verified ProofRun binary, runs proofrun run-all for real, and gates on proofrun status --strict. Nothing about a receipt checked into the PR is trusted—every result is produced fresh by the Action. One known limitation: the Action warns (via a build annotation) if .proofrun.yml differs from the base branch but does not block on that, since a PR could loosen a check's command and the Action would faithfully run the weaker version. Teams are advised to review the .proofrun.yml diff like any other change.

The roadmap includes structured output support for common test runners (pytest, Jest, JUnit) in v0.3, signed and tamper-evident receipts (on the radar but not yet designed), and protection for .proofrun.yml itself to prevent it being weakened within the same PR that changes the code. The project is licensed under MIT and deliberately narrow in scope—a young, pre-1.0 tool with a focus on STALE detection and the receipt schema as its most critical guarantees.