Epistemic Engine: open-source tool verifies AI code, forecasts failures

Epistemic Engine is an open-source Python tool (Apache-2.0 licensed, authored by Aswin S., copyright 2026) that analyzes a codebase by reading it as a system of justified beliefs—formal or informal claims the code makes about itself—and reconstructs how well-justified each claim is by examining the git history. The tool then tracks how that confidence changes over time.

The toolkit provides three user-facing features. ee guard is a pre-commit or CI verifier that extracts the beliefs a code change introduces and blocks on unsafe or unjustified ones; it also layers on direct OWASP-class scanners and known-CVE matching. It catches weak crypto (md5, sha1, mt_rand used as security primitives), injection (SQL built by string concatenation, os.system/popen on user input), unsafe sinks (eval, innerHTML, unserialize, pickle.loads on untrusted data), hardcoded secrets, and misconfigs (TLS verification disabled, Access-Control-Allow-Origin: *, debug mode enabled). Each finding carries a severity, the reason, a concrete fix, and a stable fingerprint. The tool supports adoption features including inline # ee-ignore[rule] suppression, a disabled_rules config, and baseline mode (--update-baseline / --baseline FILE) so teams can adopt on existing codebases without fixing everything at once. Field testing on 1,500+ real production files reported precision / recall / false-positive rates of 1.0 / 1.0 / 0.0 (95% CI lower bound 0.92 on a 114-sample corpus).

ee predict-chain forecasts which beliefs are about to collapse by projecting confidence erosion forward. For each at-risk belief it outputs a causal chain (ranked factors driving collapse, tagged as measured from repo history or assumption), a collapse probability within a given horizon computed via seeded Monte Carlo, a calendar ETA derived from the repo's own commit cadence, and a fix plus ROI computed under user-provided cost assumptions. Importantly, the tool refuses to lie: ee calibrate runs a leakage-free back-test against the repo's own history and reports a Brier skill score versus baseline; if the model doesn't beat guessing on that specific repo, predict-chain prints that verdict and declines to output probabilities. It never fabricates probabilities for undisclosed future CVEs and never presents a dollar figure as fact.

ee dashboard provides a local, offline, self-contained web UI with three tabs: Beliefs (every claim the code makes, grouped by domain, colored by confidence), Predictions (collapse forecasts with the calibration verdict), and Guard (ee guard findings grouped by severity with a BLOCK/PASS banner). Installation is via pip install epistemic-engine (core, zero heavy dependencies) or pip install "epistemic-engine[parsing]" to add tree-sitter for entity-level beliefs; requires Python 3.10+. All computation is deterministic and offline—no LLM, no sampling, no network access except explicit ee sync—making results reproducible and suitable for CI. The project is backed by 167 tests.

Epistemic Engine frames code verification around the concept of beliefs—formal and informal claims the code makes about itself ("this function authenticates the caller", "this value is never null")—and reconstructs their justification from git history. This is a meaningful shift from traditional static analysis: instead of pattern-matching vulnerability signatures, the tool models how well-founded a piece of code is and how that confidence erodes as the codebase evolves. The dual tool design (ee guard for immediate blocking, ee predict-chain for long-term forecasting) addresses two separate use cases: teams reviewing AI-generated code can catch obvious flaws before commit, while maintainers can forecast which subsystems are at risk of collapse and prioritize remediation by cost-benefit.

The tool's honesty-first design—refusing to output a prediction if back-testing shows it doesn't outperform baseline, labeling all cost figures as assumptions, and explicitly reporting when it can't trust its own forecast—reflects a deliberate choice to avoid the temptation to appear confident when uncertainty is real. Field testing on 1,500+ production files with reported precision / recall / false-positive rates of 1.0 / 1.0 / 0.0 (lower bound 0.92 on the validation set) suggests the guard tool has genuine practical accuracy, though the lower bound on a 114-sample corpus is the more conservative metric. The implementation is deterministic and offline throughout, with no LLM or network dependency in any computed value, making results reproducible in CI and suitable for long-term code audit trails.