WMTrace: Open forensic tool for detecting LLM text watermarks

WMTrace is a forensic workbench for detecting, analyzing, and understanding text watermarks embedded by large language models. It is distributed as an open-source Python package (pip install -e ".[dev]", Python 3.10+) and operates on a principle of evidence over verdicts: rather than producing an opaque AI detection score, it provides versioned evidence bundles with explicit categorical grades (E0–E5) and mandatory scope statements that spell out exactly what the result does and does not prove.

The tool implements five core design principles. First, results must be evidence, not verdicts—every output includes a grade and a scope statement that clarifies what the finding means. Second, four distinct forensic problems (watermark detection, generic AI-text detection, model attribution, and provenance verification) are kept separate in the code, API, and user interface and are never averaged together. Third, abstention is treated as a valid answer: detectors declare themselves unsupported (E0) or insufficient (E1) on short, low-entropy, or incompatible input rather than forcing a binary guess. Fourth, the tool includes no detector for proprietary keys from Anthropic or OpenAI—all keys in the current build are published research keys, and the developers pledge never to add secret-key detectors until an official verification API or reproducible public specification exists. Fifth, documentation is executable: every worked example in the research specification is reproduced exactly by golden tests, so documentation cannot drift from code.

The user interface comes in three forms. The web UI (wmtrace serve) opens a responsive, dark/light, mobile-friendly interface at http://127.0.0.1:8177 with three tabs: Analyze (paste text and get evidence cards per detector, with grade badges, exact statistics including z-scores and Holm-adjusted p-values, per-token green heatmaps, and scope statements), Embed (generate green-biased sample text under the demo key or hide zero-width and lexical payloads in custom text, then round-trip into the analyzer), and About (evidence-grade table, detector manifests, and responsible-interpretation guidance). The CLI offers commands like wmtrace scan document.txt (run all detectors with Holm correction across the run), wmtrace detect --scheme kgw-toy document.txt (check one declared scheme), wmtrace inspect document.txt (Unicode and formatting inspection only), and wmtrace embed --scheme kgw-toy --length 150 --seed 11 (generate a reproducible watermarked sample). The tool also exposes a JSON API (wmtrace scan --json) for programmatic access to full evidence bundles.

WMTrace includes several detector schemes, each marked as illustrative, implementation, or research-grounded. The kgw-toy detector (contextual green-list, γ=0.25, 2-word context) reaches evidence grade E3 and works with a published demo key; the unigram-toy detector (fixed keyed green-list) also reaches E3; zero-width (invisible Unicode payload) reaches E2 and is unkeyed; lexical-codebook (synonym-pair payload bits) reaches E2 with a public codebook; and unicode-inspector (detection of zero-width, bidirectional, unusual-space, and mixed-script indicators) reaches E2 and is keyless. Model-based detectors (SynthID-Text reference, Fast-DetectGPT, Binoculars), C2PA verification, and provider adapters are listed as roadmap items.

The evidence grade taxonomy runs E0–E5. E0 (unsupported) means a required key, tokenizer, or configuration is unavailable; E1 (insufficient) means the configuration is compatible but the input carries too little usable signal; E2 (heuristic) is indicator-level evidence without keys or calibration; E3 (scheme evidence) is calibrated evidence for a declared scheme, key, and configuration; E4 (provider-bound) indicates trusted key provenance binds the result (no detector in the current build reaches this grade); and E5 (cryptographically verified) is reserved for future provenance adapters. A grade is explicitly not a posterior probability of authorship. The research specification, maintained at docs/RESEARCH.md and current as of 2026-08-14, documents the full taxonomy of watermarking techniques, the adoption status of Anthropic, OpenAI, Google, and Meta as of that date, attack and robustness analyses, benchmark methodology, and the P0–P6 implementation roadmap with citations. The tool is released under the Apache-2.0 license and includes 41 golden tests covering examples from the research spec, calibration, round-trips, and API behavior.