Open lab tests AI coding sandboxes on 196 contracts

The lab's premise is that a sandbox's marketing claims are only meaningful if they can be reproduced. Each contract therefore bundles an attack, a control and a post-hoc effect check, and the lab insists the attack must succeed without a sandbox on the same host before a "blocked" result is allowed to count. The corpus is versioned and credited: v1.1 (2026-10-26) holds 196 contracts drawn from the public test suites of sandbox-runtime, nono, Codex, Deno, bubblewrap and OpenHands, with by-omission variants and six masked-credential contracts.

The published runs show the lab iterating on its own methodology as much as on the sandboxes. The 2026-10-16 run was the first judged by spec/verdicts.md; the old runner reproduced the 2026-10-15 results on a fresh host, while the new one changed five, each traced to a cause. A by-omission run re-measures the read-deny contracts in a capability sandbox's own model, so nono is measured where the neutral model only returned unsupported.

The author's own project, TartarOS, is both a subject and a motivation: escapes the lab finds are tried there to see whether they can be mitigated, and TartarOS has been measured alongside the others since the 2026-10-22 run, with the conflict of interest recorded. Reports sent to each vendor's security program as of 2026-10-27 are listed — sandbox-runtime 4 reported and 1 still open, codex 2 and 1, nono and landrun none — and open reports are not published until the vendor fixes them.