AIToday
Zenn AI/MLPublished: Nov 21, 2026, 16:00 JST

Stop letting AI write tests; write rules instead in Quint

Send on LINE
Stop letting AI write tests; write rules instead in Quint

3 Key Points

  1. What happened

    In a write-up on Zenn, a developer used the TLA+-family language Quint to state one rule — tickets sold never exceed initial stock — and model-checked a ticket-sales design. Quint found an oversell violation in 12ms, after the AI-written implementation and its 3 AI-written tests all passed.

  2. Why it matters

    An AI writing both code and tests grades its own homework — the author's "self-scored perfect" — since the tests only cover scenarios the same model imagined; keeping the rule-making human while machines generate the cases is his proposed division of labor.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Context & Analysis

The piece is a practitioner's walkthrough rather than a product announcement, and its structure is a single worked example. A ticket service is specified in Quint with three users and two tickets; the invariant noOversell states that sold never exceeds the initial stock. The AI-written implementation passed its own three tests, then failed 578 of 1,000 randomly generated traces — a gap the author attributes to the same model having invented both the code and the scenarios it was checked against.

The author places this in the long-standing Test Oracle Problem: tests need both an input and a way to judge the result correct, and the judgment is the hard part. His examples of weak substitutes are concrete — a multi-agent adversarial review once returned 47 nitpicks while missing the real bug, which he likens to a classmate from the same cram school writing the exam. TLA+, created by Turing Award winner Leslie Lamport, and AWS's reported use of it on S3 and DynamoDB designs supply the pedigree; Quint is presented as the approachable, TypeScript-like syntax with the same TLC checker underneath.

The conclusion is deliberately scoped. Test cases do not disappear, the author says — they remain fast, readable "living documentation" for simple screens. Rules are for the places where concurrency, money, inventory or permissions make exhaustive human enumeration impractical, and the suggested starting posture is to write them only where interruption bugs feel frightening.

FAQ
What exactly did the AI-written tests miss?
All 3 tests covered users buying one at a time in sequence. None covered a second user confirming a purchase after someone else bought the last ticket in between — a race the author admits he might not have thought of either.
How strong is the guarantee after the fix?
For the small world of 3 users and 2 tickets, quint verify checked all 29 reachable states with 0 states left in the queue — "tested everything" rather than "tested a lot and found nothing." The author notes this does not prove correctness at any scale.
Can AI help with the Quint work itself?
Yes — Quint ships an official skill for AI agents such as Claude Code, installed via a plugin marketplace command, so you can ask an agent to transcribe code into Quint. The Quint team's line: LLMs don't think, they translate; the deterministic tools do the reasoning.
Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Next articleContra Pace: Ben Pace's one-graph case for LessWrong's writer decline