Why did the adversarial review rounds stop improving?

From the second round on, each reply began with 'not committable' and raised a new counterexample using a different syntax to break the same identifier logic. Most later findings were synthetic code not present in the actual product source.

What mechanism fixed the false-positive problem?

The team wrote 10 assumptions (later 13), each with a reason and a detection method, and built syntax-tree detectors that flag any source violating an assumption and mark the whole measurement as failing rather than crediting unknown elements as covered.

How many real fixes came out of the whole process?

Across the 15 rounds after the first few, real-data measurements moved on only one element, from 387 to 388. Two fixes affected real data: correcting handling of an external product's components and fixing double-counted table rows.

Large Language ModelsAI Coding AssistantsZenn AI/MLOct 7, 2026Oct 7, 2026, 22:00 JST

Codex review rounds stalled until TOKIUM QA set stop rules

Send on LINE

3 Key Points

What happened
A TOKIUM QA engineer ran 22 Codex review rounds in two days building an E2E-test coverage tool with Claude Code; in rounds 5–6 the fix pushed covered elements from 388 to 1 instead of zeroing false positives.
Why it matters
Without assumptions and a stop rule, adversarial review keeps surfacing synthetic counterexamples that never change real measurements, so the team added three finding categories and a rule basing blockers only on real-data misjudgments.
What to watch
The tool shipped with all 956 related tests passing and now runs in normal operation, with 13 documented assumptions and a detector that found 0 violations in the real product source — but unproven impossible cases are kept as known constraints.

WHO IT HITSQA and test-automation engineers who point a second AI model at their own code will find that the review's value depends on the acceptance criteria they hand over, not on the reviewer's intelligence; teams that skip those criteria risk burning days on review loops that change nothing in production data.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, in your inbox every morning.

Get news like this every morning, free

Context & Analysis

The engineer was building a tool that reads a React/TypeScript frontend to find interactive elements, then matches E2E test records against them to measure how much is actually exercised. Because a false positive — crediting an element that was never operated — is the worst error for a measurement tool, the implementation was handed to a different vendor's AI, Codex CLI, for adversarial review without the generation context.

Early rounds paid off: the reviewer reproduced an error that confused operations from another environment and demonstrated 5 incorrect credits in a measurement report. Then the dynamic shifted. Fixes landed, but new counterexamples kept appearing in the same class of syntax, and the engineer moved the variable-binding resolution to the TypeScript compiler API, only to be caught by type-forged values via as unknown as or any. The decisive round came when a policy of trusting only syntactic constants collapsed coverage from 388 elements to 1 — proof that maximum caution can make the instrument useless.

The resolution was to write down which assumptions the tool is allowed to trust (for example, no getters or setters on attribute objects carrying test identifiers), justify each with library implementation lines, and build syntax-tree detectors that flag violations. The review request was restructured to force each finding into one of three categories: fails despite satisfying assumptions, detector misses an assumption violation, or the assumption's evidence is wrong. Counting dipped from a peak of 8 blockers to 3 by round 8, then bounced back to 5 in round 9 — evidence that findings shrink slowly even with the new frame. The remaining blockers were resolved by treating only real-data misjudgments as blockers, listing synthetic-only counterexamples as known constraints with a note on how many times that pattern appears in the actual source, and switching the value-flow tracking to a single engine with a syntactic allowlist. The tool was adopted with all 956 related tests passing, and the engineer now describes adversarial review as a dialogue that grows the tool rather than an endless search for counterexamples.

FAQ

Why did the adversarial review rounds stop improving?: From the second round on, each reply began with 'not committable' and raised a new counterexample using a different syntax to break the same identifier logic. Most later findings were synthetic code not present in the actual product source.
What mechanism fixed the false-positive problem?: The team wrote 10 assumptions (later 13), each with a reason and a detection method, and built syntax-tree detectors that flag any source violating an assumption and mark the whole measurement as failing rather than crediting unknown elements as covered.
How many real fixes came out of the whole process?: Across the 15 rounds after the first few, real-data measurements moved on only one element, from 387 to 388. Two fixes affected real data: correcting handling of an external product's components and fixing double-counted table rows.

Zenn AI/MLRead Original Article

Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Get it free

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.