Mutation generator misses test holes structural mutations catch

The author's setup is a pre-push self-review run by an AI agent, which reads the implementation and hands a JSON list of breakages to a harness that applies them one at a time and runs tests. At a few hundred mutations per review, writing the mechanical swaps by hand consumed much of the agent's time and tokens, so a generator based on Python's tokenize library was added. That is where the two jobs split: the generator reliably produces substitution-level mutations, while choosing which information to discard so the fix stops working is a different task.

The replay evidence is thinner than a controlled experiment. The original hand-written mutation JSON was lost along with a temporary workspace, so what remained were PR descriptions and commits, and the table's counts are rounded to convey scale rather than to compare detection rates. What the record does show is that in the version of Fix B with the added tests removed, the generator did not reproduce the structural hole, because its substitution table has no operation that changes how a set is built, swaps an argument for another set, or drops a branch entirely.

The proposed operating rule keeps both roles. The generator covers a declared, narrow range — comparison, logical and boolean, arithmetic, and integer-literal swaps — while the AI agent reads that range and its exclusions and separately thinks up argument mix-ups, ordering swaps between filtered and full collection, and disabling multiple entry points at once. The harness first confirms the unmutated tests pass, runs narrowed tests where a record exists, then reruns any green mutant against the full suite; a green across everything goes back to the agent to classify as a hole, an equivalent, or unclassified. Switching away from this setup is tied to reproducing known holes, not to a calendar date.