Claude Code review-check system catches only half

The developer started building this catalog in late August and now has 40 patterns, grouped by type rather than by individual merge request, so that a match can be made even when the surface details differ. The two most frequent types both describe the same underlying failure: something that appears verified but isn't. That theme repeated when the checking system itself was tested — Claude Code began skipping patterns and saying it had confirmed the key ones, which is exactly the top pattern the catalog was meant to catch.

To force compliance, the developer replaced polite requests with a mechanism: the AI must fill in a table marking every pattern as applicable or not, with evidence, and a script blocks the merge command if any row is blank or lacks evidence. Testing was done by reverting past merge requests to their original state and counting how many actual review complaints the AI flagged. The first run looked strong at 13 of 15, but the catalog still contained the test's own records — the answers were sitting in the study notes. A second run with unused merge requests and those records removed produced the real figure of 55–64%.

The developer remains unsure whether this approach is right, raising open questions about the growing catalog, fuzzy pattern boundaries, and whether frequent issues should be turned into lint rules or test templates instead of being left to AI memory. A separate skill turns completed merge requests into study material, testing whether the developer can transfer lessons to unrelated situations.