COMPASS QA engineer triages AI test fails with Code / Limit / Defect

A previous piece argued against treating AI-generated E2E tests as a simple Pass/Fail binary. Here the author focuses on what happens after a test fails: the sorting work that becomes the new bottleneck once tests can be generated in volume. The core design choice is to make Defect a high-bar label. A timeout message, for example, can mean a changed selector, an insufficient wait, or a genuinely broken feature, so the system declines to guess from the string alone. Mechanical rules handle signals like intercepts pointer events, and a higher-tier AI re-reads logs, screenshots, and test code for the rest; what remains unsettled is held for a person. Reproduction checks are not retries but a separate AI driving a real browser through MCP, with the original signal withheld to avoid bias. The body also notes that skip reasons are made machine-readable, with category and autoSolvable fields, and that the framework and project-specific rule data are kept apart so other services can reuse the classifier. The author lists six open items, including the waitForResponse confidence sitting exactly at the auto-healing threshold, the risk that an AI mislabels a missing element as Code and heals over a real bug, and the inability to distinguish data shortage from environment failure. The author says quantitative verification of effort reduction is still to come and that current confidence is based on operational feel.