AWS ties Kiro prompt tuning to LLM judge, A/B tests

AWS's disclosure centers on a problem it frames as inherent to agents like Kiro, which combine a model, tools, a codebase, tasks and users. Because those ingredients change, the company argues no prebuilt test suite can validate every combination. Its answer is a four-stage loop: diagnosing recurring failure patterns from internal traffic, designing changes around recall, control, tools or coding, testing them with control and cohort groups, and reviewing LLM-judge scores per cohort.

The LLM judge scores conversations on 15 criteria, including task completion, whether Kiro split off a conclusion before verification, and whether it ran code or checked compilation. AWS separates results into textual dissatisfaction and product-issue signals, and it says judgments are revised only when there is objective evidence, to limit hallucinations.

In past validation, AWS screened 27 changes at once and found that applying them together instantly degraded performance, which is why it favors hash-based splitting and A/B comparison. It also applied the mechanism to reasoning-effort settings, lowering the setting to see whether quality improvements from more effort stopped. The initial prompt-and-structure comparison produced the reported reductions in unspecified errors and style mismatches, but prompt-only gains from a model update were limited to 4%. AWS's own reading is that the new model simply made fewer mistakes, leaving less room for prompt improvement, and it later found cases where the model followed instructions too literally and produced unintended side effects. The practical implication is that prompt quality may look like a one-time setup but behaves like an operational cycle, and the test of AWS's approach is whether retuning the model and prompt together keeps improvements from backsliding.