AWS announces AgentCore Optimization preview: automated recommendations and A/B testing for agent performance

Amazon Bedrock AgentCore now includes three new capabilities: a Recommendations API that analyzes production traces to propose changes to system prompts or tool descriptions, batch evaluation to test recommendations against curated datasets, and A/B testing through AgentCore Gateway to validate changes against live traffic with confidence intervals and p-values.

The system replaces manual debugging cycles where developers read traces, form hypotheses, and test fixes by hand. Traces are captured as OpenTelemetry-compatible records; recommendations propose changes weighted by a specified evaluator (built-in or custom); and validation happens either offline against known test cases or online by splitting production traffic between control and candidate versions.

Winning configurations from A/B tests become the new baseline, and their traces feed into the next optimization cycle. The service is designed for developer control: you choose when to generate recommendations, which evaluator to optimize for, and whether to promote results.