AWS open-sources Universal Agent Evaluation Framework (UAEF)

The release addresses a practical problem AWS describes in its post: teams building AI agents on AWS use a range of frameworks, including LangGraph, LangChain, Strands Agents, and Amazon Bedrock AgentCore, and each records agent execution in its own way. Evaluation tooling has tended to follow the same boundaries, so teams build metrics, scoring logic, and result storage against a single framework's trace format and must rebuild them when they switch frameworks or run agents on more than one. UAEF's approach is to normalize first and score second, through adapters that convert each framework's native execution into a canonical AgentTrace. Because every metric, mode, and report operates on that AgentTrace rather than a framework-native format, the scoring that follows is identical no matter where the trace came from. AWS says customers across 12 organizations in telecom, healthcare, and financial services used UAEF to evaluate their agents with one consistent set of metrics and compare versions before release. The system also records each evaluation as an experiment in Amazon DynamoDB and Amazon S3, with a comparison engine that classifies metrics as a regression, improvement, or unchanged using a configurable threshold defaulting to 5 percent. UAEF can be used two ways over the same engine: through a Python library for notebooks, scripts, and CI/CD pipelines, or through a deployable AWS service with a low-code web interface, and both paths write to the same stores so a result does not depend on which path produced it.