Motorway & AWS cut AI agent errors 8× with production evaluation blueprint

Traditional LLM evaluation metrics—coherence, factual accuracy, response relevance—do not adequately assess AI agents that orchestrate multiple tools and take autonomous actions. Agent evaluation must verify task completion across multi-step workflows, correct tool selection and parameter passing, reasoning coherence under changing conditions, and reliability consistency (since non-deterministic outputs make single-trial testing unreliable). Motorway's specific challenge illustrates this gap: a query like "Petrol, Hybrid and electric cars up to 5 years old" requires the agent to parse multiple constraints correctly, and semantic search misinterpretations or context drift in multi-turn conversations directly erode dealer trust when real money is on the line.

The solution Motorway and AWS implemented combines two evaluation phases mapped to the GenAIOps lifecycle. Build-time evaluation using strands-agents-evals catches issues before deployment, while production monitoring via Amazon Bedrock AgentCore Evaluations catches failures that synthetic tests miss. The three-layer framework—Layer 1 validating tool usage (>95%), Layer 2 assessing reasoning (>85%), Layer 3 measuring output quality (>90%)—requires all layers to pass before a release proceeds. A key insight for handling LLM non-determinism is the pass^k metric (probability of succeeding in k consecutive trials), which matters far more than pass@k for customer-facing agents; an agent with 75% per-trial success rate has only a 42% chance of passing three consecutive trials, unacceptable when dealers expect consistent quality on every interaction.