Motorway & AWS cut AI agent errors 8× with production evaluation blueprint

Motorway operates a daily online car auction where up to 8,000 dealers bid on up to 2,500 vehicles. Dealers traditionally spent hours manually browsing listings using CSVs and rigid filters. The company built an AI-powered dealer stock search agent to replace this workflow, allowing dealers to submit natural language queries like "Find me diesel SUVs under 25k near my dealership" or "something sporty and automatic for a family." With around 1,500 concurrent users during peak hours, agent reliability is mission-critical.

Motorway and AWS Prototyping and AI Customer Engineering (PACE) team identified a fundamental problem: agent evaluation differs from LLM evaluation. Traditional LLM metrics assess text generation quality—coherence, factual accuracy, response relevance—but do not measure whether an agent selected the correct tool, passed correct filter parameters, or handled multi-turn refinements. The agent faced four specific failure modes: tool selection errors causing wrong search results, semantic search misinterpretations returning irrelevant results (e.g., "Petrol, Hybrid and electric cars up to 5 years old" requires parsing multiple constraints), context drift losing dealer refinements across conversation turns, and non-deterministic outputs making single-trial testing unreliable.

The agent itself runs on the Strands Agents SDK and Amazon Bedrock AgentCore, a fully managed service for deploying and operating AI agents at scale. It exposes eight tools combining structured filtering across over 89 vehicle attributes with vector similarity search powered by LanceDB and Amazon Titan Text Embeddings V2. The end-to-end request flow routes dealer queries through a web interface to Amazon Bedrock AgentCore Runtime, which orchestrates tool calls while using Claude for reasoning and Amazon Titan for embeddings; tool responses flow back through the runtime to generate final results.

The evaluation pipeline reduced incorrect results from 1 in 8 queries to 1 in 50 and cut issue detection time from a few hours to a few minutes. The blueprint implements a two-phase strategy: build-time testing with strands-agents-evals (an open-source evaluation library for Strands Agents) and production monitoring with Amazon Bedrock AgentCore Evaluations. The core assessment uses three layers. Layer 1 (Tool Usage, >95% threshold) validates correct tool selection and parameter passing using deterministic code-based graders like ToolSelectionGrader and TrajectoryOrderGrader; a query like "Diesel vehicles from £7,000 to £20,000" should call search_vehicles with typed filters (fuel_type=diesel, min_price=7000, max_price=20000), while "Modern hatchback with low mileage" should trigger hybrid_search combining semantic embeddings with structured filters. Layer 2 (Reasoning, >85% threshold) assesses logical decision-making using LLM-as-judge scoring (Claude Sonnet 4.6) via HelpfulnessEvaluator and TrajectoryEvaluator; an agent arriving at the right response through illogical reasoning fails unpredictably as conditions change. Layer 3 (Output Quality, >90% threshold) measures response helpfulness and accuracy using LLM-as-judge evaluation via OutputEvaluator and GoalSuccessRateEvaluator; the user must receive a useful, well-formatted response.

The blueprint also introduces a five-stage deployment pipeline with quality gates that block releases when metrics fall below thresholds. To handle LLM non-determinism, the run_all_layers() function accepts a num_trials parameter and employs two metrics from code generation research: pass@k (likelihood of succeeding at least once in k attempts, useful when one correct solution suffices) and pass^k (probability of succeeding in k consecutive trials, critical for customer-facing agents expecting reliable behavior every time). An agent with a 75% per-trial success rate has only a 42% chance of passing three consecutive trials (0.75³), illustrating why pass^k matters for production deployments. A companion repository provides a deployable blueprint adapting the core principles—the three-layer evaluation framework and pass^k metric—to any AI agent; initial deployment takes 30–45 minutes and customization 2–3 hours. Running the sample evaluation suite costs approximately $5–10 in Amazon Bedrock inference charges, with production monitoring costs varying by sampling rate.

Traditional LLM evaluation metrics—coherence, factual accuracy, response relevance—do not adequately assess AI agents that orchestrate multiple tools and take autonomous actions. Agent evaluation must verify task completion across multi-step workflows, correct tool selection and parameter passing, reasoning coherence under changing conditions, and reliability consistency (since non-deterministic outputs make single-trial testing unreliable). Motorway's specific challenge illustrates this gap: a query like "Petrol, Hybrid and electric cars up to 5 years old" requires the agent to parse multiple constraints correctly, and semantic search misinterpretations or context drift in multi-turn conversations directly erode dealer trust when real money is on the line.

The solution Motorway and AWS implemented combines two evaluation phases mapped to the GenAIOps lifecycle. Build-time evaluation using strands-agents-evals catches issues before deployment, while production monitoring via Amazon Bedrock AgentCore Evaluations catches failures that synthetic tests miss. The three-layer framework—Layer 1 validating tool usage (>95%), Layer 2 assessing reasoning (>85%), Layer 3 measuring output quality (>90%)—requires all layers to pass before a release proceeds. A key insight for handling LLM non-determinism is the pass^k metric (probability of succeeding in k consecutive trials), which matters far more than pass@k for customer-facing agents; an agent with 75% per-trial success rate has only a 42% chance of passing three consecutive trials, unacceptable when dealers expect consistent quality on every interaction.

No comments yet. Be the first to share your thoughts!