AI Benchmark Maker Agent Seer Synthesizes Scenarios from MCP Specs

Evaluating AI agents that call external tools typically requires hand-crafted test scenarios, which are costly to build and become outdated as APIs evolve. Agent Seer addresses this by generating scenarios directly from MCP specifications, which already encode enough semantic detail to produce realistic tests. The pipeline enriches raw schemas, generates graded scenarios with synthetic outputs, and expands them into multi-turn dialogues, all without live tool execution or domain-specific tuning.

The method was tested on seven MCP specifications covering diverse domains and tool-suite sizes. It achieved strong quality across all domains and complete tool coverage on smaller specifications. Two insights stand out: parameter schema complexity is the strongest predictor of quality variation, while tool-suite size plays a minor orthogonal role. Additionally, argument value accuracy is the most common error in imperfect scenarios, a nuance that coarse name-match metrics miss.

These findings suggest that future benchmarks should pay closer attention to parameter complexity and the accuracy of argument values, which are often overlooked. Agent Seer was accepted at the Fifth Workshop on Natural Language Generation, Evaluation, and Metrics at ACL 2026, signaling academic interest in scalable scenario generation for tool-using agents.