LangChain details eval-driven development in new guide

LangChain frames the problem in familiar terms: hand-testing a prompt works at first, but as an agent grows it becomes hard to tell whether the latest change helped or hurt. Its answer borrows the discipline of test-driven development, adapted because agents are non-deterministic — the same question can get a different answer, and more than one answer can be right.

The guide's worked example is an internal docs agent adding a create_ticket tool. After the tool is built, the average score rises, but a how-to question about VPN connection starts filing a ticket instead of searching the docs. The trace points to a tool description that is too broad; narrowing it to requests needing IT action gets every example passing.

LangChain also warns that a perfect score on a small dataset is not always the target, since tuning against the same examples can overfit. It suggests automation — fast checks on pull requests, slower LLM-judge evals nightly or before release — plus production-time scoring of live traces, annotation queues, and dashboards to catch regressions after shipping.