Dosu's Devin Stein: LangSmith scales evals across thousands of repositories

Dosu's account traces a common arc for AI products: what works at low volume stops working at scale. Early on, the team inspected every response by hand with grep and print statements, which helped shape Dosu's architecture. But as Dosu spread to thousands of repositories, that level of review became impractical, and the team needed a way to keep measuring the impact of changes.

The company's answer was evaluation driven development, where a set of evals acts as the baseline for any change to Dosu's core logic, models, or prompts. LangSmith supplied the production monitoring and search to find new failure modes, which then feed back into those offline evals. Dosu is now working to automate dataset collection from live traffic, which could shorten the loop further.

The post also notes a cooperative dynamic with LangChain: better Dosu performance reduces support burden on the LangChain team, freeing time for LangSmith work that in turn speeds Dosu's development.