How do I run AgentCheck?

Install with `pip install -e .`, set your ANTHROPIC_API_KEY, and run `agentcheck run examples/tests.yaml`. Test cases are defined in YAML and specify either a command (run as subprocess) or a Python function (agent: module.path:function_name).

Can AgentCheck compare results across runs?

Yes. Use `--baseline baseline.json` to diff the current run against a previous one. Results are bucketed as unchanged, regressed (baseline passed, now fails), improved (baseline failed, now passes), new, or removed. The PR comment automatically highlights regressions when `--post-pr-comment` is also set.

Is this a production observability tool?

No. AgentCheck is deliberately narrow—it is a pre-deployment regression check for CI, not a production-observability platform. The creator avoids competing head-on with Langfuse, Braintrust, and Arize, which handle logging and monitoring production agent traffic.

Large Language ModelsAI Coding AssistantsOpen-Source AIHacker NewsAug 21, 2026Aug 22, 2026, 06:00 JST

AgentCheck: regression testing for AI agents as a CI step

Send on LINE

3 Key Points

What happened
AgentCheck, a new open-source CLI tool, lets developers define test cases in plain YAML, run them against an AI agent (via subprocess or Python function), and use an LLM to score pass/fail on each response. It integrates into GitHub Actions and can post PR comments showing test results and diffs against a baseline.
Why it matters
Observability platforms have raised $50–80M rounds recently and dominate the production-monitoring space, but almost none offer a lightweight, pre-deployment regression suite that fits into CI in five minutes the way pytest does. AgentCheck targets that gap—developers can catch prompt changes, tool swaps, or model upgrades that silently break agent behavior before they ship.
What to watch
The creator plans to dogfood the CLI this week, open-source it next week in developer communities (r/LocalLLaMA, LangChain/LlamaIndex Discords, Hacker News), and then add the most-requested feature in weeks 3–4 (likely a hosted dashboard, Slack/GitHub reporting, or framework-specific integration). Monetization via a paid hosted tier comes only after establishing free users.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, in your inbox every morning.

Get news like this every morning, free

Context & Analysis

AgentCheck addresses a specific gap in the AI-agent testing landscape. Observability and evaluation platforms—Langfuse, Braintrust, Arize—have raised $50–80M rounds in the past year and dominate the post-deployment monitoring space. However, almost none of them offer a fast, developer-friendly regression suite that integrates into CI workflows the way pytest does for Python testing. The creator explicitly avoids direct competition with these well-funded platforms, instead positioning AgentCheck as a lightweight, git-native tool for solo developers and small teams to catch breaking changes before they reach production.

The distribution strategy reflects this positioning: the tool is open-source, self-serve, and targets developer communities (r/LocalLLaMA, Discord channels, Hacker News) rather than enterprise sales. The roadmap prioritizes early user feedback over rapid monetization—the plan is to establish a user base, validate the core tool, and only then introduce paid features (hosted dashboard, team sharing, trend charts) based on what free users would miss. This approach acknowledges a consistent weakness among solo AI founders: enterprise sales cycles and large upfront build-outs often fail when the core problem and audience are not yet proven.

FAQ

How do I run AgentCheck?: Install with `pip install -e .`, set your ANTHROPIC_API_KEY, and run `agentcheck run examples/tests.yaml`. Test cases are defined in YAML and specify either a command (run as subprocess) or a Python function (agent: module.path:function_name).
Can AgentCheck compare results across runs?: Yes. Use `--baseline baseline.json` to diff the current run against a previous one. Results are bucketed as unchanged, regressed (baseline passed, now fails), improved (baseline failed, now passes), new, or removed. The PR comment automatically highlights regressions when `--post-pr-comment` is also set.
Is this a production observability tool?: No. AgentCheck is deliberately narrow—it is a pre-deployment regression check for CI, not a production-observability platform. The creator avoids competing head-on with Langfuse, Braintrust, and Arize, which handle logging and monitoring production agent traffic.

Hacker NewsRead Original Article

Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Get it free

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.