AWS releases Agent-EvalKit, an open-source toolkit that helps development teams systematically test AI agents by tracing their full execution—tool calls, data returned, and reasoning steps—rather than just checking final outputs.

What happened

AWS published Agent-EvalKit, an Apache 2.0 licensed open-source toolkit that integrates with AI coding assistants (Claude Code, Kiro CLI, Kilo Code) to evaluate AI agents across six phases: planning evaluation metrics, generating test cases, adding execution tracing, running agents against tests, computing evaluation scores, and producing code-level improvement recommendations.

Why it matters

Most teams evaluate AI agents by checking whether outputs look correct, but agents can produce plausible answers while hallucinating facts or skipping verification steps. The toolkit addresses this gap by making it practical to inspect the agent's full decision path—which tools it called, what data returned, and whether responses match that data—without teams having to build evaluation infrastructure from scratch.

What to watch

The toolkit works through existing development environments (your AI coding assistant) rather than as a separate platform, and supports multiple agent frameworks including Strands, LangGraph, and CrewAI. It uses natural language guidance to let you focus evaluation on your agent's specific failure modes, such as hallucinations triggered by empty tool results.