AI agents can't find half their own tools without metadata

The team faced an ambitious challenge: close gaps in security, code quality, and system observability for a production coaching platform's API. They designed a three-phase parallel installation. Phase 1 brought in four external security skills: OWASP Top 10 checks, six Trail of Bits audit methodologies (insecure defaults detection, variant analysis, differential review, sharp edges identification, static analysis, and Semgrep rule creation), a destructive-command safety net, and Anthropic's official skill library. Phase 2 added three infrastructure tools: a read-only MongoDB connector for live debugging, Playwright browser automation for end-to-end testing, and a configuration linter for agent setup files. Phase 3 consisted of seven custom skills built from codebase analysis: a controller extraction workflow for decomposing 2,500-line files, React component patterns specific to the codebase, a silent failure detector for swallowed errors, system observability standards, API error handling conventions, per-endpoint security checklists, and canonical test fixtures. Each phase ran as parallel subagents, with the entire installation completing in a single session. Every tool passed its functional test.

Then the team ran a discovery audit. The question was simple: can a future agent, facing a relevant problem, find this tool through automated discovery? The results were stark. Every custom-built skill was discoverable; every externally-sourced skill was invisible. The pattern was consistent and revealing. The seven custom skills had been written with discovery metadata built in — keyword trigger phrases that matching systems use to connect agent queries to relevant skills. The eight external skills, sourced from respected security research firms and open-source repositories, had everything except that metadata.

The fix was absurdly small. Three to five lines of YAML per skill — for example, for the OWASP security skill, adding triggers: "security vulnerability", "injection prevention", "XSS prevention", "security review". Without those lines, a security skill covering the entire OWASP Top 10 sits unused while an agent writes authentication code without security awareness. With them, the skill activates automatically.

The invisibility is silent and dangerous. When an agent builds a tool that other agents cannot discover, nothing visibly breaks. The system does not error. No alert fires. The tool sits in its directory, fully functional, while future agents — facing the exact problem it solves — build their own solution from scratch or proceed without it and ship vulnerable code that the undiscoverable security auditor would have caught. The team identified five independent discovery surfaces: protocol auto-listing, trigger matching, tooling registry, semantic swarm search, and registry semantic search. The eight invisible skills failed at trigger matching and were also absent from the tooling database (Surfaces 3 and 5). Some were partially visible through semantic search, but partial visibility is worse than no visibility — it creates inconsistent behavior where the same query finds the tool sometimes but not others. After patching the eight skills with trigger arrays and fixing seven name mismatches where the internal identifier did not match the folder name, verification showed 15 of 15 discoverable across trigger matching and semantic search. The core lesson: the compounding effect is what makes discoverability dangerous. Every session that builds tools without verifying discoverability adds to the invisible pile. At scale — a thousand tools at a 53% invisibility rate — the gap between what tools exist and what agents can find becomes a strategic problem rather than a housekeeping one.

The discovery problem emerges at the intersection of two dynamics: tools work independently of how they are found, and agents lack the human feedback loop that would expose knowledge gaps. When a human team loses institutional knowledge, someone eventually notices — a new hire asks an unanswerable question, or a process breaks. AI agents have no such signal. A tool that works but cannot be found produces no error, fires no alert, and leaves no trace. Instead, it accumulates silently in the system directory while future agents facing the exact problem it solves rebuild solutions from scratch or proceed without it entirely.

The audit results made the gap concrete: every custom-built skill included discovery metadata because the team anticipated the problem during creation; every external skill lacked it because external sources had no way to know this particular system's discovery requirements. The fix — adding trigger phrases — costs almost nothing at creation time but is expensive to retrofit. The team discovered this through explicit verification, yet the deeper insight is that most systems lack any such verification step at all. Without running a discovery audit, all 15 tools would have appeared to work perfectly, and the 53% invisibility would have compounded indefinitely through future sessions. At scale — a thousand tools — the problem becomes a strategic bottleneck rather than a housekeeping issue.

No comments yet. Be the first to share your thoughts!