Claude Code's claude plugin eval scores plugins at +0.46 mean delta

The article is a third installment of an applied Claude Code series that organizes videos #11 through #15, and it picks up exactly where the earlier parts left off. Skills and hooks were built in part one (the test-and-report skill and the hook that blocks rm -rf), and here those same two pieces are repackaged into a single plugin named safe-dev, stored in the public repository zun-tools/ccadv-sample under an MIT license. The packaging step is what makes the measurement step possible: once a skill lives in a plugin with a .claude-plugin/plugin.json, claude plugin eval can compare results with and without it.

The demonstration is deliberately small. A skill that forces meeting notes into the three headings 決定事項, 宿題, and 担当 is measured against three meeting-note cases (support, release, and sales), each graded partly by regex and partly by a separate model reading the answer. The interesting detail is that the without-plugin runs did produce reasonable summaries — what they failed to do was hold the three-heading format. That is why the author frames the score gap as evidence for keeping a skill rather than as proof the model cannot do the task at all.

Distribution and safety get similar treatment. Plugins are installed from a marketplace with two commands, and the installer chooses whether the plugin applies to all of the user's repositories, the whole team on that repository, or just that user locally. The article warns that bumping the version in plugin.json is required before pushing an update, since otherwise the installing side reports it is already up to date and keeps the old version. For plugins written by other people, mods in particular, the recommended step is to inspect what events they react to and what operations they request with claude plugin validate before installing, because mods run outside the sandbox with the same permissions as Claude Code.