AIToday
Large Language ModelsAI Coding AssistantsZenn AI/MLPublished: Oct 17, 2026, 10:00 JST

K-Dense-AI skills test: Claude's answer held, support improved

K-Dense-AI skills test: Claude's answer held, support improved

3 Key Points

  1. What happened

    The author tested K-Dense-AI/scientific-agent-skills (177 skills, MIT) on two skills needing no extra API keys; statistical-analysis kept the same Mann-Whitney U answer but added Welch t(5.03)=1.453, p=.206, d=.954 and MWU U=53, p=.0024, plus an outlier-removed rerun (p<.001, d=2.784).

  2. Why it matters

    The value was backing a verbal conclusion with reproducible figures a reviewer can check, not correcting a wrong statistical decision — the skill mainly changes how well the answer is evidenced, not which test is picked.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, in your inbox every morning.

Context & Analysis

The test was run as day 17 of a daily AI-news verification challenge, following a CLAUDE.md rule to avoid candidates that require paid API keys. That narrowed the 177-skill repository to statistical-analysis (built on Pingouin, SciPy, statsmodels and PyMC) and literature-review (using public PubMed and arXiv APIs). Setup itself surfaced two environment issues: the install command was blocked by Claude Code's auto mode classifier as external code execution until a human allowed it, and installing via Windows Node.js from a WSL UNC path failed with an EPERM mkdir error because the CLI fell back to C:\Windows; running from a non-UNC Windows path succeeded.

The statistical comparison used a fictional pilot with an outlier in Group A. In the no-skill condition Claude Code cited outlier, unequal-variance and small-sample reasons and already named Mann-Whitney U, but gave no test statistic or p-value. With the skill, assumption_checks.py ran and produced descriptives, a Shapiro-Wilk check (Group A p=0.000127, not normal; Group B p=0.248496, normal), a Levene result (p=0.241, variance_ratio=224.37), a Welch t-test and a Mann-Whitney U result, then a sensitivity run excluding the 55. The conclusion matched, but the report now included the figures and a caveat that a non-significant Welch result at n=6 should not be read as no effect.

The literature-review side showed a structure not visible from the README: the bundled script is a post-processing tool, not a search engine. When arXiv returned 0 hits for 'Mann-Whitney AND outlier', Claude Code reworded the query across four patterns and ended with 8 hits under cat:stat.ME AND abs:"Mann-Whitney" AND abs:robust. The skill's own report noted that few papers center on the word outlier and that the hits were adjacent concepts such as robustness and heavy tails.

FAQ
Did the statistical-analysis skill change the recommended test?
No. Both the no-skill and skill runs concluded Mann-Whitney U was appropriate; the skill added actual test statistics, p-values and effect sizes from running the code.
Does the literature-review skill search PubMed or arXiv directly?
No. Its bundled search_databases.py only deduplicates and ranks already-fetched results; the actual API calls are made by Claude Code via WebFetch following SKILL.md instructions.
Were the papers the literature-review skill found real?
The author opened two of the five selected papers directly via WebFetch and found titles, authors and content summaries matched, with no fabrication.

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleClaude Opus 5.5 multi-agent tips: let managers delegate