AIToday
Large Language ModelsAI Coding AssistantsZenn AI/MLPublished: Nov 1, 2026, 19:01 JST

AI skips ./bin/scan unless told; 1 line fixes it

Send on LINE
AI skips ./bin/scan unless told; 1 line fixes it

3 Key Points

  1. What happened

    With the wrapper placed and documented but no instruction line, 0/4 sub-agents called ./bin/scan; adding one line made it 4/4, and even a strongly worded instruction left 1/4 returning to raw tools.

  2. Why it matters

    The experiment suggests instruction strength is not what keeps an AI using your wrapper; what matters is whether the wrapper outputs everything the AI needs, or the AI will switch back to raw tools.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Context & Analysis

The experiment was designed to separate whether an AI follows an instruction from whether it can complete the task. A wrapper identical in output to grep -n was placed alongside a one-line description, and three conditions varied only the instruction file: no instruction, one line, and a strongly worded version using techniques the author had previously found effective. All 12 runs scored 12/12 on the task, so difficulty did not explain the differences.

The key finding is that a wrapper's presence on disk is separate from whether it gets called. In the no-instruction condition, sub-agents never opened bin/ even though ls showed it. Once one line named ./bin/scan, every sub-agent called it, and all first read the wrapper's contents with cat bin/scan before using it. The return to raw tools happened later: sub-agents switched to cat -A or sed to inspect leading whitespace, information the wrapper's output did not make easy to check. The author acknowledges the experiment cannot distinguish whether the wrapper's output was sufficient or insufficient, because the wrapper was deliberately designed not to outperform raw tools. The sample of 4 per condition also means only the 0/4 versus 4/4 difference is strong enough to assert.

FAQ
Did writing a stronger instruction in the instruction file keep the AI using the wrapper?
No. Even with all four previously effective instruction techniques combined, 1 of 4 sub-agents returned to raw tools before finishing.
Why did the AI switch back to raw tools in the experiment?
The wrapper ./bin/scan output only line numbers with lines, making it hard to check leading whitespace; the AI needed cat -A or sed to inspect that and then stayed with raw tools.
What does the author recommend instead of rewriting the instruction line?
Check whether the wrapper provides all the information the AI needs for the task, so the AI has no reason to return to raw tools.
Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleTANGLE's Naviu AR tourist app launches Nov 1, 2026, in 29 languages