Plan-then-Retrieve beats Retrieve-then-Plan on multi-tool queries

The study targets a practical bottleneck in tool-augmented LLM agents: finite context windows prohibit injecting all tool definitions into a single prompt, so agents must discover relevant APIs before planning execution. The dominant approach, which the authors term Retrieve-then-Plan (RtP), embeds the user query and retrieves similar tools via approximate nearest-neighbor search.

The researchers characterize a stub-hallucination failure mode — the LLM invents plausible but non-existent API names — that acts as a double-edged sword. On simple queries, Plan-then-Retrieve's stubs abstract away proper-noun signals that Retrieve-then-Plan matches directly, yet on complex queries the same mechanism decomposes multi-faceted intent into functional sub-needs. This crossover held at every retrieval budget and was consistent across Gemini 2.5 Flash and gpt-oss-120b, and robust across registry sizes from 1k to 47k APIs.

Because Plan-then-Retrieve's extra discovery call roughly doubles inference cost, the authors motivate complexity-adaptive routing as a practical direction — one that can be driven by a zero-cost signal already present in the pipeline (RtP's top-1 retrieval score). The related work cited includes a paper accepted at the UncertaiNLP workshop at EACL 2024 on single-step versus sequential retrieval with task decomposition.