Agents Need Better Web Search—Full Documents, Not Snippets

AI agents that write code or manage workflows often need to query the web to stay current with production reality. Markets move, people change roles, news breaks, and an agent's usefulness depends on how well it reads the web. Today's standard approach bolts a search tool onto the agent and assumes it can read the web, but a token audit reveals the assumption is expensive.

A typical search API returns links and thirty-word snippets. The agent then has to fetch the raw HTML, strip the markup, and extract usable text before it can reason about the answer. On a single query, the overhead is small. But in a task requiring multiple searches—a research run, a briefing pipeline, or a market-monitoring loop—the agent pays this retrieval tax on every single hop. The article calls this the "retrieval tax," the tokens burned preparing content before reasoning begins.

To measure the cost, the analysis ran the same question ("What was Y2K?") through three retrieval setups and counted tokens billed. The model already knew the answer from training, so any tokens above the baseline came purely from retrieval. From memory alone, the answer took roughly 600 tokens. Through an owned index—a service that crawls and cleans pages ahead of time and stores processed copies—it took roughly 6,900 tokens. But a single web search hop cost roughly 3,750 tokens, and because snippets are often too thin to answer from, the agent refetched and refined. By the third hop, the total had climbed to roughly 28,700 tokens, more than 4× the owned index. The gap between the loop and the index is the retrieval tax made visible: by the third pass, the agent was mostly paying to carry pages it had already read.

The article compares three retrieval shapes. A SERP—the plain list of results a search API returns—gives URLs and asks the agent to go read the web itself. A scraper-first tool returns raw HTML or markdown, but the agent still has to clean it. An owned index does all the work before the query arrives, so what comes back is already readable. For a lookup of Christoph Molnar, the author of the Interpretable Machine Learning book, a SERP returned a list of tabs with no answer. A neural search tool found the right person as highlight fragments but required another call to get the full record. An owned index returned a complete record in one call, and the agent went straight to reasoning.

Seltz, described as a web index built for agents, crawls the web ahead of time, processes every page into a structured document, and returns finished content through a single API call. It comes with three scopes: the People Scope returns full structured profiles with role dates, education, websites, and organizations; the News Scope returns complete article text, date-windowed; and the Wiki Scope returns clean Wikipedia documents. One caveat is that the People Scope works best at the director and regional-leader layer; for the most senior executives, the article recommends starting with open web search and using Seltz to enrich everyone below them.

Full documents unlock a capability beyond cutting the retrieval tax: they enable questions that don't live in any single search result. Consider a question a go-to-market team actually runs: finding every target account that hired someone into a data or AI leadership role in the last quarter. The answer needs two things together—full role histories to find who started a new position and when, plus recent news to confirm the timing and surface the trigger event. A query of the People Scope finds who moved into a relevant role recently; cross-referencing the News Scope surfaces the trigger event. The result is a ranked list of warm accounts with enough context to write relevant outreach. That answer doesn't exist anywhere on the web as a single page; it only exists once you join the records, and the join works only when you hold the full records to begin with.

The article concludes that Seltz is not a replacement for open web search, which handles discovery well—finding who currently holds a title, confirming something launched last week, reading a page that went live this morning. But once you know what you're looking for, Seltz returns the complete record in one hop rather than a headline and a link. Many pipelines use the same search tool for every query, regardless of what the question needs, so discovery queries and deep-dive lookups go through the same call and pay the retrieval tax on both. Chaining the two instead—using open web search for discovery and an indexed service for deep dives—lets each loop iteration get the right retrieval shape for what it's actually trying to do.

The article frames a structural mismatch in how AI agents interact with the web. Coding agents have collapsed the cost of software generation, but validation capacity—the ability to catch failures before production—has not kept pace. AI-generated code fails on unknown-unknowns because no human formed a mental model while writing it, forcing validation to happen live on real traffic. The same challenge applies to prompt changes and retrieval-augmented generation (RAG): offline tests cannot predict production breakdowns. This shift to production-first validation demands raw, high-cardinality telemetry that can be sliced after the fact by any attribute the operator needs.

Within that frame, web search emerges as a critical bottleneck for agents dealing with present-day data—market changes, new hires, published news, price updates. The article quantifies a hidden cost: every search call that returns snippets instead of full documents forces the agent to fetch, parse, and clean HTML, repaying this cost on every loop iteration. The token audit (28,700 tokens for three hops through snippets vs. 6,900 for a single query of an indexed document) makes visible the compounding tax on context-window size. The article argues the fix is not a smarter model or prompt, but a change to the search interface itself—returning finished, structured documents so agents reason rather than preprocess.

No comments yet. Be the first to share your thoughts!