Stop stuffing giant prompts into expensive models; decompose and route instead

Qolca, an AI consulting firm, published a guide arguing that most AI products are bleeding money and latency through a single architectural mistake: sending one enormous system prompt to the most expensive model on every request. The company illustrates the hidden costs with concrete numbers. Frontier-tier models cost on the order of a few dollars to fifteen dollars per million output tokens and a few dollars per million input tokens, while small fast models in the Haiku class cost roughly a third to a fifth of that. A modest 4,000-token system prompt, once stuffed with a product catalog, tone guidelines, and a dozen examples, is re-read by the model on every turn of a ten-turn conversation—totaling 40,000 input tokens just to restate instructions the model already absorbed on the first turn. The waste compounds across every conversation and every day and scales linearly with the product's success. Cost is only half the problem; the other half is latency. A model cannot emit its first token until it has processed the entire prompt, so a larger prompt means more to read and more to reason over before anything appears on screen. Users experience this as sluggishness, and teams often blame the model when the real culprit is the wall of context forced on it with every request.

Qolca proposes five patterns for treating tokens as a scarce resource. First, decompose the flow into steps so each model call carries only the context that specific step requires. A single "handle this customer message" request is really several smaller jobs: figure out the language, classify intent, gather relevant history, and write a reply. Bundled into one prompt, every sub-job pays for the full context and the full model. Split apart, language detection needs just the message; intent classification needs the message and maybe one line of history; only the final reply needs the persona, tone, and product knowledge. Second, retrieve instead of context-stuffing. The most common form of token waste is pasting an entire knowledge base into the prompt on the theory that the model might need it. If a customer asks about a refund window, the model needs the refund policy, not shipping zones, warranty terms, and company history. Retrieval turns a 6,000-token knowledge dump into a 400-token on-topic context and improves accuracy because a focused context is easier for the model to reason over. Third, cache the static prefix. Modern model APIs let teams cache parts of the prompt that never change—the persona, hard rules, format instructions—so they are not reprocessed from scratch every time. Cached input tokens bill at roughly a tenth of normal input price and process faster, but caching is a prefix match, so static content must come first in the prompt and volatile content (user message, timestamp, per-request data) must come last, or the cache is invalidated on every request. Fourth, classify and route cheaply, using a small fast model to decide what happens next. Fifth, reserve the expensive model for the one step that genuinely needs its intelligence.

Qolca's own production AI sales assistant for its website is built on exactly these principles. A single incoming message runs through four deliberate steps instead of one giant call. First, two cheap classifiers run in parallel on a small fast model: one detects language, the other classifies intent into a fixed set of categories. These calls are tiny—a handful of tokens in, a word or two out—and finish in the time of a single call because they run simultaneously. Second, a context summary step runs on the same cheap model, but only conditionally; it fires only when the conversation is long enough to actually need summarizing, so short conversations skip it entirely and pay nothing. Third, only the final step—generating the actual conversational reply—uses the expensive, more capable model, and it receives the detected language, the classified intent, a compact context, and the message, not a giant everything-prompt. Fourth, a fifth step decides the call-to-action (book a call, open WhatsApp, show a contact form) with plain deterministic code and no model at all. The result is a system where the overwhelming majority of token spend goes to the cheap model, the frontier model is invoked once with lean context, and static parts are cached. The user experience is the same as the naive one-prompt version, but at a fraction of the cost and latency.

For teams with existing monster prompts in production, Qolca recommends a safe retrofit sequence. First, instrument the system to log tokens in and out per request and the model used, so the team can see where money actually goes before making changes. Second, move the frozen parts of the prompt to the top and enable prefix caching—the cheapest win that changes no behavior. Third, pull the largest static block (usually a knowledge dump) out and replace it with retrieval of the relevant slice. Fourth, find the cheap decisions hiding inside the big call—language, intent, routing—and split them into separate calls on a small fast model. Fifth, downgrade any step that is classification or extraction from the frontier model to the cheap one and confirm quality holds. Only the final generative step should remain on the expensive model, and only with the lean context it needs. The article concludes that teams winning on unit economics are not the ones who found a cheaper model; they are the ones who stopped paying frontier prices to answer questions a router could have handled and reserved their most expensive tokens for the moments that actually earn them.

The article identifies a widespread architectural antipattern in AI products: the monolithic prompt. Teams build comprehensive system prompts packing every rule, example, and piece of company knowledge, then send that entire context to the most expensive available model on every single request. The instinct is to optimize by finding cheaper vendors, but the article argues this misdiagnoses the problem. The real issue is treating tokens as a free or near-free resource when they are actually metered and expensive.

The economic case is straightforward: a 4,000-token prompt re-read across ten conversation turns costs 40,000 input tokens just to repeat static instructions. At a few dollars per million input tokens, the cost per request seems negligible until multiplied across all users and all days—at which point the waste scales linearly with the product's own success. The latency cost is equally real but less obvious. A model cannot emit its first token until it has read the entire prompt, so a bloated prompt forces the model to process thousands of tokens before the user sees anything on screen. The article describes this as paying twice for the same mistake: once in dollars, once in seconds.

Qolca proposes five patterns that shift the mental model from "what could the model possibly need?" to "what does this specific step actually require?" Decomposing a complex request into separate steps (language detection, intent classification, context retrieval, reply generation) allows each step to carry only its necessary context. Retrieval replaces context-stuffing, pulling in the two or three relevant paragraphs instead of the entire knowledge base. Prefix caching bills static content at roughly a tenth of normal price. Cheap models handle classification and routing decisions. The expensive model is reserved for the one step that truly needs frontier-level fluency. The company's own production assistant exemplifies this: two parallel cheap classifiers, an optional conditional summary, a single expensive call for the reply, and deterministic code for routing—resulting in the same user experience at a fraction of cost and latency.

No comments yet. Be the first to share your thoughts!