Most AI products overpay and experience unnecessary latency because they send enormous system prompts containing all rules, examples, and knowledge to expensive models on every request.
The solution is not finding a cheaper vendor but redesigning the workflow: decompose requests into separate steps, retrieve only relevant context instead of stuffing everything into the prompt, cache static parts, use cheap models for classification and routing, and reserve expensive models for the one step that genuinely needs advanced reasoning.
Qolca's own production sales assistant uses this approach—two cheap classifiers run in parallel, then a cheap optional summary, then a single expensive call for reply generation—achieving the same user experience at a fraction of the cost and latency.
What happened
Qolca published a guide on token efficiency in AI products, arguing that the standard practice of sending massive system prompts with all rules, examples, and knowledge to the most expensive model on every request wastes both money and latency. The company's own production AI sales assistant uses a four-step pipeline instead: two cheap parallel classifiers (language detection, intent classification), a conditional context summary (only when conversations are long), and a final reply generation on the expensive model with lean context.
Why it matters
Input tokens to frontier-tier models cost on the order of a few dollars per million, and every token is re-billed on every request. A modest 4,000-token system prompt re-read in a ten-turn conversation costs 40,000 input tokens just to repeat the same instructions; across all conversations and days, the waste scales linearly with success. Latency is also a product feature—larger prompts force models to process more context before emitting the first token, making chatbots feel sluggish to users. Teams blaming slow responses or high bills often have an architecture problem, not a vendor problem.
What to watch
The guide recommends a specific retrofit sequence for existing prompts: enable prefix caching on frozen parts first (billed at roughly a tenth of normal input price), pull large static blocks (like knowledge bases) into retrieval-based slices, split cheap classification decisions into separate small-model calls, and downgrade classification or extraction from frontier models to cheaper alternatives. Only the final generative step should remain on the expensive model, and only with the lean context it actually needs.
Ask the AI about this article →
The article identifies a widespread architectural antipattern in AI products: the monolithic prompt. Teams build comprehensive system prompts packing every rule, example, and piece of company knowledge, then send that entire context to the most expensive available model on every single request. The instinct is to optimize by finding cheaper vendors, but the article argues this misdiagnoses the problem. The real issue is treating tokens as a free or near-free resource when they are actually metered and expensive.
The economic case is straightforward: a 4,000-token prompt re-read across ten conversation turns costs 40,000 input tokens just to repeat static instructions. At a few dollars per million input tokens, the cost per request seems negligible until multiplied across all users and all days—at which point the waste scales linearly with the product's own success. The latency cost is equally real but less obvious. A model cannot emit its first token until it has read the entire prompt, so a bloated prompt forces the model to process thousands of tokens before the user sees anything on screen. The article describes this as paying twice for the same mistake: once in dollars, once in seconds.
Qolca proposes five patterns that shift the mental model from "what could the model possibly need?" to "what does this specific step actually require?" Decomposing a complex request into separate steps (language detection, intent classification, context retrieval, reply generation) allows each step to carry only its necessary context. Retrieval replaces context-stuffing, pulling in the two or three relevant paragraphs instead of the entire knowledge base. Prefix caching bills static content at roughly a tenth of normal price. Cheap models handle classification and routing decisions. The expensive model is reserved for the one step that truly needs frontier-level fluency. The company's own production assistant exemplifies this: two parallel cheap classifiers, an optional conditional summary, a single expensive call for the reply, and deterministic code for routing—resulting in the same user experience at a fraction of cost and latency.
For example, today's edition would include:
AI-summarized, only the topics you pick — one digest a day via Email, Slack, or Discord.
Free · takes 30 seconds · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. Q&As are published on this page for other readers too.
Visko raised $10 million in pre-seed funding from Llama Ventures and opened public access to its first foundat…
AI company Runway has unveiled Solaris, the first model in a new category it calls "Interface World Models." I…
Google's AI search gave advice to call emergency services for users alone with an African, Indian, or Pakistan…
John Deere introduced JD, a conversational AI tool that lets farmers ask open-ended questions about their hist…
Nvidia CEO Jensen Huang said on Fox Business that AI is creating 'hundreds of thousands' of jobs, including in…
Israeli startup DataAgent Ltd
