
What happened
An Nvidia research team says feeding models entire novels is often expensive overkill; in many tests, periodic summaries plus targeted keyword lookups matched full-context results at a fraction of the cost.
Why it matters
If that holds, teams running large language models could cut cost and error rates by keeping working memory short instead of paying for maximum context, according to the team's argument.
WHO IT HITSEnterprise teams that run large language models on long documents—such as legal, research, and customer-support groups—could see lower inference costs if the summarized-note approach holds up beyond the team's tests.
Summaries like this, every morning.
The growth of context windows has been one of the most visible measures of progress in language models. GPT-4 arrived in March 2023 with 8,192 tokens, and GPT 5.6 Sol now handles around one million tokens—roughly 120 times as much, enough to fit Tolstoy's "War and Peace" in a single view.
The Nvidia team's argument is that this capacity is often wasted. In many of their tests, periodic summaries combined with targeted keyword lookups worked about as well as feeding the model the full context, but at a fraction of the cost. That suggests the practical bottleneck may be how information is selected and condensed rather than how much of it can be held at once.
The team goes further by proposing a new class of models built around short notes instead of long context. If such designs prove out, the economics of running models on long documents could shift toward cheaper, more targeted retrieval and summarization—though the body offers no timeline for when that might happen.
Pick your industry and the AI tools you use, and get news related to your work every day.
Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.
Positron's Cameron McCaskill said about two-thirds of AI infrastructure workloads are now inference versus one…
The writer told Muse only their height and that they wanted slim, cropped tops, then let it browse ZOZOTOWN in…

After a reader comment, the Search Console-based service 検索カルテ began attaching explicit start and end dates (f…

A practitioner customizing an agent called Muse had other models review documents and design proposals, found…

OpenAI published copy-ready prompts for GPT-6.2 Astra and Sol covering writing style, initiative, and coding…

Seramount's survey of more than 1,800 U.S
