AIToday
Large Language ModelsTHE DECODERPublished: Nov 22, 2026, 04:00 JST

Nvidia researchers: big context windows are expensive overkill

Send on LINE
Nvidia researchers: big context windows are expensive overkill

3 Key Points

  1. What happened

    An Nvidia research team says feeding models entire novels is often expensive overkill; in many tests, periodic summaries plus targeted keyword lookups matched full-context results at a fraction of the cost.

  2. Why it matters

    If that holds, teams running large language models could cut cost and error rates by keeping working memory short instead of paying for maximum context, according to the team's argument.

WHO IT HITSEnterprise teams that run large language models on long documents—such as legal, research, and customer-support groups—could see lower inference costs if the summarized-note approach holds up beyond the team's tests.

Not sure about something? Ask the AI

Questions and answers are published on this page.

Summaries like this, every morning.

Context & Analysis

The growth of context windows has been one of the most visible measures of progress in language models. GPT-4 arrived in March 2023 with 8,192 tokens, and GPT 5.6 Sol now handles around one million tokens—roughly 120 times as much, enough to fit Tolstoy's "War and Peace" in a single view.

The Nvidia team's argument is that this capacity is often wasted. In many of their tests, periodic summaries combined with targeted keyword lookups worked about as well as feeding the model the full context, but at a fraction of the cost. That suggests the practical bottleneck may be how information is selected and condensed rather than how much of it can be held at once.

The team goes further by proposing a new class of models built around short notes instead of long context. If such designs prove out, the economics of running models on long documents could shift toward cheaper, more targeted retrieval and summarization—though the body offers no timeline for when that might happen.

FAQ
How much has context window size grown?
GPT-4 launched in March 2023 with 8,192 tokens, while GPT 5.6 Sol today handles around one million tokens—about 120 times as much and enough for Tolstoy's "War and Peace."
What alternative do the Nvidia researchers propose?
They propose periodic summaries and targeted keyword search, and even a new class of models that works with short notes instead of long context.
Send on LINE

AI news that matters for your work, delivered every morning.

Pick your industry and the AI tools you use, and get news related to your work every day.

Free · 30 seconds with Google · unsubscribe anytimeWhat is AIToday? →

Ask AI

Ask AI anything about this article. The AI reads this article, earlier AIToday articles, and Wikipedia, and cites its sources. Q&As are published on this page for other readers too.

Questions and answers are published on this page.

Related Articles

Next articleInference hits two-thirds of AI workloads, says Positron's Cameron McCaskill