A new preprint says sliding-window attention with sinks beats linear attention on long-context reasoning.
It reports 2 to 10 times higher performance on two benchmarks.
The method needs no post-training and runs fast.
What happened
A new arXiv preprint by Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron, and Emy Gervais claims that sliding-window attention with sinks performs as well or better than linear-attention variants on long-context reasoning. On the benchmarks Needle-in-a-Haystack and BABILong, the abstract reports 2 to 10 times higher performance than linear attention.
Why it matters
The authors argue that the post-training-to-linear pipeline has not been properly compared with simpler baselines. Their alternative requires no post-training, runs fast, and keeps memory low, suggesting that simpler fixes may be more effective than expensive linear-attention approaches.
What to watch
The paper strongly recommends switching to sliding-window attention with sinks for long-context reasoning, which could influence future model design if the results are validated.
Ask the AI about this article →
The preprint challenges a common assumption in the AI field that linear attention variants, which often require significant post-training compute, are necessary to handle long contexts efficiently. The authors position sliding-window attention with sinks—a simpler, existing fix—as a competitive or superior baseline. They argue that this line of research has not been properly benchmarked against simpler alternatives.
If the results hold, the implication is that labs may be spending substantial resources on complex linear-attention pipelines when a straightforward method could achieve better reasoning performance. The recommendation to switch to sliding-window attention with sinks is direct, though the paper is a preprint and its findings would likely require peer review and replication before broad adoption.
For example, today's edition would include:
AI-summarized, only the topics you pick — one digest a day via Email, Slack, or Discord.
Free · takes 30 seconds · unsubscribe anytimeWhat is AIToday? →
Ask AI anything about this article. Q&As are published on this page for other readers too.
Analyst Ming-Chi Kuo says Nvidia has revived the Rubin CPX AI accelerator with a substantially redesigned arch…
A UK study by UK AI Security Institute and Limbic AI surveyed 6,474 British adults
Broadcom's Clayton Donley says companies are doing mission-critical work with AI agents quickly, but without t…
OpenAI released a new evaluation framework on July 17, 2026, urging companies to measure AI ROI by 'useful out…
As AI agents perform real business tasks, 'Agentic Identity' (giving each AI a unique employee-like ID) and 'D…
The European Union is expanding regulation of ChatGPT and will mandate protections for minors
