Sliding-window attention outperforms linear on long-context

A new arXiv preprint by Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron, and Emy Gervais claims that sliding-window attention with sinks performs as well or better than linear-attention variants on long-context reasoning. On the benchmarks Needle-in-a-Haystack and BABILong, the abstract reports 2 to 10 times higher performance than linear attention.