Sliding-window attention beats linear on long-context reasoning [R]

Sliding Window Attention with sinks, one of the simplest existing fixes for the quadratic-cost problem in LLMs, holds up as well or better than the linear-attention variants labs have been spending post-training compute to produce. That is the claim of a [new arXiv preprint]( https://arxiv.org/abs/2608.28444 ) by Alexia Jolicoeur-Martineau, Rhea Sanjay Sukthanker, Pashmina Cameron and Emy Gervais. On the long-context reasoning benchmarks the paper singles out, the gap is not close. "SWA achieves massively higher performance (2 to 10 times higher than linear attention)," the abstract reports, naming Needle-in-a-Haystack and BABILong as the two tasks. The pitch is that the whole post-training-to-linear pipeline has been benchmarked against the wrong thing. "This line of research has not been properly compared to simpler baselines," the authors write. Their alternative needs no post-training, runs fast, and holds memory low. The recommendation is blunt: "we strongly recommend switching to SWA instead of post-training linear models." Linear attention, the abstract concedes, "may have shown some promise, but they likely require to be trained from scratch or extensive post-training in order to even match SWA." --- submitted by /u/Justgototheeffinmoon [link] [留言]