How to make any Sparse Attention / KV Compression look good? [D] [R]

Original Article - https://x.com/p_nawrot/status/2089315591010079034 I've spent the last few years working on efficient attention and KV Cache Compression. I've read many papers, dug deep into reference or official implementations of methods, and inspected appendices—and I think I've learned a few things. One of them is definitely "how to make things look good, even when they aren't." I'm guilty too, but trying to get better every day. 1. For single-hop retrieval, make sure there are no distractors and context is useless The three most cooperative settings for compression / sparsity are: Needle in a haystack with a single OOD key-value pair and context built out of a repeated sentence or irrelevant background text. Contaminated benchmarks from years ago for which models don't even look at the context anymore. Few-shot in-context learning, where extra shots are useless and don't improve the accuracy over 0-shot. With 1) synthetic tasks, 2) real-data QA, and 3) in-context learning, you get a semblance of broad coverage without the inconvenience of testing much diversity within any of them. Most tasks in these settings should pass under Sliding Window Attention, so it doesn't matter that much whether your method works. Combine it with SWA and you should be good to report 5–10x compression or sparsity. 2. NEVER isolate your contribution Short context: Most of a dense model's performance is recovered by a local window + attention sinks + the ability to retrieve an answer sentence that is largely n-gram matchable with the question. The remaining part is significantly more difficult, but it's neither relevant to nor the subject of this post. Say prior work developed an algorithm X, and its implementation separately keeps a local window of 256 tokens. You find that your method is on par with X in a matched setting, but better and more stable with a window size of 512—let's go, don't look back. Do the same with block size. Smaller blocks can give you finer granularity and mo