Lil: Less is Less When Applying Post-Training Sparse-Attention Algorithms in Long-Decode Stage
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911605257666560 |
|---|---|
| author | Hu, Junhao Li, Fangze Xu, Mingtao Meng, Feifan Zhao, Shiju Hu, Tiancheng Peng, Ting Liu, Anmin Huang, Wenrui Liu, Chenxu Hua, Ziyue Xie, Tao |
| author_facet | Hu, Junhao Li, Fangze Xu, Mingtao Meng, Feifan Zhao, Shiju Hu, Tiancheng Peng, Ting Liu, Anmin Huang, Wenrui Liu, Chenxu Hua, Ziyue Xie, Tao |
| contents | Large language models (LLMs) demonstrate strong capabilities across a wide range of complex tasks and are increasingly deployed at scale, placing significant demands on inference efficiency. Prior work typically decomposes inference into prefill and decode stages, with the decode stage dominating total latency. To reduce time and memory complexity in the decode stage, a line of work introduces sparse-attention algorithms. In this paper, we show, both empirically and theoretically, that sparse attention can paradoxically increase end-to-end complexity: information loss often induces significantly longer sequences, a phenomenon we term ``Less is Less'' (Lil). To mitigate the Lil problem, we propose an early-stopping algorithm that detects the threshold where information loss exceeds information gain during sparse decoding. Our early-stopping algorithm reduces token consumption by up to 90% with a marginal accuracy degradation of less than 2% across reasoning-intensive benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_03043 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Lil: Less is Less When Applying Post-Training Sparse-Attention Algorithms in Long-Decode Stage Hu, Junhao Li, Fangze Xu, Mingtao Meng, Feifan Zhao, Shiju Hu, Tiancheng Peng, Ting Liu, Anmin Huang, Wenrui Liu, Chenxu Hua, Ziyue Xie, Tao Computation and Language Artificial Intelligence Machine Learning Large language models (LLMs) demonstrate strong capabilities across a wide range of complex tasks and are increasingly deployed at scale, placing significant demands on inference efficiency. Prior work typically decomposes inference into prefill and decode stages, with the decode stage dominating total latency. To reduce time and memory complexity in the decode stage, a line of work introduces sparse-attention algorithms. In this paper, we show, both empirically and theoretically, that sparse attention can paradoxically increase end-to-end complexity: information loss often induces significantly longer sequences, a phenomenon we term ``Less is Less'' (Lil). To mitigate the Lil problem, we propose an early-stopping algorithm that detects the threshold where information loss exceeds information gain during sparse decoding. Our early-stopping algorithm reduces token consumption by up to 90% with a marginal accuracy degradation of less than 2% across reasoning-intensive benchmarks. |
| title | Lil: Less is Less When Applying Post-Training Sparse-Attention Algorithms in Long-Decode Stage |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2601.03043 |