AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910115438788608 |
|---|---|
| author | Hu, Yuxuan Tan, Jianchao Zhang, Jiaqi Zan, Wen Sun, Pingwei Lu, Yifan Sun, Yerui Xie, Yuchen Cai, Xunliang Zhang, Jing |
| author_facet | Hu, Yuxuan Tan, Jianchao Zhang, Jiaqi Zan, Wen Sun, Pingwei Lu, Yifan Sun, Yerui Xie, Yuchen Cai, Xunliang Zhang, Jing |
| contents | Long-context inference in LLMs faces the dual challenges of quadratic attention complexity and prohibitive KV cache memory. While token-level sparse attention offers superior accuracy, its indexing overhead is costly; block-level methods improve efficiency but sacrifice precision. We propose AsyncTLS, a hierarchical sparse attention system that combines coarse-grained block filtering with fine-grained token selection to balance accuracy and efficiency, coupled with an asynchronous offloading engine that overlaps KV cache transfers with computation via temporal locality exploitation. Evaluated on Qwen3 and GLM-4.7-Flash across GQA, and MLA architectures, AsyncTLS achieves accuracy comparable to full attention while delivering 1.2x - 10.0x operator speedups and 1.3x - 4.7x end-to-end throughput improvements on 48k - 96k contexts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_07815 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention Hu, Yuxuan Tan, Jianchao Zhang, Jiaqi Zan, Wen Sun, Pingwei Lu, Yifan Sun, Yerui Xie, Yuchen Cai, Xunliang Zhang, Jing Computation and Language I.2.7 Long-context inference in LLMs faces the dual challenges of quadratic attention complexity and prohibitive KV cache memory. While token-level sparse attention offers superior accuracy, its indexing overhead is costly; block-level methods improve efficiency but sacrifice precision. We propose AsyncTLS, a hierarchical sparse attention system that combines coarse-grained block filtering with fine-grained token selection to balance accuracy and efficiency, coupled with an asynchronous offloading engine that overlaps KV cache transfers with computation via temporal locality exploitation. Evaluated on Qwen3 and GLM-4.7-Flash across GQA, and MLA architectures, AsyncTLS achieves accuracy comparable to full attention while delivering 1.2x - 10.0x operator speedups and 1.3x - 4.7x end-to-end throughput improvements on 48k - 96k contexts. |
| title | AsyncTLS: Efficient Generative LLM Inference with Asynchronous Two-level Sparse Attention |
| topic | Computation and Language I.2.7 |
| url | https://arxiv.org/abs/2604.07815 |