LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909585465409536 |
|---|---|
| author | Yang, Shang Guo, Junxian Tang, Haotian Hu, Qinghao Xiao, Guangxuan Tang, Jiaming Lin, Yujun Liu, Zhijian Lu, Yao Han, Song |
| author_facet | Yang, Shang Guo, Junxian Tang, Haotian Hu, Qinghao Xiao, Guangxuan Tang, Jiaming Lin, Yujun Liu, Zhijian Lu, Yao Han, Song |
| contents | Large language models (LLMs) have shown remarkable potential in processing long sequences and complex reasoning tasks, yet efficiently serving these models remains challenging due to the quadratic computational complexity of attention in the prefilling stage and the large memory footprint of the KV cache in the decoding stage. To address these issues, we introduce LServe, an efficient system that accelerates long-sequence LLM serving via hybrid sparse attention. This method unifies different hardware-friendly, structured sparsity patterns for both prefilling and decoding attention into a single framework, where computations on less important tokens are skipped block-wise. LServe demonstrates the compatibility of static and dynamic sparsity in long-context LLM attention. This design enables multiplicative speedups by combining these optimizations. Specifically, we convert half of the attention heads to nearly free streaming heads in both the prefilling and decoding stages. Additionally, we find that only a constant number of KV pages is required to preserve long-context and reasoning capabilities, irrespective of context length. We then design a hierarchical KV page selection policy that dynamically prunes KV pages based on query-centric similarity. On average, LServe accelerates LLM prefilling by up to 2.9x and decoding by 1.3-2.1x over vLLM, maintaining long-context accuracy. Code is released at https://github.com/mit-han-lab/omniserve. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_14866 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention Yang, Shang Guo, Junxian Tang, Haotian Hu, Qinghao Xiao, Guangxuan Tang, Jiaming Lin, Yujun Liu, Zhijian Lu, Yao Han, Song Computation and Language Artificial Intelligence Distributed, Parallel, and Cluster Computing Machine Learning Performance Large language models (LLMs) have shown remarkable potential in processing long sequences and complex reasoning tasks, yet efficiently serving these models remains challenging due to the quadratic computational complexity of attention in the prefilling stage and the large memory footprint of the KV cache in the decoding stage. To address these issues, we introduce LServe, an efficient system that accelerates long-sequence LLM serving via hybrid sparse attention. This method unifies different hardware-friendly, structured sparsity patterns for both prefilling and decoding attention into a single framework, where computations on less important tokens are skipped block-wise. LServe demonstrates the compatibility of static and dynamic sparsity in long-context LLM attention. This design enables multiplicative speedups by combining these optimizations. Specifically, we convert half of the attention heads to nearly free streaming heads in both the prefilling and decoding stages. Additionally, we find that only a constant number of KV pages is required to preserve long-context and reasoning capabilities, irrespective of context length. We then design a hierarchical KV page selection policy that dynamically prunes KV pages based on query-centric similarity. On average, LServe accelerates LLM prefilling by up to 2.9x and decoding by 1.3-2.1x over vLLM, maintaining long-context accuracy. Code is released at https://github.com/mit-han-lab/omniserve. |
| title | LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attention |
| topic | Computation and Language Artificial Intelligence Distributed, Parallel, and Cluster Computing Machine Learning Performance |
| url | https://arxiv.org/abs/2502.14866 |