Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttention
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914380600311808 |
|---|---|
| author | Liao, Mengqi Wang, Lu Zhang, Chaoyun Qiao, Bo Qin, Si Lin, Qingwei Rajmohan, Saravan Zhang, Dongmei Wan, Huaiyu |
| author_facet | Liao, Mengqi Wang, Lu Zhang, Chaoyun Qiao, Bo Qin, Si Lin, Qingwei Rajmohan, Saravan Zhang, Dongmei Wan, Huaiyu |
| contents | With reasoning becoming the generative paradigm for large language models (LLMs), the memory bottleneck caused by KV cache during the decoding phase has become a critical factor limiting high-concurrency service. Although existing KV cache eviction methods address the memory issue, most of them are impractical for industrial-grade applications. This paper introduces Compressed PagedAttention, a method that combines token-wise KV cache eviction with PagedAttention. We propose a comprehensive scheduling strategy and support prefix caching and asynchronous compression for Compressed PagedAttention. Based on this, we have developed a high-concurrency LLM inference engine, Zipage. On large-scale mathematical reasoning tasks, Zipage achieves around 95\% of the performance of Full KV inference engines while delivering over 2.1$\times$ speedup. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_08743 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttention Liao, Mengqi Wang, Lu Zhang, Chaoyun Qiao, Bo Qin, Si Lin, Qingwei Rajmohan, Saravan Zhang, Dongmei Wan, Huaiyu Distributed, Parallel, and Cluster Computing Artificial Intelligence With reasoning becoming the generative paradigm for large language models (LLMs), the memory bottleneck caused by KV cache during the decoding phase has become a critical factor limiting high-concurrency service. Although existing KV cache eviction methods address the memory issue, most of them are impractical for industrial-grade applications. This paper introduces Compressed PagedAttention, a method that combines token-wise KV cache eviction with PagedAttention. We propose a comprehensive scheduling strategy and support prefix caching and asynchronous compression for Compressed PagedAttention. Based on this, we have developed a high-concurrency LLM inference engine, Zipage. On large-scale mathematical reasoning tasks, Zipage achieves around 95\% of the performance of Full KV inference engines while delivering over 2.1$\times$ speedup. |
| title | Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttention |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence |
| url | https://arxiv.org/abs/2603.08743 |