PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915479322361856 |
|---|---|
| author | Chitty-Venkata, Krishna Teja Ye, Jie Sun, Xian-He Kougkas, Anthony Emani, Murali Vishwanath, Venkatram Nicolae, Bogdan |
| author_facet | Chitty-Venkata, Krishna Teja Ye, Jie Sun, Xian-He Kougkas, Anthony Emani, Murali Vishwanath, Venkatram Nicolae, Bogdan |
| contents | KV caching significantly improves the efficiency of Large Language Model (LLM) inference by storing attention states from previously processed tokens, enabling faster generation of subsequent tokens. However, as sequence length increases, the KV cache quickly becomes a major memory bottleneck. To address this, we propose PagedEviction, a novel fine-grained, structured KV cache pruning strategy that enhances the memory efficiency of vLLM's PagedAttention. Unlike existing approaches that rely on attention-based token importance or evict tokens across different vLLM pages, PagedEviction introduces an efficient block-wise eviction algorithm tailored for paged memory layouts. Our method integrates seamlessly with PagedAttention without requiring any modifications to its CUDA attention kernels. We evaluate PagedEviction across Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct, and Llama-3.2-3B-Instruct models on the LongBench benchmark suite, demonstrating improved memory usage with better accuracy than baselines on long context tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_04377 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference Chitty-Venkata, Krishna Teja Ye, Jie Sun, Xian-He Kougkas, Anthony Emani, Murali Vishwanath, Venkatram Nicolae, Bogdan Machine Learning KV caching significantly improves the efficiency of Large Language Model (LLM) inference by storing attention states from previously processed tokens, enabling faster generation of subsequent tokens. However, as sequence length increases, the KV cache quickly becomes a major memory bottleneck. To address this, we propose PagedEviction, a novel fine-grained, structured KV cache pruning strategy that enhances the memory efficiency of vLLM's PagedAttention. Unlike existing approaches that rely on attention-based token importance or evict tokens across different vLLM pages, PagedEviction introduces an efficient block-wise eviction algorithm tailored for paged memory layouts. Our method integrates seamlessly with PagedAttention without requiring any modifications to its CUDA attention kernels. We evaluate PagedEviction across Llama-3.1-8B-Instruct, Llama-3.2-1B-Instruct, and Llama-3.2-3B-Instruct models on the LongBench benchmark suite, demonstrating improved memory usage with better accuracy than baselines on long context tasks. |
| title | PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inference |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2509.04377 |