KV Cache Compression for Inference Efficiency in LLMs: A Review
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915434777804800 |
|---|---|
| author | Liu, Yanyu Fu, Jingying Liu, Sixiang Zou, Yitian Fu, You Zhou, Jiehan Zhang, Shouhua |
| author_facet | Liu, Yanyu Fu, Jingying Liu, Sixiang Zou, Yitian Fu, You Zhou, Jiehan Zhang, Shouhua |
| contents | Withtherapid advancement of large language models (LLMs), the context length for inference has been continuously increasing, leading to an exponential growth in the demand for Key-Value (KV) caching. This has resulted in a significant memory bottleneck, limiting the inference efficiency and scalability of the models. Therefore, optimizing the KV cache during inference is crucial for enhancing performance and efficiency. This review systematically examines current KV cache optimization techniques, including compression strategies such as selective token strategies, quantization, and attention compression. We evaluate the effectiveness, trade-offs, and application scenarios of these methods, providing a comprehensive analysis of their impact on memory usage and inference speed. We focus on identifying the limitations and challenges of existing methods, such as compatibility issues with different models and tasks. Additionally, this review highlights future research directions, including hybrid optimization techniques, adaptive dynamic strategies, and software-hardware co-design. These approaches aim to improve inference efficiency and promote the practical application of large language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_06297 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | KV Cache Compression for Inference Efficiency in LLMs: A Review Liu, Yanyu Fu, Jingying Liu, Sixiang Zou, Yitian Fu, You Zhou, Jiehan Zhang, Shouhua Distributed, Parallel, and Cluster Computing Withtherapid advancement of large language models (LLMs), the context length for inference has been continuously increasing, leading to an exponential growth in the demand for Key-Value (KV) caching. This has resulted in a significant memory bottleneck, limiting the inference efficiency and scalability of the models. Therefore, optimizing the KV cache during inference is crucial for enhancing performance and efficiency. This review systematically examines current KV cache optimization techniques, including compression strategies such as selective token strategies, quantization, and attention compression. We evaluate the effectiveness, trade-offs, and application scenarios of these methods, providing a comprehensive analysis of their impact on memory usage and inference speed. We focus on identifying the limitations and challenges of existing methods, such as compatibility issues with different models and tasks. Additionally, this review highlights future research directions, including hybrid optimization techniques, adaptive dynamic strategies, and software-hardware co-design. These approaches aim to improve inference efficiency and promote the practical application of large language models. |
| title | KV Cache Compression for Inference Efficiency in LLMs: A Review |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2508.06297 |