Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.16886 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909982413291520 |
|---|---|
| author | Ni, Xuanfan Xu, Liyan Lyu, Chenyang Wang, Longyue Yu, Mo Liu, Lemao Meng, Fandong Zhou, Jie Li, Piji |
| author_facet | Ni, Xuanfan Xu, Liyan Lyu, Chenyang Wang, Longyue Yu, Mo Liu, Lemao Meng, Fandong Zhou, Jie Li, Piji |
| contents | To reduce memory consumption during LLM inference, prior works have proposed numerous methods that focus on KV cache pruning based on various criteria. While these techniques often accomplish lossless memory reduction on many datasets, they often rely on an under-emphasized condition: a dataset/domain-specific budget size threshold needs to be pre-determined to achieve the optimal performance. However, such input-specific tuning may be considerably limited in real-world scenarios, as open-domain inputs span diverse domains, lengths and difficulty levels, without clear boundaries for pre-tuning. Thus, the dependence of an input-sensitive threshold can be an inherent limitation that may cause large degradation on arbitrary inputs. In this work, we propose a new objective that lifts the threshold constraints for robust KV pruning, calling for "threshold-free" methods that automatically adjust budget sizes while ensuring full-cache performance. We then propose a novel method ReFreeKV as the first solution fulfilling this objective, validated by intensive experiments on 13 datasets of diverse context lengths, task types, and model sizes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_16886 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Threshold-Free KV Cache Pruning Ni, Xuanfan Xu, Liyan Lyu, Chenyang Wang, Longyue Yu, Mo Liu, Lemao Meng, Fandong Zhou, Jie Li, Piji Computation and Language Artificial Intelligence To reduce memory consumption during LLM inference, prior works have proposed numerous methods that focus on KV cache pruning based on various criteria. While these techniques often accomplish lossless memory reduction on many datasets, they often rely on an under-emphasized condition: a dataset/domain-specific budget size threshold needs to be pre-determined to achieve the optimal performance. However, such input-specific tuning may be considerably limited in real-world scenarios, as open-domain inputs span diverse domains, lengths and difficulty levels, without clear boundaries for pre-tuning. Thus, the dependence of an input-sensitive threshold can be an inherent limitation that may cause large degradation on arbitrary inputs. In this work, we propose a new objective that lifts the threshold constraints for robust KV pruning, calling for "threshold-free" methods that automatically adjust budget sizes while ensuring full-cache performance. We then propose a novel method ReFreeKV as the first solution fulfilling this objective, validated by intensive experiments on 13 datasets of diverse context lengths, task types, and model sizes. |
| title | Towards Threshold-Free KV Cache Pruning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.16886 |