DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908380433481728 |
|---|---|
| author | Zhou, Xiabin Wang, Wenbin Zeng, Minyan Guo, Jiaxian Liu, Xuebo Shen, Li Zhang, Min Ding, Liang |
| author_facet | Zhou, Xiabin Wang, Wenbin Zeng, Minyan Guo, Jiaxian Liu, Xuebo Shen, Li Zhang, Min Ding, Liang |
| contents | Efficient KV cache management in LLMs is crucial for long-context tasks like RAG and summarization. Existing KV cache compression methods enforce a fixed pattern, neglecting task-specific characteristics and reducing the retention of essential information. However, we observe distinct activation patterns across layers in various tasks, highlighting the need for adaptive strategies tailored to each task's unique demands. Based on this insight, we propose DynamicKV, a method that dynamically optimizes token retention by adjusting the number of tokens retained at each layer to adapt to the specific task. DynamicKV establishes global and per-layer maximum KV cache budgets, temporarily retaining the maximum budget for the current layer, and periodically updating the KV cache sizes of all preceding layers during inference. Our method retains only 1.7% of the KV cache size while achieving ~85% of the Full KV cache performance on LongBench. Notably, even under extreme compression (0.9%), DynamicKV surpasses state-of-the-art (SOTA) methods by 11% in the Needle-in-a-Haystack test using Mistral-7B-Instruct-v0.2. The code will be released. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_14838 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs Zhou, Xiabin Wang, Wenbin Zeng, Minyan Guo, Jiaxian Liu, Xuebo Shen, Li Zhang, Min Ding, Liang Computation and Language Efficient KV cache management in LLMs is crucial for long-context tasks like RAG and summarization. Existing KV cache compression methods enforce a fixed pattern, neglecting task-specific characteristics and reducing the retention of essential information. However, we observe distinct activation patterns across layers in various tasks, highlighting the need for adaptive strategies tailored to each task's unique demands. Based on this insight, we propose DynamicKV, a method that dynamically optimizes token retention by adjusting the number of tokens retained at each layer to adapt to the specific task. DynamicKV establishes global and per-layer maximum KV cache budgets, temporarily retaining the maximum budget for the current layer, and periodically updating the KV cache sizes of all preceding layers during inference. Our method retains only 1.7% of the KV cache size while achieving ~85% of the Full KV cache performance on LongBench. Notably, even under extreme compression (0.9%), DynamicKV surpasses state-of-the-art (SOTA) methods by 11% in the Needle-in-a-Haystack test using Mistral-7B-Instruct-v0.2. The code will be released. |
| title | DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.14838 |