Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.13035 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915195121565696 |
|---|---|
| author | Wan, Zhongwei Wu, Xinjian Zhang, Yu Xin, Yi Tao, Chaofan Zhu, Zhihong Wang, Xin Luo, Siqi Xiong, Jing Wang, Longyue Zhang, Mi |
| author_facet | Wan, Zhongwei Wu, Xinjian Zhang, Yu Xin, Yi Tao, Chaofan Zhu, Zhihong Wang, Xin Luo, Siqi Xiong, Jing Wang, Longyue Zhang, Mi |
| contents | Generative inference in Large Language Models (LLMs) is impeded by the growing memory demands of Key-Value (KV) cache, especially for longer sequences. Traditional KV cache eviction strategies, which discard less critical KV pairs based on attention scores, often degrade generation quality, leading to issues such as context loss or hallucinations. In this work, we introduce Dynamic Discriminative Operations (D2O), a KV cache compression method that optimizes KV cache size dynamically and discriminatively at two levels without fine-tuning, while preserving essential context. At layer level, D2O leverages the varying densities of attention weights between shallow and deep layers to dynamically determine which layers should avoid excessive eviction via a novel dynamic allocation strategy to minimize information loss. At token level, D2O incorporates a compensation mechanism that maintains a similarity threshold to re-discriminate the importance of currently discarded tokens, determining whether they should be recalled and merged with similar tokens. We conduct experiments on various benchmarks and LLM architectures. Our results show that D2O not only achieves significant memory savings and enhances inference throughput by more than 3$\times$ but also maintains high-quality long-text generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_13035 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models Wan, Zhongwei Wu, Xinjian Zhang, Yu Xin, Yi Tao, Chaofan Zhu, Zhihong Wang, Xin Luo, Siqi Xiong, Jing Wang, Longyue Zhang, Mi Computation and Language Generative inference in Large Language Models (LLMs) is impeded by the growing memory demands of Key-Value (KV) cache, especially for longer sequences. Traditional KV cache eviction strategies, which discard less critical KV pairs based on attention scores, often degrade generation quality, leading to issues such as context loss or hallucinations. In this work, we introduce Dynamic Discriminative Operations (D2O), a KV cache compression method that optimizes KV cache size dynamically and discriminatively at two levels without fine-tuning, while preserving essential context. At layer level, D2O leverages the varying densities of attention weights between shallow and deep layers to dynamically determine which layers should avoid excessive eviction via a novel dynamic allocation strategy to minimize information loss. At token level, D2O incorporates a compensation mechanism that maintains a similarity threshold to re-discriminate the importance of currently discarded tokens, determining whether they should be recalled and merged with similar tokens. We conduct experiments on various benchmarks and LLM architectures. Our results show that D2O not only achieves significant memory savings and enhances inference throughput by more than 3$\times$ but also maintains high-quality long-text generation. |
| title | D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.13035 |