Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.11057 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914950787629056 |
|---|---|
| author | Lv, Bo Zhou, Quan Ding, Xuanang Wang, Yan Ma, Zeming |
| author_facet | Lv, Bo Zhou, Quan Ding, Xuanang Wang, Yan Ma, Zeming |
| contents | The bottleneck associated with the key-value(KV) cache presents a significant challenge during the inference processes of large language models. While depth pruning accelerates inference, it requires extensive recovery training, which can take up to two weeks. On the other hand, width pruning retains much of the performance but offers slight speed gains. To tackle these challenges, we propose KVPruner to improve model efficiency while maintaining performance. Our method uses global perplexity-based analysis to determine the importance ratio for each block and provides multiple strategies to prune non-essential KV channels within blocks. Compared to the original model, KVPruner reduces runtime memory usage by 50% and boosts throughput by over 35%. Additionally, our method requires only two hours of LoRA fine-tuning on small datasets to recover most of the performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_11057 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | KVPruner: Structural Pruning for Faster and Memory-Efficient Large Language Models Lv, Bo Zhou, Quan Ding, Xuanang Wang, Yan Ma, Zeming Computation and Language The bottleneck associated with the key-value(KV) cache presents a significant challenge during the inference processes of large language models. While depth pruning accelerates inference, it requires extensive recovery training, which can take up to two weeks. On the other hand, width pruning retains much of the performance but offers slight speed gains. To tackle these challenges, we propose KVPruner to improve model efficiency while maintaining performance. Our method uses global perplexity-based analysis to determine the importance ratio for each block and provides multiple strategies to prune non-essential KV channels within blocks. Compared to the original model, KVPruner reduces runtime memory usage by 50% and boosts throughput by over 35%. Additionally, our method requires only two hours of LoRA fine-tuning on small datasets to recover most of the performance. |
| title | KVPruner: Structural Pruning for Faster and Memory-Efficient Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2409.11057 |