Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2412.11494 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909430392553472 |
|---|---|
| author | Li, Zekai Zheng, Jintu Liu, Ji Liu, Han Zhu, Haowei Li, Zeping Yang, Fuwei Huang, Haiduo Peng, Jinzhang Li, Dong Tian, Lu Barsoum, Emad |
| author_facet | Li, Zekai Zheng, Jintu Liu, Ji Liu, Han Zhu, Haowei Li, Zeping Yang, Fuwei Huang, Haiduo Peng, Jinzhang Li, Dong Tian, Lu Barsoum, Emad |
| contents | Recently, large language models (LLMs) have demonstrated superior performance across various tasks by adhering to scaling laws, which significantly increase model size. However, the huge computation overhead during inference hinders the deployment in industrial applications. Many works leverage traditional compression approaches to boost model inference, but these always introduce additional training costs to restore the performance and the pruning results typically show noticeable performance drops compared to the original model when aiming for a specific level of acceleration. To address these issues, we propose a fine-grained token-wise pruning approach for the LLMs, which presents a learnable router to adaptively identify the less important tokens and skip them across model blocks to reduce computational cost during inference. To construct the router efficiently, we present a search-based sparsity scheduler for pruning sparsity allocation, a trainable router combined with our proposed four low-dimensional factors as input and three proposed losses. We conduct extensive experiments across different benchmarks on different LLMs to demonstrate the superiority of our method. Our approach achieves state-of-the-art (SOTA) pruning results, surpassing other existing pruning methods. For instance, our method outperforms BlockPruner and ShortGPT by approximately 10 points on both LLaMA2-7B and Qwen1.5-7B in accuracy retention at comparable token sparsity levels. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_11494 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing Li, Zekai Zheng, Jintu Liu, Ji Liu, Han Zhu, Haowei Li, Zeping Yang, Fuwei Huang, Haiduo Peng, Jinzhang Li, Dong Tian, Lu Barsoum, Emad Computation and Language Recently, large language models (LLMs) have demonstrated superior performance across various tasks by adhering to scaling laws, which significantly increase model size. However, the huge computation overhead during inference hinders the deployment in industrial applications. Many works leverage traditional compression approaches to boost model inference, but these always introduce additional training costs to restore the performance and the pruning results typically show noticeable performance drops compared to the original model when aiming for a specific level of acceleration. To address these issues, we propose a fine-grained token-wise pruning approach for the LLMs, which presents a learnable router to adaptively identify the less important tokens and skip them across model blocks to reduce computational cost during inference. To construct the router efficiently, we present a search-based sparsity scheduler for pruning sparsity allocation, a trainable router combined with our proposed four low-dimensional factors as input and three proposed losses. We conduct extensive experiments across different benchmarks on different LLMs to demonstrate the superiority of our method. Our approach achieves state-of-the-art (SOTA) pruning results, surpassing other existing pruning methods. For instance, our method outperforms BlockPruner and ShortGPT by approximately 10 points on both LLaMA2-7B and Qwen1.5-7B in accuracy retention at comparable token sparsity levels. |
| title | FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.11494 |