Numerical Pruning for Efficient Autoregressive Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915067121893376 |
|---|---|
| author | Shen, Xuan Song, Zhao Zhou, Yufa Chen, Bo Liu, Jing Zhang, Ruiyi Rossi, Ryan A. Tan, Hao Yu, Tong Chen, Xiang Zhou, Yufan Sun, Tong Zhao, Pu Wang, Yanzhi Gu, Jiuxiang |
| author_facet | Shen, Xuan Song, Zhao Zhou, Yufa Chen, Bo Liu, Jing Zhang, Ruiyi Rossi, Ryan A. Tan, Hao Yu, Tong Chen, Xiang Zhou, Yufan Sun, Tong Zhao, Pu Wang, Yanzhi Gu, Jiuxiang |
| contents | Transformers have emerged as the leading architecture in deep learning, proving to be versatile and highly effective across diverse domains beyond language and image processing. However, their impressive performance often incurs high computational costs due to their substantial model size. This paper focuses on compressing decoder-only transformer-based autoregressive models through structural weight pruning to improve the model efficiency while preserving performance for both language and image generation tasks. Specifically, we propose a training-free pruning method that calculates a numerical score with Newton's method for the Attention and MLP modules, respectively. Besides, we further propose another compensation algorithm to recover the pruned model for better performance. To verify the effectiveness of our method, we provide both theoretical support and extensive experiments. Our experiments show that our method achieves state-of-the-art performance with reduced memory usage and faster generation speeds on GPUs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_12441 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Numerical Pruning for Efficient Autoregressive Models Shen, Xuan Song, Zhao Zhou, Yufa Chen, Bo Liu, Jing Zhang, Ruiyi Rossi, Ryan A. Tan, Hao Yu, Tong Chen, Xiang Zhou, Yufan Sun, Tong Zhao, Pu Wang, Yanzhi Gu, Jiuxiang Machine Learning Artificial Intelligence Transformers have emerged as the leading architecture in deep learning, proving to be versatile and highly effective across diverse domains beyond language and image processing. However, their impressive performance often incurs high computational costs due to their substantial model size. This paper focuses on compressing decoder-only transformer-based autoregressive models through structural weight pruning to improve the model efficiency while preserving performance for both language and image generation tasks. Specifically, we propose a training-free pruning method that calculates a numerical score with Newton's method for the Attention and MLP modules, respectively. Besides, we further propose another compensation algorithm to recover the pruned model for better performance. To verify the effectiveness of our method, we provide both theoretical support and extensive experiments. Our experiments show that our method achieves state-of-the-art performance with reduced memory usage and faster generation speeds on GPUs. |
| title | Numerical Pruning for Efficient Autoregressive Models |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2412.12441 |