Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.19126 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911969592737792 |
|---|---|
| author | Li, Jianwei Dong, Yijun Lei, Qi |
| author_facet | Li, Jianwei Dong, Yijun Lei, Qi |
| contents | To remove redundant components of large language models (LLMs) without incurring significant computational costs, this work focuses on single-shot pruning without a retraining phase. We simplify the pruning process for Transformer-based LLMs by identifying a depth-2 pruning structure that functions independently. Additionally, we propose two inference-aware pruning criteria derived from the optimization perspective of output approximation, which outperforms traditional training-aware metrics such as gradient and Hessian. We also introduce a two-step reconstruction technique to mitigate pruning errors without model retraining. Experimental results demonstrate that our approach significantly reduces computational costs and hardware requirements while maintaining superior performance across various datasets and models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_19126 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Greedy Output Approximation: Towards Efficient Structured Pruning for LLMs Without Retraining Li, Jianwei Dong, Yijun Lei, Qi Artificial Intelligence To remove redundant components of large language models (LLMs) without incurring significant computational costs, this work focuses on single-shot pruning without a retraining phase. We simplify the pruning process for Transformer-based LLMs by identifying a depth-2 pruning structure that functions independently. Additionally, we propose two inference-aware pruning criteria derived from the optimization perspective of output approximation, which outperforms traditional training-aware metrics such as gradient and Hessian. We also introduce a two-step reconstruction technique to mitigate pruning errors without model retraining. Experimental results demonstrate that our approach significantly reduces computational costs and hardware requirements while maintaining superior performance across various datasets and models. |
| title | Greedy Output Approximation: Towards Efficient Structured Pruning for LLMs Without Retraining |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2407.19126 |