Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910048715800576 |
|---|---|
| author | Liu, Jun Kong, Zhenglun Zhao, Pu Yang, Changdi Tang, Hao Shen, Xuan Yuan, Geng Niu, Wei Zhang, Wenbin Lin, Xue Huang, Dong Wang, Yanzhi |
| author_facet | Liu, Jun Kong, Zhenglun Zhao, Pu Yang, Changdi Tang, Hao Shen, Xuan Yuan, Geng Niu, Wei Zhang, Wenbin Lin, Xue Huang, Dong Wang, Yanzhi |
| contents | Structured pruning for large language models (LLMs) has garnered significant academic interest due to its ability to efficiently compress and accelerate LLMs by eliminating redundant weight groups at a coarse-grained granularity. Current structured pruning methods for LLMs typically depend on a singular granularity for assessing weight importance, resulting in notable performance degradation in downstream tasks. Intriguingly, our empirical investigations reveal that utilizing unstructured pruning, which achieves better performance retention by pruning weights at a finer granularity, \emph{i.e.}, individual weights, yields significantly varied sparse LLM structures when juxtaposed to structured pruning. This suggests that evaluating both holistic and individual assessment for weight importance is essential for LLM pruning. Building on this insight, we introduce the Hybrid-grained Weight Importance Assessment (HyWIA), a novel method that merges fine-grained and coarse-grained evaluations of weight importance for the pruning of LLMs. Leveraging an attention mechanism, HyWIA adaptively determines the optimal blend of granularity in weight importance assessments in an end-to-end pruning manner. Extensive experiments on LLaMA-V1/V2, Vicuna, Baichuan, and Bloom across various benchmarks demonstrate the effectiveness of HyWIA in pruning LLMs. For example, HyWIA surpasses the cutting-edge LLM-Pruner by an average margin of 2.82% in accuracy across seven downstream tasks when pruning LLaMA-7B by 50%. Code:https://github.com/azuryl/LLM-HWIA |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_10799 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment Liu, Jun Kong, Zhenglun Zhao, Pu Yang, Changdi Tang, Hao Shen, Xuan Yuan, Geng Niu, Wei Zhang, Wenbin Lin, Xue Huang, Dong Wang, Yanzhi Computation and Language Artificial Intelligence Machine Learning Structured pruning for large language models (LLMs) has garnered significant academic interest due to its ability to efficiently compress and accelerate LLMs by eliminating redundant weight groups at a coarse-grained granularity. Current structured pruning methods for LLMs typically depend on a singular granularity for assessing weight importance, resulting in notable performance degradation in downstream tasks. Intriguingly, our empirical investigations reveal that utilizing unstructured pruning, which achieves better performance retention by pruning weights at a finer granularity, \emph{i.e.}, individual weights, yields significantly varied sparse LLM structures when juxtaposed to structured pruning. This suggests that evaluating both holistic and individual assessment for weight importance is essential for LLM pruning. Building on this insight, we introduce the Hybrid-grained Weight Importance Assessment (HyWIA), a novel method that merges fine-grained and coarse-grained evaluations of weight importance for the pruning of LLMs. Leveraging an attention mechanism, HyWIA adaptively determines the optimal blend of granularity in weight importance assessments in an end-to-end pruning manner. Extensive experiments on LLaMA-V1/V2, Vicuna, Baichuan, and Bloom across various benchmarks demonstrate the effectiveness of HyWIA in pruning LLMs. For example, HyWIA surpasses the cutting-edge LLM-Pruner by an average margin of 2.82% in accuracy across seven downstream tasks when pruning LLaMA-7B by 50%. Code:https://github.com/azuryl/LLM-HWIA |
| title | Toward Adaptive Large Language Models Structured Pruning via Hybrid-grained Weight Importance Assessment |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2403.10799 |