Pangu Light: Weight Re-Initialization for Pruning and Accelerating LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910968981749760 |
|---|---|
| author | Chen, Hanting Qin, Jiarui Guo, Jialong Yuan, Tao Yin, Yichun Zhen, Huiling Wang, Yasheng Li, Jinpeng Meng, Xiaojun Zhang, Meng Ruan, Rongju Bai, Zheyuan Tang, Yehui Chen, Can Chen, Xinghao Yu, Fisher Tang, Ruiming Wang, Yunhe |
| author_facet | Chen, Hanting Qin, Jiarui Guo, Jialong Yuan, Tao Yin, Yichun Zhen, Huiling Wang, Yasheng Li, Jinpeng Meng, Xiaojun Zhang, Meng Ruan, Rongju Bai, Zheyuan Tang, Yehui Chen, Can Chen, Xinghao Yu, Fisher Tang, Ruiming Wang, Yunhe |
| contents | Large Language Models (LLMs) deliver state-of-the-art capabilities across numerous tasks, but their immense size and inference costs pose significant computational challenges for practical deployment. While structured pruning offers a promising avenue for model compression, existing methods often struggle with the detrimental effects of aggressive, simultaneous width and depth reductions, leading to substantial performance degradation. This paper argues that a critical, often overlooked, aspect in making such aggressive joint pruning viable is the strategic re-initialization and adjustment of remaining weights to improve the model post-pruning training accuracies. We introduce Pangu Light, a framework for LLM acceleration centered around structured pruning coupled with novel weight re-initialization techniques designed to address this ``missing piece''. Our framework systematically targets multiple axes, including model width, depth, attention heads, and RMSNorm, with its effectiveness rooted in novel re-initialization methods like Cross-Layer Attention Pruning (CLAP) and Stabilized LayerNorm Pruning (SLNP) that mitigate performance drops by providing the network a better training starting point. Further enhancing efficiency, Pangu Light incorporates specialized optimizations such as absorbing Post-RMSNorm computations and tailors its strategies to Ascend NPU characteristics. The Pangu Light models consistently exhibit a superior accuracy-efficiency trade-off, outperforming prominent baseline pruning methods like Nemotron and established LLMs like Qwen3 series. For instance, on Ascend NPUs, Pangu Light-32B's 81.6 average score and 2585 tokens/s throughput exceed Qwen3-32B's 80.9 average score and 2225 tokens/s. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_20155 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Pangu Light: Weight Re-Initialization for Pruning and Accelerating LLMs Chen, Hanting Qin, Jiarui Guo, Jialong Yuan, Tao Yin, Yichun Zhen, Huiling Wang, Yasheng Li, Jinpeng Meng, Xiaojun Zhang, Meng Ruan, Rongju Bai, Zheyuan Tang, Yehui Chen, Can Chen, Xinghao Yu, Fisher Tang, Ruiming Wang, Yunhe Computation and Language Large Language Models (LLMs) deliver state-of-the-art capabilities across numerous tasks, but their immense size and inference costs pose significant computational challenges for practical deployment. While structured pruning offers a promising avenue for model compression, existing methods often struggle with the detrimental effects of aggressive, simultaneous width and depth reductions, leading to substantial performance degradation. This paper argues that a critical, often overlooked, aspect in making such aggressive joint pruning viable is the strategic re-initialization and adjustment of remaining weights to improve the model post-pruning training accuracies. We introduce Pangu Light, a framework for LLM acceleration centered around structured pruning coupled with novel weight re-initialization techniques designed to address this ``missing piece''. Our framework systematically targets multiple axes, including model width, depth, attention heads, and RMSNorm, with its effectiveness rooted in novel re-initialization methods like Cross-Layer Attention Pruning (CLAP) and Stabilized LayerNorm Pruning (SLNP) that mitigate performance drops by providing the network a better training starting point. Further enhancing efficiency, Pangu Light incorporates specialized optimizations such as absorbing Post-RMSNorm computations and tailors its strategies to Ascend NPU characteristics. The Pangu Light models consistently exhibit a superior accuracy-efficiency trade-off, outperforming prominent baseline pruning methods like Nemotron and established LLMs like Qwen3 series. For instance, on Ascend NPUs, Pangu Light-32B's 81.6 average score and 2585 tokens/s throughput exceed Qwen3-32B's 80.9 average score and 2225 tokens/s. |
| title | Pangu Light: Weight Re-Initialization for Pruning and Accelerating LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.20155 |