Wanda++: Pruning Large Language Models via Regional Gradients
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912407920574464 |
|---|---|
| author | Yang, Yifan Zhen, Kai Ganesh, Bhavana Galstyan, Aram Huybrechts, Goeric Müller, Markus Kübler, Jonas M. Swaminathan, Rupak Vignesh Mouchtaris, Athanasios Bodapati, Sravan Babu Susanj, Nathan Zhang, Zheng FitzGerald, Jack Kumar, Abhishek |
| author_facet | Yang, Yifan Zhen, Kai Ganesh, Bhavana Galstyan, Aram Huybrechts, Goeric Müller, Markus Kübler, Jonas M. Swaminathan, Rupak Vignesh Mouchtaris, Athanasios Bodapati, Sravan Babu Susanj, Nathan Zhang, Zheng FitzGerald, Jack Kumar, Abhishek |
| contents | Large Language Models (LLMs) pruning seeks to remove unimportant weights for inference speedup with minimal accuracy impact. However, existing methods often suffer from accuracy degradation without full-model sparsity-aware fine-tuning. This paper presents Wanda++, a novel pruning framework that outperforms the state-of-the-art methods by utilizing decoder-block-level \textbf{regional} gradients. Specifically, Wanda++ improves the pruning score with regional gradients for the first time and proposes an efficient regional optimization method to minimize pruning-induced output discrepancies between the dense and sparse decoder output. Notably, Wanda++ improves perplexity by up to 32\% over Wanda in the language modeling task and generalizes effectively to downstream tasks. Moreover, despite updating weights with regional optimization, Wanda++ remains orthogonal to sparsity-aware fine-tuning, further reducing perplexity with LoRA in great extend. Our approach is lightweight, pruning a 7B LLaMA model in under 10 minutes on a single H100 GPU. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_04992 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Wanda++: Pruning Large Language Models via Regional Gradients Yang, Yifan Zhen, Kai Ganesh, Bhavana Galstyan, Aram Huybrechts, Goeric Müller, Markus Kübler, Jonas M. Swaminathan, Rupak Vignesh Mouchtaris, Athanasios Bodapati, Sravan Babu Susanj, Nathan Zhang, Zheng FitzGerald, Jack Kumar, Abhishek Machine Learning Artificial Intelligence Computation and Language Large Language Models (LLMs) pruning seeks to remove unimportant weights for inference speedup with minimal accuracy impact. However, existing methods often suffer from accuracy degradation without full-model sparsity-aware fine-tuning. This paper presents Wanda++, a novel pruning framework that outperforms the state-of-the-art methods by utilizing decoder-block-level \textbf{regional} gradients. Specifically, Wanda++ improves the pruning score with regional gradients for the first time and proposes an efficient regional optimization method to minimize pruning-induced output discrepancies between the dense and sparse decoder output. Notably, Wanda++ improves perplexity by up to 32\% over Wanda in the language modeling task and generalizes effectively to downstream tasks. Moreover, despite updating weights with regional optimization, Wanda++ remains orthogonal to sparsity-aware fine-tuning, further reducing perplexity with LoRA in great extend. Our approach is lightweight, pruning a 7B LLaMA model in under 10 minutes on a single H100 GPU. |
| title | Wanda++: Pruning Large Language Models via Regional Gradients |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2503.04992 |