Revisiting Large Language Model Pruning using Neuron Semantic Attribution
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909522439700480 |
|---|---|
| author | Ding, Yizhuo Sun, Xinwei Fu, Yanwei Hu, Guosheng |
| author_facet | Ding, Yizhuo Sun, Xinwei Fu, Yanwei Hu, Guosheng |
| contents | Model pruning technique is vital for accelerating large language models by reducing their size and computational requirements. However, the generalizability of existing pruning methods across diverse datasets and tasks remains unclear. Thus, we conduct extensive evaluations on 24 datasets and 4 tasks using popular pruning methods. Based on these evaluations, we find and then investigate that calibration set greatly affect the performance of pruning methods. In addition, we surprisingly find a significant performance drop of existing pruning methods in sentiment classification tasks. To understand the link between performance drop and pruned neurons, we propose Neuron Semantic Attribution, which learns to associate each neuron with specific semantics. This method first makes the unpruned neurons of LLMs explainable. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01542 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Revisiting Large Language Model Pruning using Neuron Semantic Attribution Ding, Yizhuo Sun, Xinwei Fu, Yanwei Hu, Guosheng Computation and Language Artificial Intelligence Model pruning technique is vital for accelerating large language models by reducing their size and computational requirements. However, the generalizability of existing pruning methods across diverse datasets and tasks remains unclear. Thus, we conduct extensive evaluations on 24 datasets and 4 tasks using popular pruning methods. Based on these evaluations, we find and then investigate that calibration set greatly affect the performance of pruning methods. In addition, we surprisingly find a significant performance drop of existing pruning methods in sentiment classification tasks. To understand the link between performance drop and pruned neurons, we propose Neuron Semantic Attribution, which learns to associate each neuron with specific semantics. This method first makes the unpruned neurons of LLMs explainable. |
| title | Revisiting Large Language Model Pruning using Neuron Semantic Attribution |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.01542 |