Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.21174 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909597559685120 |
|---|---|
| author | Mugnaini, Leandro Giusti Yamamoto, Bruno Lopes de Alcantara, Lucas Lauton Zacarias, Victor Bollis, Edson Pellicer, Lucas Costa, Anna Helena Reali Jordao, Artur |
| author_facet | Mugnaini, Leandro Giusti Yamamoto, Bruno Lopes de Alcantara, Lucas Lauton Zacarias, Victor Bollis, Edson Pellicer, Lucas Costa, Anna Helena Reali Jordao, Artur |
| contents | Deep learning drives a new wave in computing systems and triggers the automation of increasingly complex problems. In particular, Large Language Models (LLMs) have significantly advanced cognitive tasks, often matching or even surpassing human-level performance. However, their extensive parameters result in high computational costs and slow inference, posing challenges for deployment in resource-limited settings. Among the strategies to overcome the aforementioned challenges, pruning emerges as a successful mechanism since it reduces model size while maintaining predictive ability. In this paper, we introduce AMP: Attention Heads and MLP Pruning, a novel structured pruning method that efficiently compresses LLMs by removing less critical structures within Multi-Head Attention (MHA) and Multilayer Perceptron (MLP). By projecting the input data onto weights, AMP assesses structural importance and overcomes the limitations of existing techniques, which often fall short in flexibility or efficiency. In particular, AMP surpasses the current state-of-the-art on commonsense reasoning tasks by up to 1.49 percentage points, achieving a 30% pruning ratio with minimal impact on zero-shot task performance. Moreover, AMP also improves inference speeds, making it well-suited for deployment in resource-constrained environments. We confirm the flexibility of AMP on different families of LLMs, including LLaMA and Phi. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_21174 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Efficient LLMs with AMP: Attention Heads and MLP Pruning Mugnaini, Leandro Giusti Yamamoto, Bruno Lopes de Alcantara, Lucas Lauton Zacarias, Victor Bollis, Edson Pellicer, Lucas Costa, Anna Helena Reali Jordao, Artur Machine Learning Deep learning drives a new wave in computing systems and triggers the automation of increasingly complex problems. In particular, Large Language Models (LLMs) have significantly advanced cognitive tasks, often matching or even surpassing human-level performance. However, their extensive parameters result in high computational costs and slow inference, posing challenges for deployment in resource-limited settings. Among the strategies to overcome the aforementioned challenges, pruning emerges as a successful mechanism since it reduces model size while maintaining predictive ability. In this paper, we introduce AMP: Attention Heads and MLP Pruning, a novel structured pruning method that efficiently compresses LLMs by removing less critical structures within Multi-Head Attention (MHA) and Multilayer Perceptron (MLP). By projecting the input data onto weights, AMP assesses structural importance and overcomes the limitations of existing techniques, which often fall short in flexibility or efficiency. In particular, AMP surpasses the current state-of-the-art on commonsense reasoning tasks by up to 1.49 percentage points, achieving a 30% pruning ratio with minimal impact on zero-shot task performance. Moreover, AMP also improves inference speeds, making it well-suited for deployment in resource-constrained environments. We confirm the flexibility of AMP on different families of LLMs, including LLaMA and Phi. |
| title | Efficient LLMs with AMP: Attention Heads and MLP Pruning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2504.21174 |