On Pruning State-Space LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909824735772672 |
|---|---|
| author | Ghattas, Tamer Hassid, Michael Schwartz, Roy |
| author_facet | Ghattas, Tamer Hassid, Michael Schwartz, Roy |
| contents | Recent work proposed state-space models (SSMs) as an efficient alternative to transformer-based LLMs. Can these models be pruned to further reduce their computation costs? We adapt several pruning methods to the SSM structure, and apply them to four SSM-based LLMs across multiple tasks. We find that such models are quite robust to some pruning methods (e.g. WANDA), while using other methods lead to fast performance degradation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_18886 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | On Pruning State-Space LLMs Ghattas, Tamer Hassid, Michael Schwartz, Roy Computation and Language Machine Learning Recent work proposed state-space models (SSMs) as an efficient alternative to transformer-based LLMs. Can these models be pruned to further reduce their computation costs? We adapt several pruning methods to the SSM structure, and apply them to four SSM-based LLMs across multiple tasks. We find that such models are quite robust to some pruning methods (e.g. WANDA), while using other methods lead to fast performance degradation. |
| title | On Pruning State-Space LLMs |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2502.18886 |