Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915126498557952 |
|---|---|
| author | Muñoz, J. Pablo Yuan, Jinjie Jain, Nilesh |
| author_facet | Muñoz, J. Pablo Yuan, Jinjie Jain, Nilesh |
| contents | Large pre-trained models have achieved outstanding results in sequence modeling. The Transformer block and its attention mechanism have been the main drivers of the success of these models. Recently, alternative architectures, such as Selective Structured State Space Models (SSMs), have been proposed to address the inefficiencies of Transformers. This paper explores the compression of SSM-based models, particularly Mamba and its hybrids. We study the sensitivity of these models to the removal of selected components at different granularities to reduce the model size and computational overhead, thus improving their efficiency while maintaining accuracy. The proposed solutions, collectively referred to as Mamba-Shedder, achieve a speedup of up to 1.4x during inference, demonstrating that model efficiency can be improved by eliminating several redundancies with minimal impact on the overall model performance. The code is available at https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_17088 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models Muñoz, J. Pablo Yuan, Jinjie Jain, Nilesh Machine Learning Artificial Intelligence Computation and Language I.2.0 Large pre-trained models have achieved outstanding results in sequence modeling. The Transformer block and its attention mechanism have been the main drivers of the success of these models. Recently, alternative architectures, such as Selective Structured State Space Models (SSMs), have been proposed to address the inefficiencies of Transformers. This paper explores the compression of SSM-based models, particularly Mamba and its hybrids. We study the sensitivity of these models to the removal of selected components at different granularities to reduce the model size and computational overhead, thus improving their efficiency while maintaining accuracy. The proposed solutions, collectively referred to as Mamba-Shedder, achieve a speedup of up to 1.4x during inference, demonstrating that model efficiency can be improved by eliminating several redundancies with minimal impact on the overall model performance. The code is available at https://github.com/IntelLabs/Hardware-Aware-Automated-Machine-Learning. |
| title | Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models |
| topic | Machine Learning Artificial Intelligence Computation and Language I.2.0 |
| url | https://arxiv.org/abs/2501.17088 |