DeciMamba: Exploring the Length Extrapolation Potential of Mamba
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916681885941760 |
|---|---|
| author | Ben-Kish, Assaf Zimerman, Itamar Abu-Hussein, Shady Cohen, Nadav Globerson, Amir Wolf, Lior Giryes, Raja |
| author_facet | Ben-Kish, Assaf Zimerman, Itamar Abu-Hussein, Shady Cohen, Nadav Globerson, Amir Wolf, Lior Giryes, Raja |
| contents | Long-range sequence processing poses a significant challenge for Transformers due to their quadratic complexity in input length. A promising alternative is Mamba, which demonstrates high performance and achieves Transformer-level capabilities while requiring substantially fewer computational resources. In this paper we explore the length-generalization capabilities of Mamba, which we find to be relatively limited. Through a series of visualizations and analyses we identify that the limitations arise from a restricted effective receptive field, dictated by the sequence length used during training. To address this constraint, we introduce DeciMamba, a context-extension method specifically designed for Mamba. This mechanism, built on top of a hidden filtering mechanism embedded within the S6 layer, enables the trained model to extrapolate well even without additional training. Empirical experiments over real-world long-range NLP tasks show that DeciMamba can extrapolate to context lengths that are significantly longer than the ones seen during training, while enjoying faster inference. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14528 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DeciMamba: Exploring the Length Extrapolation Potential of Mamba Ben-Kish, Assaf Zimerman, Itamar Abu-Hussein, Shady Cohen, Nadav Globerson, Amir Wolf, Lior Giryes, Raja Machine Learning Artificial Intelligence Long-range sequence processing poses a significant challenge for Transformers due to their quadratic complexity in input length. A promising alternative is Mamba, which demonstrates high performance and achieves Transformer-level capabilities while requiring substantially fewer computational resources. In this paper we explore the length-generalization capabilities of Mamba, which we find to be relatively limited. Through a series of visualizations and analyses we identify that the limitations arise from a restricted effective receptive field, dictated by the sequence length used during training. To address this constraint, we introduce DeciMamba, a context-extension method specifically designed for Mamba. This mechanism, built on top of a hidden filtering mechanism embedded within the S6 layer, enables the trained model to extrapolate well even without additional training. Empirical experiments over real-world long-range NLP tasks show that DeciMamba can extrapolate to context lengths that are significantly longer than the ones seen during training, while enjoying faster inference. |
| title | DeciMamba: Exploring the Length Extrapolation Potential of Mamba |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2406.14528 |