On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912219134951424 |
|---|---|
| author | Cohen-Karlik, Edo Zimerman, Itamar Galanti, Liane Atad, Ido Globerson, Amir Wolf, Lior |
| author_facet | Cohen-Karlik, Edo Zimerman, Itamar Galanti, Liane Atad, Ido Globerson, Amir Wolf, Lior |
| contents | Recent advances in efficient sequence modeling have introduced selective state-space layers, a key component of the Mamba architecture, which have demonstrated remarkable success in a wide range of NLP and vision tasks. While Mamba's empirical performance has matched or surpassed SoTA transformers on such diverse benchmarks, the theoretical foundations underlying its powerful representational capabilities remain less explored. In this work, we investigate the expressivity of selective state-space layers using multivariate polynomials, and prove that they surpass linear transformers in expressiveness. Consequently, our findings reveal that Mamba offers superior representational power over linear attention-based models for long sequences, while not sacrificing their generalization. Our theoretical insights are validated by a comprehensive set of empirical experiments on various datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_02209 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach Cohen-Karlik, Edo Zimerman, Itamar Galanti, Liane Atad, Ido Globerson, Amir Wolf, Lior Machine Learning 14J60 F.2.2; I.2.7 Recent advances in efficient sequence modeling have introduced selective state-space layers, a key component of the Mamba architecture, which have demonstrated remarkable success in a wide range of NLP and vision tasks. While Mamba's empirical performance has matched or surpassed SoTA transformers on such diverse benchmarks, the theoretical foundations underlying its powerful representational capabilities remain less explored. In this work, we investigate the expressivity of selective state-space layers using multivariate polynomials, and prove that they surpass linear transformers in expressiveness. Consequently, our findings reveal that Mamba offers superior representational power over linear attention-based models for long sequences, while not sacrificing their generalization. Our theoretical insights are validated by a comprehensive set of empirical experiments on various datasets. |
| title | On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach |
| topic | Machine Learning 14J60 F.2.2; I.2.7 |
| url | https://arxiv.org/abs/2502.02209 |