MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910342964051968 |
|---|---|
| author | Pióro, Maciej Ciebiera, Kamil Król, Krystian Ludziejewski, Jan Krutul, Michał Krajewski, Jakub Antoniak, Szymon Miłoś, Piotr Cygan, Marek Jaszczur, Sebastian |
| author_facet | Pióro, Maciej Ciebiera, Kamil Król, Krystian Ludziejewski, Jan Krutul, Michał Krajewski, Jakub Antoniak, Szymon Miłoś, Piotr Cygan, Marek Jaszczur, Sebastian |
| contents | State Space Models (SSMs) have become serious contenders in the field of sequential modeling, challenging the dominance of Transformers. At the same time, Mixture of Experts (MoE) has significantly improved Transformer-based Large Language Models, including recent state-of-the-art open models. We propose that to unlock the potential of SSMs for scaling, they should be combined with MoE. We showcase this on Mamba, a recent SSM-based model that achieves remarkable performance. Our model, MoE-Mamba, outperforms both Mamba and baseline Transformer-MoE. In particular, MoE-Mamba reaches the same performance as Mamba in $2.35\times$ fewer training steps while preserving the inference performance gains of Mamba against Transformer. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_04081 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts Pióro, Maciej Ciebiera, Kamil Król, Krystian Ludziejewski, Jan Krutul, Michał Krajewski, Jakub Antoniak, Szymon Miłoś, Piotr Cygan, Marek Jaszczur, Sebastian Machine Learning Artificial Intelligence Computation and Language State Space Models (SSMs) have become serious contenders in the field of sequential modeling, challenging the dominance of Transformers. At the same time, Mixture of Experts (MoE) has significantly improved Transformer-based Large Language Models, including recent state-of-the-art open models. We propose that to unlock the potential of SSMs for scaling, they should be combined with MoE. We showcase this on Mamba, a recent SSM-based model that achieves remarkable performance. Our model, MoE-Mamba, outperforms both Mamba and baseline Transformer-MoE. In particular, MoE-Mamba reaches the same performance as Mamba in $2.35\times$ fewer training steps while preserving the inference performance gains of Mamba against Transformer. |
| title | MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2401.04081 |