Flash STU: Fast Spectral Transform Units
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911380826750976 |
|---|---|
| author | Liu, Y. Isabel Nguyen, Windsor Devre, Yagiz Dogariu, Evan Majumdar, Anirudha Hazan, Elad |
| author_facet | Liu, Y. Isabel Nguyen, Windsor Devre, Yagiz Dogariu, Evan Majumdar, Anirudha Hazan, Elad |
| contents | Recent advances in state-space model architectures have shown great promise for efficient sequence modeling, but challenges remain in balancing computational efficiency with model expressiveness. We propose the Flash STU architecture, a hybrid model that interleaves spectral state space model layers with sliding window attention, enabling scalability to billions of parameters for language modeling while maintaining a near-linear time complexity. We evaluate the Flash STU and its variants on diverse sequence prediction tasks, including linear dynamical systems, robotics control, and language modeling. We find that, given a fixed parameter budget, the Flash STU architecture consistently outperforms the Transformer and other leading state-space models such as S4 and Mamba-2. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_10489 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Flash STU: Fast Spectral Transform Units Liu, Y. Isabel Nguyen, Windsor Devre, Yagiz Dogariu, Evan Majumdar, Anirudha Hazan, Elad Machine Learning Artificial Intelligence Recent advances in state-space model architectures have shown great promise for efficient sequence modeling, but challenges remain in balancing computational efficiency with model expressiveness. We propose the Flash STU architecture, a hybrid model that interleaves spectral state space model layers with sliding window attention, enabling scalability to billions of parameters for language modeling while maintaining a near-linear time complexity. We evaluate the Flash STU and its variants on diverse sequence prediction tasks, including linear dynamical systems, robotics control, and language modeling. We find that, given a fixed parameter budget, the Flash STU architecture consistently outperforms the Transformer and other leading state-space models such as S4 and Mamba-2. |
| title | Flash STU: Fast Spectral Transform Units |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2409.10489 |