Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.03464 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917794201731072 |
|---|---|
| author | Soydan, Taylan Zubić, Nikola Messikommer, Nico Mishra, Siddhartha Scaramuzza, Davide |
| author_facet | Soydan, Taylan Zubić, Nikola Messikommer, Nico Mishra, Siddhartha Scaramuzza, Davide |
| contents | A central challenge in sequence modeling is efficiently handling tasks with extended contexts. While recent state-space models (SSMs) have made significant progress in this area, they often lack input-dependent filtering or require substantial increases in model complexity to handle input variability. We address this gap by introducing S7, a simplified yet powerful SSM that can handle input dependence while incorporating stable reparameterization and specific design choices to dynamically adjust state transitions based on input content, maintaining efficiency and performance. We prove that this reparameterization ensures stability in long-sequence modeling by keeping state transitions well-behaved over time. Additionally, it controls the gradient norm, enabling efficient training and preventing issues like exploding or vanishing gradients. S7 significantly outperforms baselines across various sequence modeling tasks, including neuromorphic event-based datasets, Long Range Arena benchmarks, and various physical and biological time series. Overall, S7 offers a more straightforward approach to sequence modeling without relying on complex, domain-specific inductive biases, achieving significant improvements across key benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_03464 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | S7: Selective and Simplified State Space Layers for Sequence Modeling Soydan, Taylan Zubić, Nikola Messikommer, Nico Mishra, Siddhartha Scaramuzza, Davide Machine Learning Signal Processing Dynamical Systems A central challenge in sequence modeling is efficiently handling tasks with extended contexts. While recent state-space models (SSMs) have made significant progress in this area, they often lack input-dependent filtering or require substantial increases in model complexity to handle input variability. We address this gap by introducing S7, a simplified yet powerful SSM that can handle input dependence while incorporating stable reparameterization and specific design choices to dynamically adjust state transitions based on input content, maintaining efficiency and performance. We prove that this reparameterization ensures stability in long-sequence modeling by keeping state transitions well-behaved over time. Additionally, it controls the gradient norm, enabling efficient training and preventing issues like exploding or vanishing gradients. S7 significantly outperforms baselines across various sequence modeling tasks, including neuromorphic event-based datasets, Long Range Arena benchmarks, and various physical and biological time series. Overall, S7 offers a more straightforward approach to sequence modeling without relying on complex, domain-specific inductive biases, achieving significant improvements across key benchmarks. |
| title | S7: Selective and Simplified State Space Layers for Sequence Modeling |
| topic | Machine Learning Signal Processing Dynamical Systems |
| url | https://arxiv.org/abs/2410.03464 |