On the Expressive Power and Limitations of Multi-Layer SSMs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918449920344064 |
|---|---|
| author | Zubić, Nikola Li, Qian Wang, Yuyi Scaramuzza, Davide |
| author_facet | Zubić, Nikola Li, Qian Wang, Yuyi Scaramuzza, Davide |
| contents | We study the expressive power and limitations of multi-layer state-space models (SSMs). First, we show that multi-layer SSMs face fundamental limitations in compositional tasks, revealing an inherent gap between SSMs and streaming models. Then, we examine the role of chain-of-thought (CoT), showing that offline CoT does not fundamentally increase the expressiveness, while online CoT can substantially increase its power. Indeed, with online CoT, multi-layer SSMs become equivalent in power to streaming algorithms. Finally, we investigate the tradeoff between width and precision, showing that these resources are not interchangeable in the base model, but admit a clean equivalence once online CoT is allowed. Overall, our results offer a unified perspective on how depth, finite precision, and CoT shape the power and limits of SSMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_14501 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | On the Expressive Power and Limitations of Multi-Layer SSMs Zubić, Nikola Li, Qian Wang, Yuyi Scaramuzza, Davide Machine Learning Artificial Intelligence Computational Complexity We study the expressive power and limitations of multi-layer state-space models (SSMs). First, we show that multi-layer SSMs face fundamental limitations in compositional tasks, revealing an inherent gap between SSMs and streaming models. Then, we examine the role of chain-of-thought (CoT), showing that offline CoT does not fundamentally increase the expressiveness, while online CoT can substantially increase its power. Indeed, with online CoT, multi-layer SSMs become equivalent in power to streaming algorithms. Finally, we investigate the tradeoff between width and precision, showing that these resources are not interchangeable in the base model, but admit a clean equivalence once online CoT is allowed. Overall, our results offer a unified perspective on how depth, finite precision, and CoT shape the power and limits of SSMs. |
| title | On the Expressive Power and Limitations of Multi-Layer SSMs |
| topic | Machine Learning Artificial Intelligence Computational Complexity |
| url | https://arxiv.org/abs/2604.14501 |