Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2510.14027 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912865434206208 |
|---|---|
| author | Zattra, Riccardo Baggio, Giacomo Casti, Umberto Ferrante, Augusto Ticozzi, Francesco |
| author_facet | Zattra, Riccardo Baggio, Giacomo Casti, Umberto Ferrante, Augusto Ticozzi, Francesco |
| contents | Transformers, powered by the attention mechanism, are the backbone of most foundation models, yet they suffer from quadratic complexity and difficulties in dealing with long-range dependencies in the input sequence. Recent work has shown that state space models (SSMs) provide a promising alternative. In this paper, we introduce the COFFEE (COntext From FEEdback) model, a novel time-varying SSM that incorporates state feedback to enable context-dependent selectivity, while still allowing for parallel implementation. This idea allows the model to regulate its dynamics based on the context described by the internal state, which embodies a compact representation of the input history. State feedback allows COFFEE to improve its ability to capture long-range dependencies: on the induction head task, it achieves near-perfect accuracy with two orders of magnitude fewer parameters and training sequences compared to S6 (the SSM of Mamba). On MNIST, COFFEE largely outperforms S6 within the same architecture, reaching 97% accuracy with only 3585 parameters. These results showcase the role of state feedback as a key mechanism for building scalable and efficient sequence models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_14027 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Context-Selective State Space Models: Feedback is All You Need Zattra, Riccardo Baggio, Giacomo Casti, Umberto Ferrante, Augusto Ticozzi, Francesco Machine Learning Artificial Intelligence Transformers, powered by the attention mechanism, are the backbone of most foundation models, yet they suffer from quadratic complexity and difficulties in dealing with long-range dependencies in the input sequence. Recent work has shown that state space models (SSMs) provide a promising alternative. In this paper, we introduce the COFFEE (COntext From FEEdback) model, a novel time-varying SSM that incorporates state feedback to enable context-dependent selectivity, while still allowing for parallel implementation. This idea allows the model to regulate its dynamics based on the context described by the internal state, which embodies a compact representation of the input history. State feedback allows COFFEE to improve its ability to capture long-range dependencies: on the induction head task, it achieves near-perfect accuracy with two orders of magnitude fewer parameters and training sequences compared to S6 (the SSM of Mamba). On MNIST, COFFEE largely outperforms S6 within the same architecture, reaching 97% accuracy with only 3585 parameters. These results showcase the role of state feedback as a key mechanism for building scalable and efficient sequence models. |
| title | Context-Selective State Space Models: Feedback is All You Need |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2510.14027 |