HiPPO-Prophecy: State-Space Models can Provably Learn Dynamical Systems in Context
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911087591424000 |
|---|---|
| author | Joseph, Federico Arangath Haefeli, Kilian Konstantin Liniger, Noah Gulcehre, Caglar |
| author_facet | Joseph, Federico Arangath Haefeli, Kilian Konstantin Liniger, Noah Gulcehre, Caglar |
| contents | This work explores the in-context learning capabilities of State Space Models (SSMs) and presents, to the best of our knowledge, the first theoretical explanation of a possible underlying mechanism. We introduce a novel weight construction for SSMs, enabling them to predict the next state of any dynamical system after observing previous states without parameter fine-tuning. This is accomplished by extending the HiPPO framework to demonstrate that continuous SSMs can approximate the derivative of any input signal. Specifically, we find an explicit weight construction for continuous SSMs and provide an asymptotic error bound on the derivative approximation. The discretization of this continuous SSM subsequently yields a discrete SSM that predicts the next state. Finally, we demonstrate the effectiveness of our parameterization empirically. This work should be an initial step toward understanding how sequence models based on SSMs learn in context. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_09375 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | HiPPO-Prophecy: State-Space Models can Provably Learn Dynamical Systems in Context Joseph, Federico Arangath Haefeli, Kilian Konstantin Liniger, Noah Gulcehre, Caglar Machine Learning This work explores the in-context learning capabilities of State Space Models (SSMs) and presents, to the best of our knowledge, the first theoretical explanation of a possible underlying mechanism. We introduce a novel weight construction for SSMs, enabling them to predict the next state of any dynamical system after observing previous states without parameter fine-tuning. This is accomplished by extending the HiPPO framework to demonstrate that continuous SSMs can approximate the derivative of any input signal. Specifically, we find an explicit weight construction for continuous SSMs and provide an asymptotic error bound on the derivative approximation. The discretization of this continuous SSM subsequently yields a discrete SSM that predicts the next state. Finally, we demonstrate the effectiveness of our parameterization empirically. This work should be an initial step toward understanding how sequence models based on SSMs learn in context. |
| title | HiPPO-Prophecy: State-Space Models can Provably Learn Dynamical Systems in Context |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2407.09375 |