StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterization
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914824157396992 |
|---|---|
| author | Wang, Shida Li, Qianxiao |
| author_facet | Wang, Shida Li, Qianxiao |
| contents | In this paper, we investigate the long-term memory learning capabilities of state-space models (SSMs) from the perspective of parameterization. We prove that state-space models without any reparameterization exhibit a memory limitation similar to that of traditional RNNs: the target relationships that can be stably approximated by state-space models must have an exponential decaying memory. Our analysis identifies this "curse of memory" as a result of the recurrent weights converging to a stability boundary, suggesting that a reparameterization technique can be effective. To this end, we introduce a class of reparameterization techniques for SSMs that effectively lift its memory limitations. Besides improving approximation capabilities, we further illustrate that a principled choice of reparameterization scheme can also enhance optimization stability. We validate our findings using synthetic datasets, language models and image classifications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_14495 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterization Wang, Shida Li, Qianxiao Machine Learning Artificial Intelligence Computation and Language Dynamical Systems In this paper, we investigate the long-term memory learning capabilities of state-space models (SSMs) from the perspective of parameterization. We prove that state-space models without any reparameterization exhibit a memory limitation similar to that of traditional RNNs: the target relationships that can be stably approximated by state-space models must have an exponential decaying memory. Our analysis identifies this "curse of memory" as a result of the recurrent weights converging to a stability boundary, suggesting that a reparameterization technique can be effective. To this end, we introduce a class of reparameterization techniques for SSMs that effectively lift its memory limitations. Besides improving approximation capabilities, we further illustrate that a principled choice of reparameterization scheme can also enhance optimization stability. We validate our findings using synthetic datasets, language models and image classifications. |
| title | StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterization |
| topic | Machine Learning Artificial Intelligence Computation and Language Dynamical Systems |
| url | https://arxiv.org/abs/2311.14495 |