Towards a theory of learning dynamics in deep state space models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929415824343040 |
|---|---|
| author | Smékal, Jakub Smith, Jimmy T. H. Kleinman, Michael Biderman, Dan Linderman, Scott W. |
| author_facet | Smékal, Jakub Smith, Jimmy T. H. Kleinman, Michael Biderman, Dan Linderman, Scott W. |
| contents | State space models (SSMs) have shown remarkable empirical performance on many long sequence modeling tasks, but a theoretical understanding of these models is still lacking. In this work, we study the learning dynamics of linear SSMs to understand how covariance structure in data, latent state size, and initialization affect the evolution of parameters throughout learning with gradient descent. We show that focusing on the learning dynamics in the frequency domain affords analytical solutions under mild assumptions, and we establish a link between one-dimensional SSMs and the dynamics of deep linear feed-forward networks. Finally, we analyze how latent state over-parameterization affects convergence time and describe future work in extending our results to the study of deep SSMs with nonlinear connections. This work is a step toward a theory of learning dynamics in deep state space models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_07279 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards a theory of learning dynamics in deep state space models Smékal, Jakub Smith, Jimmy T. H. Kleinman, Michael Biderman, Dan Linderman, Scott W. Machine Learning State space models (SSMs) have shown remarkable empirical performance on many long sequence modeling tasks, but a theoretical understanding of these models is still lacking. In this work, we study the learning dynamics of linear SSMs to understand how covariance structure in data, latent state size, and initialization affect the evolution of parameters throughout learning with gradient descent. We show that focusing on the learning dynamics in the frequency domain affords analytical solutions under mild assumptions, and we establish a link between one-dimensional SSMs and the dynamics of deep linear feed-forward networks. Finally, we analyze how latent state over-parameterization affects convergence time and describe future work in extending our results to the study of deep SSMs with nonlinear connections. This work is a step toward a theory of learning dynamics in deep state space models. |
| title | Towards a theory of learning dynamics in deep state space models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2407.07279 |