Dynamically Learning to Integrate in Recurrent Neural Networks
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913754384433152 |
|---|---|
| author | Bordelon, Blake Cotler, Jordan Pehlevan, Cengiz Zavatone-Veth, Jacob A. |
| author_facet | Bordelon, Blake Cotler, Jordan Pehlevan, Cengiz Zavatone-Veth, Jacob A. |
| contents | Learning to remember over long timescales is fundamentally challenging for recurrent neural networks (RNNs). While much prior work has explored why RNNs struggle to learn long timescales and how to mitigate this, we still lack a clear understanding of the dynamics involved when RNNs learn long timescales via gradient descent. Here we build a mathematical theory of the learning dynamics of linear RNNs trained to integrate white noise. We show that when the initial recurrent weights are small, the dynamics of learning are described by a low-dimensional system that tracks a single outlier eigenvalue of the recurrent weights. This reveals the precise manner in which the long timescale associated with white noise integration is learned. We extend our analyses to RNNs learning a damped oscillatory filter, and find rich dynamical equations for the evolution of a conjugate pair of outlier eigenvalues. Taken together, our analyses build a rich mathematical framework for studying dynamical learning problems salient for both machine learning and neuroscience. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_18754 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Dynamically Learning to Integrate in Recurrent Neural Networks Bordelon, Blake Cotler, Jordan Pehlevan, Cengiz Zavatone-Veth, Jacob A. Neurons and Cognition Disordered Systems and Neural Networks Machine Learning Learning to remember over long timescales is fundamentally challenging for recurrent neural networks (RNNs). While much prior work has explored why RNNs struggle to learn long timescales and how to mitigate this, we still lack a clear understanding of the dynamics involved when RNNs learn long timescales via gradient descent. Here we build a mathematical theory of the learning dynamics of linear RNNs trained to integrate white noise. We show that when the initial recurrent weights are small, the dynamics of learning are described by a low-dimensional system that tracks a single outlier eigenvalue of the recurrent weights. This reveals the precise manner in which the long timescale associated with white noise integration is learned. We extend our analyses to RNNs learning a damped oscillatory filter, and find rich dynamical equations for the evolution of a conjugate pair of outlier eigenvalues. Taken together, our analyses build a rich mathematical framework for studying dynamical learning problems salient for both machine learning and neuroscience. |
| title | Dynamically Learning to Integrate in Recurrent Neural Networks |
| topic | Neurons and Cognition Disordered Systems and Neural Networks Machine Learning |
| url | https://arxiv.org/abs/2503.18754 |