Understanding and controlling the geometry of memory organization in RNNs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915145802842112 |
|---|---|
| author | Haputhanthri, Udith Storan, Liam Jiang, Yiqi Raheja, Tarun Shai, Adam Akengin, Orhun Miolane, Nina Schnitzer, Mark J. Dinc, Fatih Tanaka, Hidenori |
| author_facet | Haputhanthri, Udith Storan, Liam Jiang, Yiqi Raheja, Tarun Shai, Adam Akengin, Orhun Miolane, Nina Schnitzer, Mark J. Dinc, Fatih Tanaka, Hidenori |
| contents | Training recurrent neural networks (RNNs) is a high-dimensional process that requires updating numerous parameters. Therefore, it is often difficult to pinpoint the underlying learning mechanisms. To address this challenge, we propose to gain mechanistic insights into the phenomenon of \emph{abrupt learning} by studying RNNs trained to perform diverse short-term memory tasks. In these tasks, RNN training begins with an initial search phase. Following a long period of plateau in accuracy, the values of the loss function suddenly drop, indicating abrupt learning. Analyzing the neural computation performed by these RNNs reveals geometric restructuring (GR) in their phase spaces prior to the drop. To promote these GR events, we introduce a temporal consistency regularization that accelerates (bioplausible) training, facilitates attractor formation, and enables efficient learning in strongly connected networks. Our findings offer testable predictions for neuroscientists and emphasize the need for goal-agnostic secondary mechanisms to facilitate learning in biological and artificial networks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_07256 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Understanding and controlling the geometry of memory organization in RNNs Haputhanthri, Udith Storan, Liam Jiang, Yiqi Raheja, Tarun Shai, Adam Akengin, Orhun Miolane, Nina Schnitzer, Mark J. Dinc, Fatih Tanaka, Hidenori Neurons and Cognition Training recurrent neural networks (RNNs) is a high-dimensional process that requires updating numerous parameters. Therefore, it is often difficult to pinpoint the underlying learning mechanisms. To address this challenge, we propose to gain mechanistic insights into the phenomenon of \emph{abrupt learning} by studying RNNs trained to perform diverse short-term memory tasks. In these tasks, RNN training begins with an initial search phase. Following a long period of plateau in accuracy, the values of the loss function suddenly drop, indicating abrupt learning. Analyzing the neural computation performed by these RNNs reveals geometric restructuring (GR) in their phase spaces prior to the drop. To promote these GR events, we introduce a temporal consistency regularization that accelerates (bioplausible) training, facilitates attractor formation, and enables efficient learning in strongly connected networks. Our findings offer testable predictions for neuroscientists and emphasize the need for goal-agnostic secondary mechanisms to facilitate learning in biological and artificial networks. |
| title | Understanding and controlling the geometry of memory organization in RNNs |
| topic | Neurons and Cognition |
| url | https://arxiv.org/abs/2502.07256 |