Phase codes emerge in recurrent neural networks optimized for modular arithmetic
Fuente:
arXiv
Guardado en:
| Autor principal: | |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912736634470400 |
|---|---|
| author | Murray, Keith T. |
| author_facet | Murray, Keith T. |
| contents | Recurrent neural networks (RNNs) can implement complex computations by leveraging a range of dynamics, such as oscillations, attractors, and transient trajectories. A growing body of work has highlighted the emergence of phase codes, a type of oscillatory activity where information is encoded in the relative phase of network activity, in RNNs trained for working memory tasks. However, these studies rely on architectural constraints or regularization schemes that explicitly promote oscillatory solutions. Here, we investigate whether phase coding can emerge purely from task optimization by training continuous-time RNNs to perform a simple modular arithmetic task without oscillatory-promoting biases. We find that in the absence of such biases, RNNs can learn phase code solutions. Surprisingly, we also uncover a rich diversity of alternative solutions that solve our modular arithmetic task via qualitatively distinct dynamics and dynamical mechanisms. We map the solution space for our task and show that the phase code solution occupies a distinct region. These results suggest that phase coding can be a natural but not inevitable outcome of training RNNs on modular arithmetic, and highlight the diversity of solutions RNNs can learn to solve simple tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_07908 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Phase codes emerge in recurrent neural networks optimized for modular arithmetic Murray, Keith T. Neurons and Cognition Artificial Intelligence Neural and Evolutionary Computing Recurrent neural networks (RNNs) can implement complex computations by leveraging a range of dynamics, such as oscillations, attractors, and transient trajectories. A growing body of work has highlighted the emergence of phase codes, a type of oscillatory activity where information is encoded in the relative phase of network activity, in RNNs trained for working memory tasks. However, these studies rely on architectural constraints or regularization schemes that explicitly promote oscillatory solutions. Here, we investigate whether phase coding can emerge purely from task optimization by training continuous-time RNNs to perform a simple modular arithmetic task without oscillatory-promoting biases. We find that in the absence of such biases, RNNs can learn phase code solutions. Surprisingly, we also uncover a rich diversity of alternative solutions that solve our modular arithmetic task via qualitatively distinct dynamics and dynamical mechanisms. We map the solution space for our task and show that the phase code solution occupies a distinct region. These results suggest that phase coding can be a natural but not inevitable outcome of training RNNs on modular arithmetic, and highlight the diversity of solutions RNNs can learn to solve simple tasks. |
| title | Phase codes emerge in recurrent neural networks optimized for modular arithmetic |
| topic | Neurons and Cognition Artificial Intelligence Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2310.07908 |