Temporal Contrastive Learning through implicit non-equilibrium memory
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912207851225088 |
|---|---|
| author | Falk, Martin J. Strupp, Adam T. Scellier, Benjamin Murugan, Arvind |
| author_facet | Falk, Martin J. Strupp, Adam T. Scellier, Benjamin Murugan, Arvind |
| contents | The backpropagation method has enabled transformative uses of neural networks. Alternatively, for energy-based models, local learning methods involving only nearby neurons offer benefits in terms of decentralized training, and allow for the possibility of learning in computationally-constrained substrates. One class of local learning methods constrasts the desired, clamped behavior with spontaneous, free behavior. However, directly contrasting free and clamped behaviors requires explicit memory. Here, we introduce `Temporal Contrastive Learning', an approach that uses integral feedback in each learning degree of freedom to provide a simple form of implicit non-equilibrium memory. During training, free and clamped behaviors are shown in a sawtooth-like protocol over time. When combined with integral feedback dynamics, these alternating temporal protocols generate an implicit memory necessary for comparing free and clamped behaviors, broadening the range of physical and biological systems capable of contrastive learning. Finally, we show that non-equilibrium dissipation improves learning quality and determine a Landauer-like energy cost of contrastive learning through physical dynamics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_17723 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Temporal Contrastive Learning through implicit non-equilibrium memory Falk, Martin J. Strupp, Adam T. Scellier, Benjamin Murugan, Arvind Disordered Systems and Neural Networks Soft Condensed Matter Statistical Mechanics Molecular Networks The backpropagation method has enabled transformative uses of neural networks. Alternatively, for energy-based models, local learning methods involving only nearby neurons offer benefits in terms of decentralized training, and allow for the possibility of learning in computationally-constrained substrates. One class of local learning methods constrasts the desired, clamped behavior with spontaneous, free behavior. However, directly contrasting free and clamped behaviors requires explicit memory. Here, we introduce `Temporal Contrastive Learning', an approach that uses integral feedback in each learning degree of freedom to provide a simple form of implicit non-equilibrium memory. During training, free and clamped behaviors are shown in a sawtooth-like protocol over time. When combined with integral feedback dynamics, these alternating temporal protocols generate an implicit memory necessary for comparing free and clamped behaviors, broadening the range of physical and biological systems capable of contrastive learning. Finally, we show that non-equilibrium dissipation improves learning quality and determine a Landauer-like energy cost of contrastive learning through physical dynamics. |
| title | Temporal Contrastive Learning through implicit non-equilibrium memory |
| topic | Disordered Systems and Neural Networks Soft Condensed Matter Statistical Mechanics Molecular Networks |
| url | https://arxiv.org/abs/2312.17723 |