From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910856394047488 |
|---|---|
| author | Dominé, Clémentine C. J. Anguita, Nicolas Proca, Alexandra M. Braun, Lukas Kunin, Daniel Mediano, Pedro A. M. Saxe, Andrew M. |
| author_facet | Dominé, Clémentine C. J. Anguita, Nicolas Proca, Alexandra M. Braun, Lukas Kunin, Daniel Mediano, Pedro A. M. Saxe, Andrew M. |
| contents | Biological and artificial neural networks develop internal representations that enable them to perform complex tasks. In artificial networks, the effectiveness of these models relies on their ability to build task specific representation, a process influenced by interactions among datasets, architectures, initialization strategies, and optimization algorithms. Prior studies highlight that different initializations can place networks in either a lazy regime, where representations remain static, or a rich/feature learning regime, where representations evolve dynamically. Here, we examine how initialization influences learning dynamics in deep linear neural networks, deriving exact solutions for lambda-balanced initializations-defined by the relative scale of weights across layers. These solutions capture the evolution of representations and the Neural Tangent Kernel across the spectrum from the rich to the lazy regimes. Our findings deepen the theoretical understanding of the impact of weight initialization on learning regimes, with implications for continual learning, reversal learning, and transfer learning, relevant to both neuroscience and practical applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_14623 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks Dominé, Clémentine C. J. Anguita, Nicolas Proca, Alexandra M. Braun, Lukas Kunin, Daniel Mediano, Pedro A. M. Saxe, Andrew M. Machine Learning Biological and artificial neural networks develop internal representations that enable them to perform complex tasks. In artificial networks, the effectiveness of these models relies on their ability to build task specific representation, a process influenced by interactions among datasets, architectures, initialization strategies, and optimization algorithms. Prior studies highlight that different initializations can place networks in either a lazy regime, where representations remain static, or a rich/feature learning regime, where representations evolve dynamically. Here, we examine how initialization influences learning dynamics in deep linear neural networks, deriving exact solutions for lambda-balanced initializations-defined by the relative scale of weights across layers. These solutions capture the evolution of representations and the Neural Tangent Kernel across the spectrum from the rich to the lazy regimes. Our findings deepen the theoretical understanding of the impact of weight initialization on learning regimes, with implications for continual learning, reversal learning, and transfer learning, relevant to both neuroscience and practical applications. |
| title | From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2409.14623 |