Neural Scaling Laws From Large-N Field Theory: Solvable Model Beyond the Ridgeless Limit
Fuente:
arXiv
Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914816391643136 |
|---|---|
| author | Zhang, Zhengkang |
| author_facet | Zhang, Zhengkang |
| contents | Many machine learning models based on neural networks exhibit scaling laws: their performance scales as power laws with respect to the sizes of the model and training data set. We use large-N field theory methods to solve a model recently proposed by Maloney, Roberts and Sully which provides a simplified setting to study neural scaling laws. Our solution extends the result in this latter paper to general nonzero values of the ridge parameter, which are essential to regularize the behavior of the model. In addition to obtaining new and more precise scaling laws, we also uncover a duality transformation at the diagrams level which explains the symmetry between model and training data set sizes. The same duality underlies recent efforts to design neural networks to simulate quantum field theories. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_19398 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Neural Scaling Laws From Large-N Field Theory: Solvable Model Beyond the Ridgeless Limit Zhang, Zhengkang High Energy Physics - Theory Disordered Systems and Neural Networks Machine Learning High Energy Physics - Phenomenology Many machine learning models based on neural networks exhibit scaling laws: their performance scales as power laws with respect to the sizes of the model and training data set. We use large-N field theory methods to solve a model recently proposed by Maloney, Roberts and Sully which provides a simplified setting to study neural scaling laws. Our solution extends the result in this latter paper to general nonzero values of the ridge parameter, which are essential to regularize the behavior of the model. In addition to obtaining new and more precise scaling laws, we also uncover a duality transformation at the diagrams level which explains the symmetry between model and training data set sizes. The same duality underlies recent efforts to design neural networks to simulate quantum field theories. |
| title | Neural Scaling Laws From Large-N Field Theory: Solvable Model Beyond the Ridgeless Limit |
| topic | High Energy Physics - Theory Disordered Systems and Neural Networks Machine Learning High Energy Physics - Phenomenology |
| url | https://arxiv.org/abs/2405.19398 |