The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929347296755712 |
|---|---|
| author | Mao, Jialin Griniasty, Itay Teoh, Han Kheng Ramesh, Rahul Yang, Rubing Transtrum, Mark K. Sethna, James P. Chaudhari, Pratik |
| author_facet | Mao, Jialin Griniasty, Itay Teoh, Han Kheng Ramesh, Rahul Yang, Rubing Transtrum, Mark K. Sethna, James P. Chaudhari, Pratik |
| contents | We develop information-geometric techniques to analyze the trajectories of the predictions of deep networks during training. By examining the underlying high-dimensional probabilistic models, we reveal that the training process explores an effectively low-dimensional manifold. Networks with a wide range of architectures, sizes, trained using different optimization methods, regularization techniques, data augmentation techniques, and weight initializations lie on the same manifold in the prediction space. We study the details of this manifold to find that networks with different architectures follow distinguishable trajectories but other factors have a minimal influence; larger networks train along a similar manifold as that of smaller networks, just faster; and networks initialized at very different parts of the prediction space converge to the solution along a similar manifold. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_01604 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold Mao, Jialin Griniasty, Itay Teoh, Han Kheng Ramesh, Rahul Yang, Rubing Transtrum, Mark K. Sethna, James P. Chaudhari, Pratik Machine Learning Disordered Systems and Neural Networks We develop information-geometric techniques to analyze the trajectories of the predictions of deep networks during training. By examining the underlying high-dimensional probabilistic models, we reveal that the training process explores an effectively low-dimensional manifold. Networks with a wide range of architectures, sizes, trained using different optimization methods, regularization techniques, data augmentation techniques, and weight initializations lie on the same manifold in the prediction space. We study the details of this manifold to find that networks with different architectures follow distinguishable trajectories but other factors have a minimal influence; larger networks train along a similar manifold as that of smaller networks, just faster; and networks initialized at very different parts of the prediction space converge to the solution along a similar manifold. |
| title | The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold |
| topic | Machine Learning Disordered Systems and Neural Networks |
| url | https://arxiv.org/abs/2305.01604 |