Geometry of Critical Sets and Existence of Saddle Branches for Two-layer Neural Networks
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916263334248448 |
|---|---|
| author | Zhang, Leyang Zhang, Yaoyu Luo, Tao |
| author_facet | Zhang, Leyang Zhang, Yaoyu Luo, Tao |
| contents | This paper presents a comprehensive analysis of critical point sets in two-layer neural networks. To study such complex entities, we introduce the critical embedding operator and critical reduction operator as our tools. Given a critical point, we use these operators to uncover the whole underlying critical set representing the same output function, which exhibits a hierarchical structure. Furthermore, we prove existence of saddle branches for any critical set whose output function can be represented by a narrower network. Our results provide a solid foundation to the further study of optimization and training behavior of neural networks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_17501 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Geometry of Critical Sets and Existence of Saddle Branches for Two-layer Neural Networks Zhang, Leyang Zhang, Yaoyu Luo, Tao Machine Learning Optimization and Control This paper presents a comprehensive analysis of critical point sets in two-layer neural networks. To study such complex entities, we introduce the critical embedding operator and critical reduction operator as our tools. Given a critical point, we use these operators to uncover the whole underlying critical set representing the same output function, which exhibits a hierarchical structure. Furthermore, we prove existence of saddle branches for any critical set whose output function can be represented by a narrower network. Our results provide a solid foundation to the further study of optimization and training behavior of neural networks. |
| title | Geometry of Critical Sets and Existence of Saddle Branches for Two-layer Neural Networks |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2405.17501 |