Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2302.14690 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915025367597056 |
|---|---|
| author | Dereich, Steffen Jentzen, Arnulf Kassing, Sebastian |
| author_facet | Dereich, Steffen Jentzen, Arnulf Kassing, Sebastian |
| contents | In this article, we show existence of minimizers in the loss landscape for residual artificial neural networks (ANNs) with multi-dimensional input layer and one hidden layer with ReLU activation. Our work contrasts earlier results in [D. Gallon, A. Jentzen, and F. Lindner, preprint, arXiv:2211.15641, 2022] and [P. Petersen, M. Raslan, and F. Voigtlaender, Found. Comput. Math., 21 (2021), pp. 375-444] which showed that in many situations minimizers do not exist for common smooth activation functions even in the case where the target functions are polynomials. The proof of the existence property makes use of a closure of the search space containing all functions generated by ANNs and additional discontinuous generalized responses. As we will show, the additional generalized responses in this larger space are suboptimal so that the minimum is attained in the original function class. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2302_14690 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | On the existence of minimizers in shallow residual ReLU neural network optimization landscapes Dereich, Steffen Jentzen, Arnulf Kassing, Sebastian Optimization and Control Machine Learning Numerical Analysis Primary 68T07, Secondary 68T05, 41A50 In this article, we show existence of minimizers in the loss landscape for residual artificial neural networks (ANNs) with multi-dimensional input layer and one hidden layer with ReLU activation. Our work contrasts earlier results in [D. Gallon, A. Jentzen, and F. Lindner, preprint, arXiv:2211.15641, 2022] and [P. Petersen, M. Raslan, and F. Voigtlaender, Found. Comput. Math., 21 (2021), pp. 375-444] which showed that in many situations minimizers do not exist for common smooth activation functions even in the case where the target functions are polynomials. The proof of the existence property makes use of a closure of the search space containing all functions generated by ANNs and additional discontinuous generalized responses. As we will show, the additional generalized responses in this larger space are suboptimal so that the minimum is attained in the original function class. |
| title | On the existence of minimizers in shallow residual ReLU neural network optimization landscapes |
| topic | Optimization and Control Machine Learning Numerical Analysis Primary 68T07, Secondary 68T05, 41A50 |
| url | https://arxiv.org/abs/2302.14690 |