In almost all shallow analytic neural network optimization landscapes, efficient minimizers have strongly convex neighborhoods
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913789424697344 |
|---|---|
| author | Benning, Felix Dereich, Steffen |
| author_facet | Benning, Felix Dereich, Steffen |
| contents | Whether or not a local minimum of a cost function has a strongly convex neighborhood greatly influences the asymptotic convergence rate of optimizers. In this article, we rigorously analyze the prevalence of this property for the mean squared error induced by shallow, 1-hidden layer neural networks with analytic activation functions when applied to regression problems. The parameter space is divided into two domains: the 'efficient domain' (all parameters for which the respective realization function cannot be generated by a network having a smaller number of neurons) and the 'redundant domain' (the remaining parameters). In almost all regression problems on the efficient domain the optimization landscape only features local minima that are strongly convex. Formally, we will show that for certain randomly picked regression problems the optimization landscape is almost surely a Morse function on the efficient domain. The redundant domain has significantly smaller dimension than the efficient domain and on this domain, potential local minima are never isolated. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_08867 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | In almost all shallow analytic neural network optimization landscapes, efficient minimizers have strongly convex neighborhoods Benning, Felix Dereich, Steffen Machine Learning Probability 60G15, 60G60, 62J02, 62M45, 68T07 Whether or not a local minimum of a cost function has a strongly convex neighborhood greatly influences the asymptotic convergence rate of optimizers. In this article, we rigorously analyze the prevalence of this property for the mean squared error induced by shallow, 1-hidden layer neural networks with analytic activation functions when applied to regression problems. The parameter space is divided into two domains: the 'efficient domain' (all parameters for which the respective realization function cannot be generated by a network having a smaller number of neurons) and the 'redundant domain' (the remaining parameters). In almost all regression problems on the efficient domain the optimization landscape only features local minima that are strongly convex. Formally, we will show that for certain randomly picked regression problems the optimization landscape is almost surely a Morse function on the efficient domain. The redundant domain has significantly smaller dimension than the efficient domain and on this domain, potential local minima are never isolated. |
| title | In almost all shallow analytic neural network optimization landscapes, efficient minimizers have strongly convex neighborhoods |
| topic | Machine Learning Probability 60G15, 60G60, 62J02, 62M45, 68T07 |
| url | https://arxiv.org/abs/2504.08867 |