Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911773199695872 |
|---|---|
| author | Karhadkar, Kedar Murray, Michael Tseran, Hanna Montúfar, Guido |
| author_facet | Karhadkar, Kedar Murray, Michael Tseran, Hanna Montúfar, Guido |
| contents | We study the loss landscape of both shallow and deep, mildly overparameterized ReLU neural networks on a generic finite input dataset for the squared error loss. We show both by count and volume that most activation patterns correspond to parameter regions with no bad local minima. Furthermore, for one-dimensional input data, we show most activation regions realizable by the network contain a high dimensional set of global minima and no bad local minima. We experimentally confirm these results by finding a phase transition from most regions having full rank Jacobian to many regions having deficient rank depending on the amount of overparameterization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_19510 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape Karhadkar, Kedar Murray, Michael Tseran, Hanna Montúfar, Guido Machine Learning Combinatorics We study the loss landscape of both shallow and deep, mildly overparameterized ReLU neural networks on a generic finite input dataset for the squared error loss. We show both by count and volume that most activation patterns correspond to parameter regions with no bad local minima. Furthermore, for one-dimensional input data, we show most activation regions realizable by the network contain a high dimensional set of global minima and no bad local minima. We experimentally confirm these results by finding a phase transition from most regions having full rank Jacobian to many regions having deficient rank depending on the amount of overparameterization. |
| title | Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape |
| topic | Machine Learning Combinatorics |
| url | https://arxiv.org/abs/2305.19510 |