Topological obstruction to the training of shallow ReLU neural networks
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909379642523648 |
|---|---|
| author | Nurisso, Marco Leroy, Pierrick Vaccarino, Francesco |
| author_facet | Nurisso, Marco Leroy, Pierrick Vaccarino, Francesco |
| contents | Studying the interplay between the geometry of the loss landscape and the optimization trajectories of simple neural networks is a fundamental step for understanding their behavior in more complex settings. This paper reveals the presence of topological obstruction in the loss landscape of shallow ReLU neural networks trained using gradient flow. We discuss how the homogeneous nature of the ReLU activation function constrains the training trajectories to lie on a product of quadric hypersurfaces whose shape depends on the particular initialization of the network's parameters. When the neural network's output is a single scalar, we prove that these quadrics can have multiple connected components, limiting the set of reachable parameters during training. We analytically compute the number of these components and discuss the possibility of mapping one to the other through neuron rescaling and permutation. In this simple setting, we find that the non-connectedness results in a topological obstruction, which, depending on the initialization, can make the global optimum unreachable. We validate this result with numerical experiments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_14837 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Topological obstruction to the training of shallow ReLU neural networks Nurisso, Marco Leroy, Pierrick Vaccarino, Francesco Machine Learning Algebraic Geometry Algebraic Topology 68T07 (Primary) 57N65, 14R05 (Secondary) I.2.6 Studying the interplay between the geometry of the loss landscape and the optimization trajectories of simple neural networks is a fundamental step for understanding their behavior in more complex settings. This paper reveals the presence of topological obstruction in the loss landscape of shallow ReLU neural networks trained using gradient flow. We discuss how the homogeneous nature of the ReLU activation function constrains the training trajectories to lie on a product of quadric hypersurfaces whose shape depends on the particular initialization of the network's parameters. When the neural network's output is a single scalar, we prove that these quadrics can have multiple connected components, limiting the set of reachable parameters during training. We analytically compute the number of these components and discuss the possibility of mapping one to the other through neuron rescaling and permutation. In this simple setting, we find that the non-connectedness results in a topological obstruction, which, depending on the initialization, can make the global optimum unreachable. We validate this result with numerical experiments. |
| title | Topological obstruction to the training of shallow ReLU neural networks |
| topic | Machine Learning Algebraic Geometry Algebraic Topology 68T07 (Primary) 57N65, 14R05 (Secondary) I.2.6 |
| url | https://arxiv.org/abs/2410.14837 |