Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularities
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912866500608000 |
|---|---|
| author | Nurisso, Marco Leroy, Pierrick Petri, Giovanni Vaccarino, Francesco |
| author_facet | Nurisso, Marco Leroy, Pierrick Petri, Giovanni Vaccarino, Francesco |
| contents | Understanding the properties of the parameter space in feed-forward ReLU networks is critical for effectively analyzing and guiding training dynamics. After initialization, training under gradient flow decisively restricts the parameter space to an algebraic variety that emerges from the homogeneous nature of the ReLU activation function. In this study, we examine two key challenges associated with feed-forward ReLU networks built on general directed acyclic graph (DAG) architectures: the (dis)connectedness of the parameter space and the existence of singularities within it. We extend previous results by providing a thorough characterization of connectedness, highlighting the roles of bottleneck nodes and balance conditions associated with specific subsets of the network. Our findings clearly demonstrate that singularities are intricately connected to the topology of the underlying DAG and its induced sub-networks. We discuss the reachability of these singularities and establish a principled connection with differentiable pruning. We validate our theory with simple numerical experiments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_00693 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularities Nurisso, Marco Leroy, Pierrick Petri, Giovanni Vaccarino, Francesco Machine Learning Algebraic Geometry Algebraic Topology 68T07 (Primary) 57N65, 14R05 (Secondary) I.2.6 Understanding the properties of the parameter space in feed-forward ReLU networks is critical for effectively analyzing and guiding training dynamics. After initialization, training under gradient flow decisively restricts the parameter space to an algebraic variety that emerges from the homogeneous nature of the ReLU activation function. In this study, we examine two key challenges associated with feed-forward ReLU networks built on general directed acyclic graph (DAG) architectures: the (dis)connectedness of the parameter space and the existence of singularities within it. We extend previous results by providing a thorough characterization of connectedness, highlighting the roles of bottleneck nodes and balance conditions associated with specific subsets of the network. Our findings clearly demonstrate that singularities are intricately connected to the topology of the underlying DAG and its induced sub-networks. We discuss the reachability of these singularities and establish a principled connection with differentiable pruning. We validate our theory with simple numerical experiments. |
| title | Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularities |
| topic | Machine Learning Algebraic Geometry Algebraic Topology 68T07 (Primary) 57N65, 14R05 (Secondary) I.2.6 |
| url | https://arxiv.org/abs/2602.00693 |