Geometry and Optimization of Shallow Polynomial Networks
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914204820176896 |
|---|---|
| author | Arjevani, Yossi Bruna, Joan Kileel, Joe Polak, Elzbieta Trager, Matthew |
| author_facet | Arjevani, Yossi Bruna, Joan Kileel, Joe Polak, Elzbieta Trager, Matthew |
| contents | We study shallow neural networks with monomial activations and output dimension one. The function space for these models can be identified with a set of symmetric tensors with bounded rank. We describe general features of these networks, focusing on the relationship between width and optimization. We then consider teacher-student problems, which can be viewed as problems of low-rank tensor approximation with respect to non-standard inner products that are induced by the data distribution. In this setting, we introduce a teacher-metric data discriminant which encodes the qualitative behavior of the optimization as a function of the training data distribution. Finally, we focus on networks with quadratic activations, presenting an in-depth analysis of the optimization landscape. In particular, we present a variation of the Eckart-Young Theorem characterizing all critical points and their Hessian signatures for teacher-student problems with quadratic networks and Gaussian training data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_06074 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Geometry and Optimization of Shallow Polynomial Networks Arjevani, Yossi Bruna, Joan Kileel, Joe Polak, Elzbieta Trager, Matthew Machine Learning Algebraic Geometry 68T07, 14P10, 90C23, 62R01 We study shallow neural networks with monomial activations and output dimension one. The function space for these models can be identified with a set of symmetric tensors with bounded rank. We describe general features of these networks, focusing on the relationship between width and optimization. We then consider teacher-student problems, which can be viewed as problems of low-rank tensor approximation with respect to non-standard inner products that are induced by the data distribution. In this setting, we introduce a teacher-metric data discriminant which encodes the qualitative behavior of the optimization as a function of the training data distribution. Finally, we focus on networks with quadratic activations, presenting an in-depth analysis of the optimization landscape. In particular, we present a variation of the Eckart-Young Theorem characterizing all critical points and their Hessian signatures for teacher-student problems with quadratic networks and Gaussian training data. |
| title | Geometry and Optimization of Shallow Polynomial Networks |
| topic | Machine Learning Algebraic Geometry 68T07, 14P10, 90C23, 62R01 |
| url | https://arxiv.org/abs/2501.06074 |