Function Space and Critical Points of Linear Convolutional Networks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911765047017472 |
|---|---|
| author | Kohn, Kathlén Montúfar, Guido Shahverdi, Vahid Trager, Matthew |
| author_facet | Kohn, Kathlén Montúfar, Guido Shahverdi, Vahid Trager, Matthew |
| contents | We study the geometry of linear networks with one-dimensional convolutional layers. The function spaces of these networks can be identified with semi-algebraic families of polynomials admitting sparse factorizations. We analyze the impact of the network's architecture on the function space's dimension, boundary, and singular points. We also describe the critical points of the network's parameterization map. Furthermore, we study the optimization problem of training a network with the squared error loss. We prove that for architectures where all strides are larger than one and generic data, the non-zero critical points of that optimization problem are smooth interior points of the function space. This property is known to be false for dense linear networks and linear convolutional networks with stride one. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2304_05752 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Function Space and Critical Points of Linear Convolutional Networks Kohn, Kathlén Montúfar, Guido Shahverdi, Vahid Trager, Matthew Machine Learning Algebraic Geometry 68T07, 14B05, 14E99, 14J99, 14N05, 14P10, 90C23 We study the geometry of linear networks with one-dimensional convolutional layers. The function spaces of these networks can be identified with semi-algebraic families of polynomials admitting sparse factorizations. We analyze the impact of the network's architecture on the function space's dimension, boundary, and singular points. We also describe the critical points of the network's parameterization map. Furthermore, we study the optimization problem of training a network with the squared error loss. We prove that for architectures where all strides are larger than one and generic data, the non-zero critical points of that optimization problem are smooth interior points of the function space. This property is known to be false for dense linear networks and linear convolutional networks with stride one. |
| title | Function Space and Critical Points of Linear Convolutional Networks |
| topic | Machine Learning Algebraic Geometry 68T07, 14B05, 14E99, 14J99, 14N05, 14P10, 90C23 |
| url | https://arxiv.org/abs/2304.05752 |