Activation degree thresholds and expressiveness of polynomial neural networks
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912618819616768 |
|---|---|
| author | Finkel, Bella Rodriguez, Jose Israel Wu, Chenxi Yahl, Thomas |
| author_facet | Finkel, Bella Rodriguez, Jose Israel Wu, Chenxi Yahl, Thomas |
| contents | We study the expressive power of deep polynomial neural networks through the geometry of their neurovariety. We introduce the notion of the activation degree threshold of a network architecture to express when the dimension of the neurovariety achieves its theoretical maximum. We prove the existence of the activation degree threshold for all polynomial neural networks without width-one bottlenecks and demonstrate a universal upper bound that is quadratic in the width of largest size. In doing so, we prove the high activation degree conjecture of Kileel, Trager, and Bruna. Certain structured architectures have exceptional activation degree thresholds, making them especially expressive in the sense of their neurovariety dimension. In this direction, we prove that polynomial neural networks with equi-width architectures are maximally expressive by showing their activation degree threshold is one. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_04569 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Activation degree thresholds and expressiveness of polynomial neural networks Finkel, Bella Rodriguez, Jose Israel Wu, Chenxi Yahl, Thomas Machine Learning Neural and Evolutionary Computing Algebraic Geometry We study the expressive power of deep polynomial neural networks through the geometry of their neurovariety. We introduce the notion of the activation degree threshold of a network architecture to express when the dimension of the neurovariety achieves its theoretical maximum. We prove the existence of the activation degree threshold for all polynomial neural networks without width-one bottlenecks and demonstrate a universal upper bound that is quadratic in the width of largest size. In doing so, we prove the high activation degree conjecture of Kileel, Trager, and Bruna. Certain structured architectures have exceptional activation degree thresholds, making them especially expressive in the sense of their neurovariety dimension. In this direction, we prove that polynomial neural networks with equi-width architectures are maximally expressive by showing their activation degree threshold is one. |
| title | Activation degree thresholds and expressiveness of polynomial neural networks |
| topic | Machine Learning Neural and Evolutionary Computing Algebraic Geometry |
| url | https://arxiv.org/abs/2408.04569 |