Towards Lower Bounds on the Depth of ReLU Neural Networks
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2021
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929422894891008 |
|---|---|
| author | Hertrich, Christoph Basu, Amitabh Di Summa, Marco Skutella, Martin |
| author_facet | Hertrich, Christoph Basu, Amitabh Di Summa, Marco Skutella, Martin |
| contents | We contribute to a better understanding of the class of functions that can be represented by a neural network with ReLU activations and a given architecture. Using techniques from mixed-integer optimization, polyhedral theory, and tropical geometry, we provide a mathematical counterbalance to the universal approximation theorems which suggest that a single hidden layer is sufficient for learning any function. In particular, we investigate whether the class of exactly representable functions strictly increases by adding more layers (with no restrictions on size). As a by-product of our investigations, we settle an old conjecture about piecewise linear functions by Wang and Sun (2005) in the affirmative. We also present upper bounds on the sizes of neural networks required to represent functions with logarithmic depth. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2105_14835 |
| institution | arXiv |
| publishDate | 2021 |
| record_format | arxiv |
| spellingShingle | Towards Lower Bounds on the Depth of ReLU Neural Networks Hertrich, Christoph Basu, Amitabh Di Summa, Marco Skutella, Martin Machine Learning Discrete Mathematics Neural and Evolutionary Computing Combinatorics We contribute to a better understanding of the class of functions that can be represented by a neural network with ReLU activations and a given architecture. Using techniques from mixed-integer optimization, polyhedral theory, and tropical geometry, we provide a mathematical counterbalance to the universal approximation theorems which suggest that a single hidden layer is sufficient for learning any function. In particular, we investigate whether the class of exactly representable functions strictly increases by adding more layers (with no restrictions on size). As a by-product of our investigations, we settle an old conjecture about piecewise linear functions by Wang and Sun (2005) in the affirmative. We also present upper bounds on the sizes of neural networks required to represent functions with logarithmic depth. |
| title | Towards Lower Bounds on the Depth of ReLU Neural Networks |
| topic | Machine Learning Discrete Mathematics Neural and Evolutionary Computing Combinatorics |
| url | https://arxiv.org/abs/2105.14835 |