Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917017146097664 |
|---|---|
| author | Milkert, Max Hyde, David Laine, Forrest |
| author_facet | Milkert, Max Hyde, David Laine, Forrest |
| contents | In a neural network with ReLU activations, the number of piecewise linear regions in the output can grow exponentially with depth. However, this is highly unlikely to happen when the initial parameters are sampled randomly, which therefore often leads to the use of networks that are unnecessarily large. To address this problem, we introduce a novel parameterization of the network that restricts its weights so that a depth $d$ network produces exactly $2^d$ linear regions at initialization and maintains those regions throughout training under the parameterization. This approach allows us to learn approximations of convex, one dimensional functions that are several orders of magnitude more accurate than their randomly initialized counterparts. We further demonstrate a preliminary extension of our construction to multidimensional and non-convex functions, allowing the technique to replace traditional dense layers in various architectures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_18022 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training Milkert, Max Hyde, David Laine, Forrest Machine Learning Artificial Intelligence In a neural network with ReLU activations, the number of piecewise linear regions in the output can grow exponentially with depth. However, this is highly unlikely to happen when the initial parameters are sampled randomly, which therefore often leads to the use of networks that are unnecessarily large. To address this problem, we introduce a novel parameterization of the network that restricts its weights so that a depth $d$ network produces exactly $2^d$ linear regions at initialization and maintains those regions throughout training under the parameterization. This approach allows us to learn approximations of convex, one dimensional functions that are several orders of magnitude more accurate than their randomly initialized counterparts. We further demonstrate a preliminary extension of our construction to multidimensional and non-convex functions, allowing the technique to replace traditional dense layers in various architectures. |
| title | Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2311.18022 |