On the Principles of ReLU Networks with One Hidden Layer
Fuente:
arXiv
Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912475085012992 |
|---|---|
| author | Huang, Changcun |
| author_facet | Huang, Changcun |
| contents | A neural network with one hidden layer or a two-layer network (regardless of the input layer) is the simplest feedforward neural network, whose mechanism may be the basis of more general network architectures. However, even to this type of simple architecture, it is also a ``black box''; that is, it remains unclear how to interpret the mechanism of its solutions obtained by the back-propagation algorithm and how to control the training process through a deterministic way. This paper systematically studies the first problem by constructing universal function-approximation solutions. It is shown that, both theoretically and experimentally, the training solution for the one-dimensional input could be completely understood, and that for a higher-dimensional input can also be well interpreted to some extent. Those results pave the way for thoroughly revealing the black box of two-layer ReLU networks and advance the understanding of deep ReLU networks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_06728 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On the Principles of ReLU Networks with One Hidden Layer Huang, Changcun Machine Learning Artificial Intelligence Neural and Evolutionary Computing 68T07(Primary), 41A15(Secondary) I.2.6; G.1.2 A neural network with one hidden layer or a two-layer network (regardless of the input layer) is the simplest feedforward neural network, whose mechanism may be the basis of more general network architectures. However, even to this type of simple architecture, it is also a ``black box''; that is, it remains unclear how to interpret the mechanism of its solutions obtained by the back-propagation algorithm and how to control the training process through a deterministic way. This paper systematically studies the first problem by constructing universal function-approximation solutions. It is shown that, both theoretically and experimentally, the training solution for the one-dimensional input could be completely understood, and that for a higher-dimensional input can also be well interpreted to some extent. Those results pave the way for thoroughly revealing the black box of two-layer ReLU networks and advance the understanding of deep ReLU networks. |
| title | On the Principles of ReLU Networks with One Hidden Layer |
| topic | Machine Learning Artificial Intelligence Neural and Evolutionary Computing 68T07(Primary), 41A15(Secondary) I.2.6; G.1.2 |
| url | https://arxiv.org/abs/2411.06728 |