Neural Networks Trained by Weight Permutation are Universal Approximators
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915205877858304 |
|---|---|
| author | Cai, Yongqiang Chen, Gaohang Qiao, Zhonghua |
| author_facet | Cai, Yongqiang Chen, Gaohang Qiao, Zhonghua |
| contents | The universal approximation property is fundamental to the success of neural networks, and has traditionally been achieved by training networks without any constraints on their parameters. However, recent experimental research proposed a novel permutation-based training method, which exhibited a desired classification performance without modifying the exact weight values. In this paper, we provide a theoretical guarantee of this permutation training method by proving its ability to guide a ReLU network to approximate one-dimensional continuous functions. Our numerical results further validate this method's efficiency in regression tasks with various initializations. The notable observations during weight permutation suggest that permutation training can provide an innovative tool for describing network learning behavior. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_01033 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Neural Networks Trained by Weight Permutation are Universal Approximators Cai, Yongqiang Chen, Gaohang Qiao, Zhonghua Machine Learning Neural and Evolutionary Computing 41A30, 68T05, 68T07 The universal approximation property is fundamental to the success of neural networks, and has traditionally been achieved by training networks without any constraints on their parameters. However, recent experimental research proposed a novel permutation-based training method, which exhibited a desired classification performance without modifying the exact weight values. In this paper, we provide a theoretical guarantee of this permutation training method by proving its ability to guide a ReLU network to approximate one-dimensional continuous functions. Our numerical results further validate this method's efficiency in regression tasks with various initializations. The notable observations during weight permutation suggest that permutation training can provide an innovative tool for describing network learning behavior. |
| title | Neural Networks Trained by Weight Permutation are Universal Approximators |
| topic | Machine Learning Neural and Evolutionary Computing 41A30, 68T05, 68T07 |
| url | https://arxiv.org/abs/2407.01033 |