Orthogonal Transforms in Neural Networks Amount to Effective Regularization
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913662033199104 |
|---|---|
| author | Zając, Krzysztof Sopot, Wojciech Wachel, Paweł |
| author_facet | Zając, Krzysztof Sopot, Wojciech Wachel, Paweł |
| contents | We consider applications of neural networks in nonlinear system identification and formulate a hypothesis that adjusting general network structure by incorporating frequency information or other known orthogonal transform, should result in an efficient neural network retaining its universal properties. We show that such a structure is a universal approximator and that using any orthogonal transform in a proposed way implies regularization during training by adjusting the learning rate of each parameter individually. We empirically show in particular, that such a structure, using the Fourier transform, outperforms equivalent models without orthogonality support. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_06344 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Orthogonal Transforms in Neural Networks Amount to Effective Regularization Zając, Krzysztof Sopot, Wojciech Wachel, Paweł Machine Learning Neural and Evolutionary Computing Systems and Control We consider applications of neural networks in nonlinear system identification and formulate a hypothesis that adjusting general network structure by incorporating frequency information or other known orthogonal transform, should result in an efficient neural network retaining its universal properties. We show that such a structure is a universal approximator and that using any orthogonal transform in a proposed way implies regularization during training by adjusting the learning rate of each parameter individually. We empirically show in particular, that such a structure, using the Fourier transform, outperforms equivalent models without orthogonality support. |
| title | Orthogonal Transforms in Neural Networks Amount to Effective Regularization |
| topic | Machine Learning Neural and Evolutionary Computing Systems and Control |
| url | https://arxiv.org/abs/2305.06344 |