The effect of Leaky ReLUs on the training and generalization of overparameterized networks
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909119704727552 |
|---|---|
| author | Guo, Yinglong Li, Shaohan Lerman, Gilad |
| author_facet | Guo, Yinglong Li, Shaohan Lerman, Gilad |
| contents | We investigate the training and generalization errors of overparameterized neural networks (NNs) with a wide class of leaky rectified linear unit (ReLU) functions. More specifically, we carefully upper bound both the convergence rate of the training error and the generalization error of such NNs and investigate the dependence of these bounds on the Leaky ReLU parameter, $α$. We show that $α=-1$, which corresponds to the absolute value activation function, is optimal for the training error bound. Furthermore, in special settings, it is also optimal for the generalization error bound. Numerical experiments empirically support the practical choices guided by the theory. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_11942 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The effect of Leaky ReLUs on the training and generalization of overparameterized networks Guo, Yinglong Li, Shaohan Lerman, Gilad Machine Learning We investigate the training and generalization errors of overparameterized neural networks (NNs) with a wide class of leaky rectified linear unit (ReLU) functions. More specifically, we carefully upper bound both the convergence rate of the training error and the generalization error of such NNs and investigate the dependence of these bounds on the Leaky ReLU parameter, $α$. We show that $α=-1$, which corresponds to the absolute value activation function, is optimal for the training error bound. Furthermore, in special settings, it is also optimal for the generalization error bound. Numerical experiments empirically support the practical choices guided by the theory. |
| title | The effect of Leaky ReLUs on the training and generalization of overparameterized networks |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2402.11942 |