Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2021
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2102.00853 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912086003548160 |
|---|---|
| author | Kulikovskikh, Ilona Legović, Tarzan |
| author_facet | Kulikovskikh, Ilona Legović, Tarzan |
| contents | Convergence and generalization are two crucial aspects of performance in neural networks. When analyzed separately, these properties may lead to contradictory results. Optimizing a convergence rate yields fast training, but does not guarantee the best generalization error. To avoid the conflict, recent studies suggest adopting a moderately large step size for optimizers, but the added value on the performance remains unclear. We propose the LIGHT function with the four configurations which regulate explicitly an improvement in convergence and generalization on testing. This contribution allows to: 1) improve both convergence and generalization of neural networks with no need to guarantee their stability; 2) build more reliable and explainable network architectures with no need for overparameterization. We refer to it as "painless" step size adaptation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2102_00853 |
| institution | arXiv |
| publishDate | 2021 |
| record_format | arxiv |
| spellingShingle | Painless step size adaptation for SGD Kulikovskikh, Ilona Legović, Tarzan Machine Learning Neural and Evolutionary Computing Convergence and generalization are two crucial aspects of performance in neural networks. When analyzed separately, these properties may lead to contradictory results. Optimizing a convergence rate yields fast training, but does not guarantee the best generalization error. To avoid the conflict, recent studies suggest adopting a moderately large step size for optimizers, but the added value on the performance remains unclear. We propose the LIGHT function with the four configurations which regulate explicitly an improvement in convergence and generalization on testing. This contribution allows to: 1) improve both convergence and generalization of neural networks with no need to guarantee their stability; 2) build more reliable and explainable network architectures with no need for overparameterization. We refer to it as "painless" step size adaptation. |
| title | Painless step size adaptation for SGD |
| topic | Machine Learning Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2102.00853 |