Interpreting Adaptive Gradient Methods by Parameter Scaling for Learning-Rate-Free Optimization
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914632108605440 |
|---|---|
| author | Suh, Min-Kook Seo, Seung-Woo |
| author_facet | Suh, Min-Kook Seo, Seung-Woo |
| contents | We address the challenge of estimating the learning rate for adaptive gradient methods used in training deep neural networks. While several learning-rate-free approaches have been proposed, they are typically tailored for steepest descent. However, although steepest descent methods offer an intuitive approach to finding minima, many deep learning applications require adaptive gradient methods to achieve faster convergence. In this paper, we interpret adaptive gradient methods as steepest descent applied on parameter-scaled networks, proposing learning-rate-free adaptive gradient methods. Experimental results verify the effectiveness of this approach, demonstrating comparable performance to hand-tuned learning rates across various scenarios. This work extends the applicability of learning-rate-free methods, enhancing training with adaptive gradient methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_03240 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Interpreting Adaptive Gradient Methods by Parameter Scaling for Learning-Rate-Free Optimization Suh, Min-Kook Seo, Seung-Woo Machine Learning Optimization and Control We address the challenge of estimating the learning rate for adaptive gradient methods used in training deep neural networks. While several learning-rate-free approaches have been proposed, they are typically tailored for steepest descent. However, although steepest descent methods offer an intuitive approach to finding minima, many deep learning applications require adaptive gradient methods to achieve faster convergence. In this paper, we interpret adaptive gradient methods as steepest descent applied on parameter-scaled networks, proposing learning-rate-free adaptive gradient methods. Experimental results verify the effectiveness of this approach, demonstrating comparable performance to hand-tuned learning rates across various scenarios. This work extends the applicability of learning-rate-free methods, enhancing training with adaptive gradient methods. |
| title | Interpreting Adaptive Gradient Methods by Parameter Scaling for Learning-Rate-Free Optimization |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2401.03240 |