Towards Simple and Provable Parameter-Free Adaptive Gradient Methods
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913177144393728 |
|---|---|
| author | Tao, Yuanzhe Liu, Yifeng Yuan, Huizhuo Zhou, Xun Cao, Yuan Gu, Quanquan |
| author_facet | Tao, Yuanzhe Liu, Yifeng Yuan, Huizhuo Zhou, Xun Cao, Yuan Gu, Quanquan |
| contents | Optimization algorithms such as AdaGrad and Adam have significantly advanced the training of deep models by dynamically adjusting the learning rate during the optimization process. However, ad-hoc tuning of learning rates poses a challenge and leads to inefficiencies in practice. To address this issue, recent research has focused on developing ``parameter-free'' algorithms that operate effectively without the need for learning rate tuning. Despite these efforts, existing parameter-free variants of AdaGrad and Adam tend to be overly complex and/or lack formal convergence guarantees. In this paper, we present AdaGrad++ and Adam++, novel and simple parameter-free variants of AdaGrad and Adam with convergence guarantees. We prove that AdaGrad++ achieves comparable convergence rates to AdaGrad in convex optimization without predefined learning rate assumptions. Similarly, Adam++ matches the convergence rate of Adam without relying on any conditions on the learning rates. Experimental results across various deep learning tasks validate the competitive performance of Adam++. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_19444 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Simple and Provable Parameter-Free Adaptive Gradient Methods Tao, Yuanzhe Liu, Yifeng Yuan, Huizhuo Zhou, Xun Cao, Yuan Gu, Quanquan Machine Learning Optimization and Control Optimization algorithms such as AdaGrad and Adam have significantly advanced the training of deep models by dynamically adjusting the learning rate during the optimization process. However, ad-hoc tuning of learning rates poses a challenge and leads to inefficiencies in practice. To address this issue, recent research has focused on developing ``parameter-free'' algorithms that operate effectively without the need for learning rate tuning. Despite these efforts, existing parameter-free variants of AdaGrad and Adam tend to be overly complex and/or lack formal convergence guarantees. In this paper, we present AdaGrad++ and Adam++, novel and simple parameter-free variants of AdaGrad and Adam with convergence guarantees. We prove that AdaGrad++ achieves comparable convergence rates to AdaGrad in convex optimization without predefined learning rate assumptions. Similarly, Adam++ matches the convergence rate of Adam without relying on any conditions on the learning rates. Experimental results across various deep learning tasks validate the competitive performance of Adam++. |
| title | Towards Simple and Provable Parameter-Free Adaptive Gradient Methods |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2412.19444 |