Cautious Optimizers: Improving Training with One Line of Code
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917274948993024 |
|---|---|
| author | Liang, Kaizhao Chen, Lizhang Liu, Bo Liu, Qiang |
| author_facet | Liang, Kaizhao Chen, Lizhang Liu, Bo Liu, Qiang |
| contents | AdamW has been the default optimizer for transformer pretraining. For many years, our community searched for faster and more stable optimizers with only constrained positive outcomes. In this work, we propose a \textbf{one-line modification in Pytorch} to any momentum-based optimizer, which we rename cautious optimizer, e.g. C-AdamW and C-Lion. Our theoretical result shows that this modification preserves Adam's Hamiltonian function and it does not break the convergence guarantee under the Lyapunov analysis. In addition, a whole new family of optimizers is revealed by our theoretical insight. Among them, we pick the simplest one for empirical experiments, showing not only consistent speed-up on LLM pretraining, but also image classification, with minimum extra tuning on hyperparameters. Code is available at https://github.com/kyleliang919/C-Optim. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_16085 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Cautious Optimizers: Improving Training with One Line of Code Liang, Kaizhao Chen, Lizhang Liu, Bo Liu, Qiang Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Discrete Mathematics AdamW has been the default optimizer for transformer pretraining. For many years, our community searched for faster and more stable optimizers with only constrained positive outcomes. In this work, we propose a \textbf{one-line modification in Pytorch} to any momentum-based optimizer, which we rename cautious optimizer, e.g. C-AdamW and C-Lion. Our theoretical result shows that this modification preserves Adam's Hamiltonian function and it does not break the convergence guarantee under the Lyapunov analysis. In addition, a whole new family of optimizers is revealed by our theoretical insight. Among them, we pick the simplest one for empirical experiments, showing not only consistent speed-up on LLM pretraining, but also image classification, with minimum extra tuning on hyperparameters. Code is available at https://github.com/kyleliang919/C-Optim. |
| title | Cautious Optimizers: Improving Training with One Line of Code |
| topic | Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Discrete Mathematics |
| url | https://arxiv.org/abs/2411.16085 |