Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912089123061760 |
|---|---|
| author | Li, Shuaipeng Zhao, Penghao Zhang, Hailin Sun, Xingwu Wu, Hao Jiao, Dian Wang, Weiyan Liu, Chengjun Fang, Zheng Xue, Jinbao Tao, Yangyu Cui, Bin Wang, Di |
| author_facet | Li, Shuaipeng Zhao, Penghao Zhang, Hailin Sun, Xingwu Wu, Hao Jiao, Dian Wang, Weiyan Liu, Chengjun Fang, Zheng Xue, Jinbao Tao, Yangyu Cui, Bin Wang, Di |
| contents | In current deep learning tasks, Adam style optimizers such as Adam, Adagrad, RMSProp, Adafactor, and Lion have been widely used as alternatives to SGD style optimizers. These optimizers typically update model parameters using the sign of gradients, resulting in more stable convergence curves. The learning rate and the batch size are the most critical hyperparameters for optimizers, which require careful tuning to enable effective convergence. Previous research has shown that the optimal learning rate increases linearly or follows similar rules with batch size for SGD style optimizers. However, this conclusion is not applicable to Adam style optimizers. In this paper, we elucidate the connection between optimal learning rates and batch sizes for Adam style optimizers through both theoretical analysis and extensive experiments. First, we raise the scaling law between batch sizes and optimal learning rates in the sign of gradient case, in which we prove that the optimal learning rate first rises and then falls as the batch size increases. Moreover, the peak value of the surge will gradually move toward the larger batch size as training progresses. Second, we conducted experiments on various CV and NLP tasks and verified the correctness of the scaling law. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_14578 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling Li, Shuaipeng Zhao, Penghao Zhang, Hailin Sun, Xingwu Wu, Hao Jiao, Dian Wang, Weiyan Liu, Chengjun Fang, Zheng Xue, Jinbao Tao, Yangyu Cui, Bin Wang, Di Machine Learning In current deep learning tasks, Adam style optimizers such as Adam, Adagrad, RMSProp, Adafactor, and Lion have been widely used as alternatives to SGD style optimizers. These optimizers typically update model parameters using the sign of gradients, resulting in more stable convergence curves. The learning rate and the batch size are the most critical hyperparameters for optimizers, which require careful tuning to enable effective convergence. Previous research has shown that the optimal learning rate increases linearly or follows similar rules with batch size for SGD style optimizers. However, this conclusion is not applicable to Adam style optimizers. In this paper, we elucidate the connection between optimal learning rates and batch sizes for Adam style optimizers through both theoretical analysis and extensive experiments. First, we raise the scaling law between batch sizes and optimal learning rates in the sign of gradient case, in which we prove that the optimal learning rate first rises and then falls as the batch size increases. Moreover, the peak value of the surge will gradually move toward the larger batch size as training progresses. Second, we conducted experiments on various CV and NLP tasks and verified the correctness of the scaling law. |
| title | Surge Phenomenon in Optimal Learning Rate and Batch Size Scaling |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2405.14578 |