Scaling Exponents Across Parameterizations and Optimizers
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913433241255936 |
|---|---|
| author | Everett, Katie Xiao, Lechao Wortsman, Mitchell Alemi, Alexander A. Novak, Roman Liu, Peter J. Gur, Izzeddin Sohl-Dickstein, Jascha Kaelbling, Leslie Pack Lee, Jaehoon Pennington, Jeffrey |
| author_facet | Everett, Katie Xiao, Lechao Wortsman, Mitchell Alemi, Alexander A. Novak, Roman Liu, Peter J. Gur, Izzeddin Sohl-Dickstein, Jascha Kaelbling, Leslie Pack Lee, Jaehoon Pennington, Jeffrey |
| contents | Robust and effective scaling of models from small to large width typically requires the precise adjustment of many algorithmic and architectural details, such as parameterization and optimizer choices. In this work, we propose a new perspective on parameterization by investigating a key assumption in prior work about the alignment between parameters and data and derive new theoretical results under weaker assumptions and a broader set of optimizers. Our extensive empirical investigation includes tens of thousands of models trained with all combinations of three optimizers, four parameterizations, several alignment assumptions, more than a dozen learning rates, and fourteen model sizes up to 26.8B parameters. We find that the best learning rate scaling prescription would often have been excluded by the assumptions in prior work. Our results show that all parameterizations, not just maximal update parameterization (muP), can achieve hyperparameter transfer; moreover, our novel per-layer learning rate prescription for standard parameterization outperforms muP. Finally, we demonstrate that an overlooked aspect of parameterization, the epsilon parameter in Adam, must be scaled correctly to avoid gradient underflow and propose Adam-atan2, a new numerically stable, scale-invariant version of Adam that eliminates the epsilon hyperparameter entirely. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_05872 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Scaling Exponents Across Parameterizations and Optimizers Everett, Katie Xiao, Lechao Wortsman, Mitchell Alemi, Alexander A. Novak, Roman Liu, Peter J. Gur, Izzeddin Sohl-Dickstein, Jascha Kaelbling, Leslie Pack Lee, Jaehoon Pennington, Jeffrey Machine Learning Robust and effective scaling of models from small to large width typically requires the precise adjustment of many algorithmic and architectural details, such as parameterization and optimizer choices. In this work, we propose a new perspective on parameterization by investigating a key assumption in prior work about the alignment between parameters and data and derive new theoretical results under weaker assumptions and a broader set of optimizers. Our extensive empirical investigation includes tens of thousands of models trained with all combinations of three optimizers, four parameterizations, several alignment assumptions, more than a dozen learning rates, and fourteen model sizes up to 26.8B parameters. We find that the best learning rate scaling prescription would often have been excluded by the assumptions in prior work. Our results show that all parameterizations, not just maximal update parameterization (muP), can achieve hyperparameter transfer; moreover, our novel per-layer learning rate prescription for standard parameterization outperforms muP. Finally, we demonstrate that an overlooked aspect of parameterization, the epsilon parameter in Adam, must be scaled correctly to avoid gradient underflow and propose Adam-atan2, a new numerically stable, scale-invariant version of Adam that eliminates the epsilon hyperparameter entirely. |
| title | Scaling Exponents Across Parameterizations and Optimizers |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2407.05872 |