Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2507.08472 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918100847296512 |
|---|---|
| author | Schlotthauer, Joel Kroos, Christian Hinze, Chris Hangya, Viktor Hahn, Luzian Küch, Fabian |
| author_facet | Schlotthauer, Joel Kroos, Christian Hinze, Chris Hangya, Viktor Hahn, Luzian Küch, Fabian |
| contents | Optimizers play a decisive role in reducing pre-training times for LLMs and achieving better-performing models. In this study, we compare three major variants: the de-facto standard AdamW, the simpler Lion, developed through an evolutionary search, and the second-order optimizer Sophia. For better generalization, we train with two different base architectures and use a single- and a multiple-epoch approach while keeping the number of tokens constant. Using the Maximal Update Parametrization and smaller proxy models, we tune relevant hyperparameters separately for each combination of base architecture and optimizer. We found that while the results from all three optimizers were in approximately the same range, Sophia exhibited the lowest training and validation loss, Lion was fastest in terms of training GPU hours but AdamW led to the best downstream evaluation results. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_08472 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Pre-Training LLMs on a budget: A comparison of three optimizers Schlotthauer, Joel Kroos, Christian Hinze, Chris Hangya, Viktor Hahn, Luzian Küch, Fabian Machine Learning Artificial Intelligence Optimizers play a decisive role in reducing pre-training times for LLMs and achieving better-performing models. In this study, we compare three major variants: the de-facto standard AdamW, the simpler Lion, developed through an evolutionary search, and the second-order optimizer Sophia. For better generalization, we train with two different base architectures and use a single- and a multiple-epoch approach while keeping the number of tokens constant. Using the Maximal Update Parametrization and smaller proxy models, we tune relevant hyperparameters separately for each combination of base architecture and optimizer. We found that while the results from all three optimizers were in approximately the same range, Sophia exhibited the lowest training and validation loss, Lion was fastest in terms of training GPU hours but AdamW led to the best downstream evaluation results. |
| title | Pre-Training LLMs on a budget: A comparison of three optimizers |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2507.08472 |