Faster Convergence for Transformer Fine-tuning with Line Search Methods
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917624113266688 |
|---|---|
| author | Kenneweg, Philip Galli, Leonardo Kenneweg, Tristan Hammer, Barbara |
| author_facet | Kenneweg, Philip Galli, Leonardo Kenneweg, Tristan Hammer, Barbara |
| contents | Recent works have shown that line search methods greatly increase performance of traditional stochastic gradient descent methods on a variety of datasets and architectures [1], [2]. In this work we succeed in extending line search methods to the novel and highly popular Transformer architecture and dataset domains in natural language processing. More specifically, we combine the Armijo line search with the Adam optimizer and extend it by subdividing the networks architecture into sensible units and perform the line search separately on these local units. Our optimization method outperforms the traditional Adam optimizer and achieves significant performance improvements for small data sets or small training budgets, while performing equal or better for other tested cases. Our work is publicly available as a python package, which provides a hyperparameter-free pytorch optimizer that is compatible with arbitrary network architectures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_18506 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Faster Convergence for Transformer Fine-tuning with Line Search Methods Kenneweg, Philip Galli, Leonardo Kenneweg, Tristan Hammer, Barbara Machine Learning Artificial Intelligence Recent works have shown that line search methods greatly increase performance of traditional stochastic gradient descent methods on a variety of datasets and architectures [1], [2]. In this work we succeed in extending line search methods to the novel and highly popular Transformer architecture and dataset domains in natural language processing. More specifically, we combine the Armijo line search with the Adam optimizer and extend it by subdividing the networks architecture into sensible units and perform the line search separately on these local units. Our optimization method outperforms the traditional Adam optimizer and achieves significant performance improvements for small data sets or small training budgets, while performing equal or better for other tested cases. Our work is publicly available as a python package, which provides a hyperparameter-free pytorch optimizer that is compatible with arbitrary network architectures. |
| title | Faster Convergence for Transformer Fine-tuning with Line Search Methods |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2403.18506 |