Adaptive Optimization for Enhanced Efficiency in Large-Scale Language Model Training
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912146643746816 |
|---|---|
| author | Chen, Jiajing Liu, Bingying Liao, Xiaoxuan Gao, Jia Zheng, Hongye Li, Yue |
| author_facet | Chen, Jiajing Liu, Bingying Liao, Xiaoxuan Gao, Jia Zheng, Hongye Li, Yue |
| contents | With the rapid development of natural language processing technology, large-scale language models (LLM) have achieved remarkable results in a variety of tasks. However, how to effectively train these huge models and improve their performance and computational efficiency remains an important challenge. This paper proposes an improved method based on adaptive optimization algorithm, aiming to improve the training efficiency and final performance of LLM. Through comparative experiments on the SQuAD and GLUE data sets, the experimental results show that compared with traditional optimization algorithms (such as SGD, Momentum, AdaGrad, RMSProp and Adam), the adaptive optimization algorithm we proposed has better accuracy and F1 score. Both have achieved significant improvements, especially showed stronger training capabilities when processed large-scale texts and complex tasks. The research results verify the advantages of adaptive optimization algorithms in large-scale language model training and provide new ideas and directions for future optimization methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_04718 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Adaptive Optimization for Enhanced Efficiency in Large-Scale Language Model Training Chen, Jiajing Liu, Bingying Liao, Xiaoxuan Gao, Jia Zheng, Hongye Li, Yue Artificial Intelligence With the rapid development of natural language processing technology, large-scale language models (LLM) have achieved remarkable results in a variety of tasks. However, how to effectively train these huge models and improve their performance and computational efficiency remains an important challenge. This paper proposes an improved method based on adaptive optimization algorithm, aiming to improve the training efficiency and final performance of LLM. Through comparative experiments on the SQuAD and GLUE data sets, the experimental results show that compared with traditional optimization algorithms (such as SGD, Momentum, AdaGrad, RMSProp and Adam), the adaptive optimization algorithm we proposed has better accuracy and F1 score. Both have achieved significant improvements, especially showed stronger training capabilities when processed large-scale texts and complex tasks. The research results verify the advantages of adaptive optimization algorithms in large-scale language model training and provide new ideas and directions for future optimization methods. |
| title | Adaptive Optimization for Enhanced Efficiency in Large-Scale Language Model Training |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2412.04718 |