Why Gradients Rapidly Increase Near the End of Training
Fuente:
arXiv
Salvato in:
| Autore principale: | |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915335042498560 |
|---|---|
| author | Defazio, Aaron |
| author_facet | Defazio, Aaron |
| contents | During long-duration Large Language Model (LLM) training runs the gradient norm increases rapidly near the end of training. In this short note, we show that this increase is due to an unintended interaction between weight decay, normalization layers, and the learning rate schedule. We propose a simple correction that fixes this behavior while also resulting in lower loss values throughout training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_02285 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Why Gradients Rapidly Increase Near the End of Training Defazio, Aaron Machine Learning Artificial Intelligence During long-duration Large Language Model (LLM) training runs the gradient norm increases rapidly near the end of training. In this short note, we show that this increase is due to an unintended interaction between weight decay, normalization layers, and the learning rate schedule. We propose a simple correction that fixes this behavior while also resulting in lower loss values throughout training. |
| title | Why Gradients Rapidly Increase Near the End of Training |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2506.02285 |