Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2312.16903 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913959334903808 |
|---|---|
| author | Takase, Sho Kiyono, Shun Kobayashi, Sosuke Suzuki, Jun |
| author_facet | Takase, Sho Kiyono, Shun Kobayashi, Sosuke Suzuki, Jun |
| contents | Loss spikes often occur during pre-training of large language models. The spikes degrade the performance of large language models and sometimes ruin the pre-training. Since the pre-training needs a vast computational budget, we should avoid such spikes. Based on the assumption that the loss spike is caused by the sudden growth of the gradient norm, we explore factors to keep the gradient norm small through an analysis of the spectral norms of the Jacobian matrices for the sub-layers. Our findings suggest that stabilizing the pre-training process requires two conditions: small sub-layers and large shortcut. We conduct various experiments to empirically verify our theoretical analyses. Experimental results demonstrate that methods satisfying the conditions effectively prevent loss spikes during pre-training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_16903 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Spike No More: Stabilizing the Pre-training of Large Language Models Takase, Sho Kiyono, Shun Kobayashi, Sosuke Suzuki, Jun Computation and Language Artificial Intelligence Loss spikes often occur during pre-training of large language models. The spikes degrade the performance of large language models and sometimes ruin the pre-training. Since the pre-training needs a vast computational budget, we should avoid such spikes. Based on the assumption that the loss spike is caused by the sudden growth of the gradient norm, we explore factors to keep the gradient norm small through an analysis of the spectral norms of the Jacobian matrices for the sub-layers. Our findings suggest that stabilizing the pre-training process requires two conditions: small sub-layers and large shortcut. We conduct various experiments to empirically verify our theoretical analyses. Experimental results demonstrate that methods satisfying the conditions effectively prevent loss spikes during pre-training. |
| title | Spike No More: Stabilizing the Pre-training of Large Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2312.16903 |