Stable Language Model Pre-training by Reducing Embedding Variability
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916390680657920 |
|---|---|
| author | Chung, Woojin Hong, Jiwoo An, Na Min Thorne, James Yun, Se-Young |
| author_facet | Chung, Woojin Hong, Jiwoo An, Na Min Thorne, James Yun, Se-Young |
| contents | Stable pre-training is essential for achieving better-performing language models. However, tracking pre-training stability by calculating gradient variance at every step is impractical due to the significant computational costs. We explore Token Embedding Variability (TEV) as a simple and efficient proxy for assessing pre-training stability in language models with pre-layer normalization, given that shallower layers are more prone to gradient explosion (section 2.2). Moreover, we propose Multi-head Low-Rank Attention (MLRA) as an architecture to alleviate such instability by limiting the exponential growth of output embedding variance, thereby preventing the gradient explosion (section 3.2). Empirical results on GPT-2 with MLRA demonstrate increased stability and lower perplexity, particularly in deeper models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_07787 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Stable Language Model Pre-training by Reducing Embedding Variability Chung, Woojin Hong, Jiwoo An, Na Min Thorne, James Yun, Se-Young Computation and Language Stable pre-training is essential for achieving better-performing language models. However, tracking pre-training stability by calculating gradient variance at every step is impractical due to the significant computational costs. We explore Token Embedding Variability (TEV) as a simple and efficient proxy for assessing pre-training stability in language models with pre-layer normalization, given that shallower layers are more prone to gradient explosion (section 2.2). Moreover, we propose Multi-head Low-Rank Attention (MLRA) as an architecture to alleviate such instability by limiting the exponential growth of output embedding variance, thereby preventing the gradient explosion (section 3.2). Empirical results on GPT-2 with MLRA demonstrate increased stability and lower perplexity, particularly in deeper models. |
| title | Stable Language Model Pre-training by Reducing Embedding Variability |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2409.07787 |