Global Convergence in Training Large-Scale Transformers
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912097176125440 |
|---|---|
| author | Gao, Cheng Cao, Yuan Li, Zihao He, Yihan Wang, Mengdi Liu, Han Klusowski, Jason Matthew Fan, Jianqing |
| author_facet | Gao, Cheng Cao, Yuan Li, Zihao He, Yihan Wang, Mengdi Liu, Han Klusowski, Jason Matthew Fan, Jianqing |
| contents | Despite the widespread success of Transformers across various domains, their optimization guarantees in large-scale model settings are not well-understood. This paper rigorously analyzes the convergence properties of gradient flow in training Transformers with weight decay regularization. First, we construct the mean-field limit of large-scale Transformers, showing that as the model width and depth go to infinity, gradient flow converges to the Wasserstein gradient flow, which is represented by a partial differential equation. Then, we demonstrate that the gradient flow reaches a global minimum consistent with the PDE solution when the weight decay regularization parameter is sufficiently small. Our analysis is based on a series of novel mean-field techniques that adapt to Transformers. Compared with existing tools for deep networks (Lu et al., 2020) that demand homogeneity and global Lipschitz smoothness, we utilize a refined analysis assuming only $\textit{partial homogeneity}$ and $\textit{local Lipschitz smoothness}$. These new techniques may be of independent interest. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_23610 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Global Convergence in Training Large-Scale Transformers Gao, Cheng Cao, Yuan Li, Zihao He, Yihan Wang, Mengdi Liu, Han Klusowski, Jason Matthew Fan, Jianqing Machine Learning Statistics Theory 35Q93 Despite the widespread success of Transformers across various domains, their optimization guarantees in large-scale model settings are not well-understood. This paper rigorously analyzes the convergence properties of gradient flow in training Transformers with weight decay regularization. First, we construct the mean-field limit of large-scale Transformers, showing that as the model width and depth go to infinity, gradient flow converges to the Wasserstein gradient flow, which is represented by a partial differential equation. Then, we demonstrate that the gradient flow reaches a global minimum consistent with the PDE solution when the weight decay regularization parameter is sufficiently small. Our analysis is based on a series of novel mean-field techniques that adapt to Transformers. Compared with existing tools for deep networks (Lu et al., 2020) that demand homogeneity and global Lipschitz smoothness, we utilize a refined analysis assuming only $\textit{partial homogeneity}$ and $\textit{local Lipschitz smoothness}$. These new techniques may be of independent interest. |
| title | Global Convergence in Training Large-Scale Transformers |
| topic | Machine Learning Statistics Theory 35Q93 |
| url | https://arxiv.org/abs/2410.23610 |