Preparing Lessons for Progressive Training on Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913230536835072 |
|---|---|
| author | Pan, Yu Yuan, Ye Yin, Yichun Shi, Jiaxin Xu, Zenglin Zhang, Ming Shang, Lifeng Jiang, Xin Liu, Qun |
| author_facet | Pan, Yu Yuan, Ye Yin, Yichun Shi, Jiaxin Xu, Zenglin Zhang, Ming Shang, Lifeng Jiang, Xin Liu, Qun |
| contents | The rapid progress of Transformers in artificial intelligence has come at the cost of increased resource consumption and greenhouse gas emissions due to growing model sizes. Prior work suggests using pretrained small models to improve training efficiency, but this approach may not be suitable for new model structures. On the other hand, training from scratch can be slow, and progressively stacking layers often fails to achieve significant acceleration. To address these challenges, we propose a novel method called Apollo, which prep\textbf{a}res lessons for ex\textbf{p}anding \textbf{o}perations by \textbf{l}earning high-\textbf{l}ayer functi\textbf{o}nality during training of low layers. Our approach involves low-value-prioritized sampling (LVPS) to train different depths and weight sharing to facilitate efficient expansion. We also introduce an interpolation method for stable model depth extension. Experiments demonstrate that Apollo achieves state-of-the-art acceleration ratios, even rivaling methods using pretrained models, making it a universal and efficient solution for training deep models while reducing time, financial, and environmental costs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_09192 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Preparing Lessons for Progressive Training on Language Models Pan, Yu Yuan, Ye Yin, Yichun Shi, Jiaxin Xu, Zenglin Zhang, Ming Shang, Lifeng Jiang, Xin Liu, Qun Machine Learning Artificial Intelligence The rapid progress of Transformers in artificial intelligence has come at the cost of increased resource consumption and greenhouse gas emissions due to growing model sizes. Prior work suggests using pretrained small models to improve training efficiency, but this approach may not be suitable for new model structures. On the other hand, training from scratch can be slow, and progressively stacking layers often fails to achieve significant acceleration. To address these challenges, we propose a novel method called Apollo, which prep\textbf{a}res lessons for ex\textbf{p}anding \textbf{o}perations by \textbf{l}earning high-\textbf{l}ayer functi\textbf{o}nality during training of low layers. Our approach involves low-value-prioritized sampling (LVPS) to train different depths and weight sharing to facilitate efficient expansion. We also introduce an interpolation method for stable model depth extension. Experiments demonstrate that Apollo achieves state-of-the-art acceleration ratios, even rivaling methods using pretrained models, making it a universal and efficient solution for training deep models while reducing time, financial, and environmental costs. |
| title | Preparing Lessons for Progressive Training on Language Models |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2401.09192 |