Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2309.03852 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913646867644416 |
|---|---|
| author | Li, Xiang Yao, Yiqun Jiang, Xin Fang, Xuezhi Meng, Xuying Fan, Siqi Han, Peng Li, Jing Du, Li Qin, Bowen Zhang, Zheng Sun, Aixin Wang, Yequan |
| author_facet | Li, Xiang Yao, Yiqun Jiang, Xin Fang, Xuezhi Meng, Xuying Fan, Siqi Han, Peng Li, Jing Du, Li Qin, Bowen Zhang, Zheng Sun, Aixin Wang, Yequan |
| contents | Large language models (LLMs) are considered important approaches towards foundational machine intelligence, achieving remarkable success in Natural Language Processing and multimodal tasks, among others. However, the carbon footprints and financial costs originating from heavy pre-training computation is a non-negligible issue. Progressive training methods, inspired by the neurogenesis process that grows neural structures, have shown potential to accelerate LLM pre-training. However, the algorithms, implementation, and practices for progressively training LLMs beyond 100B parameters remain underexplored. In this paper, we show that our model, namely FLM-101B, trained with our growth strategy under a budget of \$100K, reaches 80\% of the baselines' performances with only 10\% of their floating-point operations. We believe that further studies on progressive training will benefit the community by cutting down the costs and promoting green AI. The checkpoint of FLM-101B is released at https://huggingface.co/CofeAI/FLM-101B. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_03852 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | FLM-101B: An Open LLM and How to Train It with $100K Budget Li, Xiang Yao, Yiqun Jiang, Xin Fang, Xuezhi Meng, Xuying Fan, Siqi Han, Peng Li, Jing Du, Li Qin, Bowen Zhang, Zheng Sun, Aixin Wang, Yequan Computation and Language Artificial Intelligence Large language models (LLMs) are considered important approaches towards foundational machine intelligence, achieving remarkable success in Natural Language Processing and multimodal tasks, among others. However, the carbon footprints and financial costs originating from heavy pre-training computation is a non-negligible issue. Progressive training methods, inspired by the neurogenesis process that grows neural structures, have shown potential to accelerate LLM pre-training. However, the algorithms, implementation, and practices for progressively training LLMs beyond 100B parameters remain underexplored. In this paper, we show that our model, namely FLM-101B, trained with our growth strategy under a budget of \$100K, reaches 80\% of the baselines' performances with only 10\% of their floating-point operations. We believe that further studies on progressive training will benefit the community by cutting down the costs and promoting green AI. The checkpoint of FLM-101B is released at https://huggingface.co/CofeAI/FLM-101B. |
| title | FLM-101B: An Open LLM and How to Train It with $100K Budget |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2309.03852 |