Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2412.17743 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929646051786752 |
|---|---|
| author | Hu, Yiwen Song, Huatong Deng, Jia Wang, Jiapeng Chen, Jie Zhou, Kun Zhu, Yutao Jiang, Jinhao Dong, Zican Zhao, Wayne Xin Wen, Ji-Rong |
| author_facet | Hu, Yiwen Song, Huatong Deng, Jia Wang, Jiapeng Chen, Jie Zhou, Kun Zhu, Yutao Jiang, Jinhao Dong, Zican Zhao, Wayne Xin Wen, Ji-Rong |
| contents | Effective pre-training of large language models (LLMs) has been challenging due to the immense resource demands and the complexity of the technical processes involved. This paper presents a detailed technical report on YuLan-Mini, a highly capable base model with 2.42B parameters that achieves top-tier performance among models of similar parameter scale. Our pre-training approach focuses on enhancing training efficacy through three key technical contributions: an elaborate data pipeline combines data cleaning with data schedule strategies, a robust optimization method to mitigate training instability, and an effective annealing approach that incorporates targeted data selection and long context training. Remarkably, YuLan-Mini, trained on 1.08T tokens, achieves performance comparable to industry-leading models that require significantly more data. To facilitate reproduction, we release the full details of the data composition for each training phase. Project details can be accessed at the following link: https://github.com/RUC-GSAI/YuLan-Mini. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_17743 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | YuLan-Mini: An Open Data-efficient Language Model Hu, Yiwen Song, Huatong Deng, Jia Wang, Jiapeng Chen, Jie Zhou, Kun Zhu, Yutao Jiang, Jinhao Dong, Zican Zhao, Wayne Xin Wen, Ji-Rong Computation and Language Effective pre-training of large language models (LLMs) has been challenging due to the immense resource demands and the complexity of the technical processes involved. This paper presents a detailed technical report on YuLan-Mini, a highly capable base model with 2.42B parameters that achieves top-tier performance among models of similar parameter scale. Our pre-training approach focuses on enhancing training efficacy through three key technical contributions: an elaborate data pipeline combines data cleaning with data schedule strategies, a robust optimization method to mitigate training instability, and an effective annealing approach that incorporates targeted data selection and long context training. Remarkably, YuLan-Mini, trained on 1.08T tokens, achieves performance comparable to industry-leading models that require significantly more data. To facilitate reproduction, we release the full details of the data composition for each training phase. Project details can be accessed at the following link: https://github.com/RUC-GSAI/YuLan-Mini. |
| title | YuLan-Mini: An Open Data-efficient Language Model |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.17743 |