Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building a Chinese-Centric LLM
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916611527540736 |
|---|---|
| author | Gu, Qingshui Li, Shu Zheng, Tianyu Zhang, Zhaoxiang |
| author_facet | Gu, Qingshui Li, Shu Zheng, Tianyu Zhang, Zhaoxiang |
| contents | Steel-LLM is a Chinese-centric language model developed from scratch with the goal of creating a high-quality, open-source model despite limited computational resources. Launched in March 2024, the project aimed to train a 1-billion-parameter model on a large-scale dataset, prioritizing transparency and the sharing of practical insights to assist others in the community. The training process primarily focused on Chinese data, with a small proportion of English data included, addressing gaps in existing open-source LLMs by providing a more detailed and practical account of the model-building journey. Steel-LLM has demonstrated competitive performance on benchmarks such as CEVAL and CMMLU, outperforming early models from larger institutions. This paper provides a comprehensive summary of the project's key contributions, including data collection, model design, training methodologies, and the challenges encountered along the way, offering a valuable resource for researchers and practitioners looking to develop their own LLMs. The model checkpoints and training script are available at https://github.com/zhanshijinwat/Steel-LLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_06635 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building a Chinese-Centric LLM Gu, Qingshui Li, Shu Zheng, Tianyu Zhang, Zhaoxiang Computation and Language Artificial Intelligence Steel-LLM is a Chinese-centric language model developed from scratch with the goal of creating a high-quality, open-source model despite limited computational resources. Launched in March 2024, the project aimed to train a 1-billion-parameter model on a large-scale dataset, prioritizing transparency and the sharing of practical insights to assist others in the community. The training process primarily focused on Chinese data, with a small proportion of English data included, addressing gaps in existing open-source LLMs by providing a more detailed and practical account of the model-building journey. Steel-LLM has demonstrated competitive performance on benchmarks such as CEVAL and CMMLU, outperforming early models from larger institutions. This paper provides a comprehensive summary of the project's key contributions, including data collection, model design, training methodologies, and the challenges encountered along the way, offering a valuable resource for researchers and practitioners looking to develop their own LLMs. The model checkpoints and training script are available at https://github.com/zhanshijinwat/Steel-LLM. |
| title | Steel-LLM:From Scratch to Open Source -- A Personal Journey in Building a Chinese-Centric LLM |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.06635 |