Orion-14B: Open-source Multilingual Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916103096107008 |
|---|---|
| author | Chen, Du Huang, Yi Li, Xiaopu Li, Yongqiang Liu, Yongqiang Pan, Haihui Xu, Leichao Zhang, Dacheng Zhang, Zhipeng Han, Kun |
| author_facet | Chen, Du Huang, Yi Li, Xiaopu Li, Yongqiang Liu, Yongqiang Pan, Haihui Xu, Leichao Zhang, Dacheng Zhang, Zhipeng Han, Kun |
| contents | In this study, we introduce Orion-14B, a collection of multilingual large language models with 14 billion parameters. We utilize a data scheduling approach to train a foundational model on a diverse corpus of 2.5 trillion tokens, sourced from texts in English, Chinese, Japanese, Korean, and other languages. Additionally, we fine-tuned a series of models tailored for conversational applications and other specific use cases. Our evaluation results demonstrate that Orion-14B achieves state-of-the-art performance across a broad spectrum of tasks. We make the Orion-14B model family and its associated code publicly accessible https://github.com/OrionStarAI/Orion, aiming to inspire future research and practical applications in the field. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_12246 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Orion-14B: Open-source Multilingual Large Language Models Chen, Du Huang, Yi Li, Xiaopu Li, Yongqiang Liu, Yongqiang Pan, Haihui Xu, Leichao Zhang, Dacheng Zhang, Zhipeng Han, Kun Computation and Language Machine Learning In this study, we introduce Orion-14B, a collection of multilingual large language models with 14 billion parameters. We utilize a data scheduling approach to train a foundational model on a diverse corpus of 2.5 trillion tokens, sourced from texts in English, Chinese, Japanese, Korean, and other languages. Additionally, we fine-tuned a series of models tailored for conversational applications and other specific use cases. Our evaluation results demonstrate that Orion-14B achieves state-of-the-art performance across a broad spectrum of tasks. We make the Orion-14B model family and its associated code publicly accessible https://github.com/OrionStarAI/Orion, aiming to inspire future research and practical applications in the field. |
| title | Orion-14B: Open-source Multilingual Large Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2401.12246 |