Yi-Lightning Technical Report
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909463643947008 |
|---|---|
| author | Wake, Alan Chen, Bei Lv, C. X. Li, Chao Huang, Chengen Cai, Chenglin Zheng, Chujie Cooper, Daniel Zhou, Fan Hu, Feng Zhang, Ge Wang, Guoyin Ji, Heng Qiu, Howard Zhu, Jiangcheng Tian, Jun Su, Katherine Zhang, Lihuan Li, Liying Song, Ming Li, Mou Liu, Peng Hu, Qicheng Wang, Shawn Zhou, Shijun Yang, Shiming Li, Shiyong Zhu, Tianhang Xie, Wen Huang, Wenhao He, Xiang Chen, Xiaobo Hu, Xiaohui Ren, Xiaoyi Niu, Xinyao Li, Yanpeng Zhao, Yongke Luo, Yongzhen Xu, Yuchi Sha, Yuxuan Yan, Zhaodong Liu, Zhiyuan Zhang, Zirui Dai, Zonghong |
| author_facet | Wake, Alan Chen, Bei Lv, C. X. Li, Chao Huang, Chengen Cai, Chenglin Zheng, Chujie Cooper, Daniel Zhou, Fan Hu, Feng Zhang, Ge Wang, Guoyin Ji, Heng Qiu, Howard Zhu, Jiangcheng Tian, Jun Su, Katherine Zhang, Lihuan Li, Liying Song, Ming Li, Mou Liu, Peng Hu, Qicheng Wang, Shawn Zhou, Shijun Yang, Shiming Li, Shiyong Zhu, Tianhang Xie, Wen Huang, Wenhao He, Xiang Chen, Xiaobo Hu, Xiaohui Ren, Xiaoyi Niu, Xinyao Li, Yanpeng Zhao, Yongke Luo, Yongzhen Xu, Yuchi Sha, Yuxuan Yan, Zhaodong Liu, Zhiyuan Zhang, Zirui Dai, Zonghong |
| contents | This technical report presents Yi-Lightning, our latest flagship large language model (LLM). It achieves exceptional performance, ranking 6th overall on Chatbot Arena, with particularly strong results (2nd to 4th place) in specialized categories including Chinese, Math, Coding, and Hard Prompts. Yi-Lightning leverages an enhanced Mixture-of-Experts (MoE) architecture, featuring advanced expert segmentation and routing mechanisms coupled with optimized KV-caching techniques. Our development process encompasses comprehensive pre-training, supervised fine-tuning (SFT), and reinforcement learning from human feedback (RLHF), where we devise deliberate strategies for multi-stage training, synthetic data construction, and reward modeling. Furthermore, we implement RAISE (Responsible AI Safety Engine), a four-component framework to address safety issues across pre-training, post-training, and serving phases. Empowered by our scalable super-computing infrastructure, all these innovations substantially reduce training, deployment and inference costs while maintaining high-performance standards. With further evaluations on public academic benchmarks, Yi-Lightning demonstrates competitive performance against top-tier LLMs, while we observe a notable disparity between traditional, static benchmark results and real-world, dynamic human preferences. This observation prompts a critical reassessment of conventional benchmarks' utility in guiding the development of more intelligent and powerful AI systems for practical applications. Yi-Lightning is now available through our developer platform at https://platform.lingyiwanwu.com. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_01253 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Yi-Lightning Technical Report Wake, Alan Chen, Bei Lv, C. X. Li, Chao Huang, Chengen Cai, Chenglin Zheng, Chujie Cooper, Daniel Zhou, Fan Hu, Feng Zhang, Ge Wang, Guoyin Ji, Heng Qiu, Howard Zhu, Jiangcheng Tian, Jun Su, Katherine Zhang, Lihuan Li, Liying Song, Ming Li, Mou Liu, Peng Hu, Qicheng Wang, Shawn Zhou, Shijun Yang, Shiming Li, Shiyong Zhu, Tianhang Xie, Wen Huang, Wenhao He, Xiang Chen, Xiaobo Hu, Xiaohui Ren, Xiaoyi Niu, Xinyao Li, Yanpeng Zhao, Yongke Luo, Yongzhen Xu, Yuchi Sha, Yuxuan Yan, Zhaodong Liu, Zhiyuan Zhang, Zirui Dai, Zonghong Computation and Language Artificial Intelligence Machine Learning This technical report presents Yi-Lightning, our latest flagship large language model (LLM). It achieves exceptional performance, ranking 6th overall on Chatbot Arena, with particularly strong results (2nd to 4th place) in specialized categories including Chinese, Math, Coding, and Hard Prompts. Yi-Lightning leverages an enhanced Mixture-of-Experts (MoE) architecture, featuring advanced expert segmentation and routing mechanisms coupled with optimized KV-caching techniques. Our development process encompasses comprehensive pre-training, supervised fine-tuning (SFT), and reinforcement learning from human feedback (RLHF), where we devise deliberate strategies for multi-stage training, synthetic data construction, and reward modeling. Furthermore, we implement RAISE (Responsible AI Safety Engine), a four-component framework to address safety issues across pre-training, post-training, and serving phases. Empowered by our scalable super-computing infrastructure, all these innovations substantially reduce training, deployment and inference costs while maintaining high-performance standards. With further evaluations on public academic benchmarks, Yi-Lightning demonstrates competitive performance against top-tier LLMs, while we observe a notable disparity between traditional, static benchmark results and real-world, dynamic human preferences. This observation prompts a critical reassessment of conventional benchmarks' utility in guiding the development of more intelligent and powerful AI systems for practical applications. Yi-Lightning is now available through our developer platform at https://platform.lingyiwanwu.com. |
| title | Yi-Lightning Technical Report |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2412.01253 |