TeleChat Technical Report
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911822898003968 |
|---|---|
| author | He, Zhongjiang Wang, Zihan Liu, Xinzhang Liu, Shixuan Yao, Yitong Huang, Yuyao Li, Xuelong Li, Yongxiang Che, Zhonghao Zhang, Zhaoxi Wang, Yan Wang, Xin Pu, Luwen Xu, Huinan Fang, Ruiyu Zhao, Yu Zhang, Jie Huang, Xiaomeng Lu, Zhilong Peng, Jiaxin Zheng, Wenjun Wang, Shiquan Yang, Bingkai he, Xuewei Jiang, Zhuoru Xie, Qiyi Zhang, Yanhan Li, Zhongqiu Shi, Lingling Fu, Weiwei Zhang, Yin Huang, Zilu Xiong, Sishi Zhang, Yuxiang Wang, Chao Song, Shuangyong |
| author_facet | He, Zhongjiang Wang, Zihan Liu, Xinzhang Liu, Shixuan Yao, Yitong Huang, Yuyao Li, Xuelong Li, Yongxiang Che, Zhonghao Zhang, Zhaoxi Wang, Yan Wang, Xin Pu, Luwen Xu, Huinan Fang, Ruiyu Zhao, Yu Zhang, Jie Huang, Xiaomeng Lu, Zhilong Peng, Jiaxin Zheng, Wenjun Wang, Shiquan Yang, Bingkai he, Xuewei Jiang, Zhuoru Xie, Qiyi Zhang, Yanhan Li, Zhongqiu Shi, Lingling Fu, Weiwei Zhang, Yin Huang, Zilu Xiong, Sishi Zhang, Yuxiang Wang, Chao Song, Shuangyong |
| contents | In this technical report, we present TeleChat, a collection of large language models (LLMs) with parameters of 3 billion, 7 billion and 12 billion. It includes pretrained language models as well as fine-tuned chat models that is aligned with human preferences. TeleChat is initially pretrained on an extensive corpus containing a diverse collection of texts from both English and Chinese languages, including trillions of tokens. Subsequently, the model undergoes fine-tuning to align with human preferences, following a detailed methodology that we describe. We evaluate the performance of TeleChat on various tasks, including language understanding, mathematics, reasoning, code generation, and knowledge-based question answering. Our findings indicate that TeleChat achieves comparable performance to other open-source models of similar size across a wide range of public benchmarks. To support future research and applications utilizing LLMs, we release the fine-tuned model checkpoints of TeleChat's 7B and 12B variant, along with code and a portion of our pretraining data, to the public community. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_03804 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TeleChat Technical Report He, Zhongjiang Wang, Zihan Liu, Xinzhang Liu, Shixuan Yao, Yitong Huang, Yuyao Li, Xuelong Li, Yongxiang Che, Zhonghao Zhang, Zhaoxi Wang, Yan Wang, Xin Pu, Luwen Xu, Huinan Fang, Ruiyu Zhao, Yu Zhang, Jie Huang, Xiaomeng Lu, Zhilong Peng, Jiaxin Zheng, Wenjun Wang, Shiquan Yang, Bingkai he, Xuewei Jiang, Zhuoru Xie, Qiyi Zhang, Yanhan Li, Zhongqiu Shi, Lingling Fu, Weiwei Zhang, Yin Huang, Zilu Xiong, Sishi Zhang, Yuxiang Wang, Chao Song, Shuangyong Computation and Language Artificial Intelligence I.2.7 In this technical report, we present TeleChat, a collection of large language models (LLMs) with parameters of 3 billion, 7 billion and 12 billion. It includes pretrained language models as well as fine-tuned chat models that is aligned with human preferences. TeleChat is initially pretrained on an extensive corpus containing a diverse collection of texts from both English and Chinese languages, including trillions of tokens. Subsequently, the model undergoes fine-tuning to align with human preferences, following a detailed methodology that we describe. We evaluate the performance of TeleChat on various tasks, including language understanding, mathematics, reasoning, code generation, and knowledge-based question answering. Our findings indicate that TeleChat achieves comparable performance to other open-source models of similar size across a wide range of public benchmarks. To support future research and applications utilizing LLMs, we release the fine-tuned model checkpoints of TeleChat's 7B and 12B variant, along with code and a portion of our pretraining data, to the public community. |
| title | TeleChat Technical Report |
| topic | Computation and Language Artificial Intelligence I.2.7 |
| url | https://arxiv.org/abs/2401.03804 |