COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910681042780160 |
|---|---|
| author | Bai, Yuelin Du, Xinrun Liang, Yiming Jin, Yonggang Zhou, Junting Liu, Ziqiang Fang, Feiteng Chang, Mingshan Zheng, Tianyu Zhang, Xincheng Ma, Nuo Wang, Zekun Yuan, Ruibin Wu, Haihong Lin, Hongquan Huang, Wenhao Zhang, Jiajun Lin, Chenghua Fu, Jie Yang, Min Ni, Shiwen Zhang, Ge |
| author_facet | Bai, Yuelin Du, Xinrun Liang, Yiming Jin, Yonggang Zhou, Junting Liu, Ziqiang Fang, Feiteng Chang, Mingshan Zheng, Tianyu Zhang, Xincheng Ma, Nuo Wang, Zekun Yuan, Ruibin Wu, Haihong Lin, Hongquan Huang, Wenhao Zhang, Jiajun Lin, Chenghua Fu, Jie Yang, Min Ni, Shiwen Zhang, Ge |
| contents | Remarkable progress on English instruction tuning has facilitated the efficacy and reliability of large language models (LLMs). However, there remains a noticeable gap in instruction tuning for Chinese, where the complex linguistic features pose significant challenges. Existing datasets, generally distilled from English-centric LLMs, are not well-aligned with Chinese users' interaction patterns. To bridge this gap, we introduce COIG-CQIA, a new Chinese instruction tuning dataset derived from various real-world resources and undergoing rigorous human verification. We conduct extensive experiments on COIG-CQIA, and compare them with strong baseline models and datasets. The experimental results show that models trained on COIG-CQIA achieve highly competitive performance in diverse benchmarks. Additionally, our findings offer several insights for designing effective Chinese instruction-tuning datasets and data-mixing strategies. Our dataset are available at https://huggingface.co/datasets/m-a-p/COIG-CQIA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_18058 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning Bai, Yuelin Du, Xinrun Liang, Yiming Jin, Yonggang Zhou, Junting Liu, Ziqiang Fang, Feiteng Chang, Mingshan Zheng, Tianyu Zhang, Xincheng Ma, Nuo Wang, Zekun Yuan, Ruibin Wu, Haihong Lin, Hongquan Huang, Wenhao Zhang, Jiajun Lin, Chenghua Fu, Jie Yang, Min Ni, Shiwen Zhang, Ge Computation and Language Artificial Intelligence Remarkable progress on English instruction tuning has facilitated the efficacy and reliability of large language models (LLMs). However, there remains a noticeable gap in instruction tuning for Chinese, where the complex linguistic features pose significant challenges. Existing datasets, generally distilled from English-centric LLMs, are not well-aligned with Chinese users' interaction patterns. To bridge this gap, we introduce COIG-CQIA, a new Chinese instruction tuning dataset derived from various real-world resources and undergoing rigorous human verification. We conduct extensive experiments on COIG-CQIA, and compare them with strong baseline models and datasets. The experimental results show that models trained on COIG-CQIA achieve highly competitive performance in diverse benchmarks. Additionally, our findings offer several insights for designing effective Chinese instruction-tuning datasets and data-mixing strategies. Our dataset are available at https://huggingface.co/datasets/m-a-p/COIG-CQIA. |
| title | COIG-CQIA: Quality is All You Need for Chinese Instruction Fine-tuning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2403.18058 |