COIG-P: A High-Quality and Large-Scale Chinese Preference Dataset for Alignment with Human Values
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915232671072256 |
|---|---|
| author | P Team Wu, Siwei Ren, Jincheng Du, Xinrun Guo, Shuyue Qu, Xingwei Liang, Yiming Liu, Jie Li, Yunwen Zheng, Tianyu Feng, Boyu Yuan, Huaqing Wang, Zenith Liu, Jiaheng Huang, Wenhao Cai, Chenglin Que, Haoran Yang, Jian Bai, Yuelin Wang, Zekun Moore Yu, Zhouliang Lin, Qunshu Pan, Ding Jiang, Yuchen Wang, Tiannan Zhou, Wangchunshu Wang, Shenzhi Bu, Xingyuan Liu, Minghao Wang, Guoyin Zhang, Ge Lin, Chenghua |
| author_facet | P Team Wu, Siwei Ren, Jincheng Du, Xinrun Guo, Shuyue Qu, Xingwei Liang, Yiming Liu, Jie Li, Yunwen Zheng, Tianyu Feng, Boyu Yuan, Huaqing Wang, Zenith Liu, Jiaheng Huang, Wenhao Cai, Chenglin Que, Haoran Yang, Jian Bai, Yuelin Wang, Zekun Moore Yu, Zhouliang Lin, Qunshu Pan, Ding Jiang, Yuchen Wang, Tiannan Zhou, Wangchunshu Wang, Shenzhi Bu, Xingyuan Liu, Minghao Wang, Guoyin Zhang, Ge Lin, Chenghua |
| contents | Aligning large language models (LLMs) with human preferences has achieved remarkable success. However, existing Chinese preference datasets are limited by small scale, narrow domain coverage, and lack of rigorous data validation. Additionally, the reliance on human annotators for instruction and response labeling significantly constrains the scalability of human preference datasets. To address these challenges, we design an LLM-based Chinese preference dataset annotation pipeline with no human intervention. Specifically, we crawled and carefully filtered 92k high-quality Chinese queries and employed 15 mainstream LLMs to generate and score chosen-rejected response pairs. Based on it, we introduce COIG-P (Chinese Open Instruction Generalist - Preference), a high-quality, large-scale Chinese preference dataset, comprises 1,009k Chinese preference pairs spanning 6 diverse domains: Chat, Code, Math, Logic, Novel, and Role. Building upon COIG-P, to reduce the overhead of using LLMs for scoring, we trained a 8B-sized Chinese Reward Model (CRM) and meticulously constructed a Chinese Reward Benchmark (CRBench). Evaluation results based on AlignBench \citep{liu2024alignbenchbenchmarkingchinesealignment} show that that COIG-P significantly outperforms other Chinese preference datasets, and it brings significant performance improvements ranging from 2% to 12% for the Qwen2/2.5 and Infinity-Instruct-3M-0625 model series, respectively. The results on CRBench demonstrate that our CRM has a strong and robust scoring ability. We apply it to filter chosen-rejected response pairs in a test split of COIG-P, and our experiments show that it is comparable to GPT-4o in identifying low-quality samples while maintaining efficiency and cost-effectiveness. Our codes and data are released in https://github.com/multimodal-art-projection/COIG-P. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_05535 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | COIG-P: A High-Quality and Large-Scale Chinese Preference Dataset for Alignment with Human Values P Team Wu, Siwei Ren, Jincheng Du, Xinrun Guo, Shuyue Qu, Xingwei Liang, Yiming Liu, Jie Li, Yunwen Zheng, Tianyu Feng, Boyu Yuan, Huaqing Wang, Zenith Liu, Jiaheng Huang, Wenhao Cai, Chenglin Que, Haoran Yang, Jian Bai, Yuelin Wang, Zekun Moore Yu, Zhouliang Lin, Qunshu Pan, Ding Jiang, Yuchen Wang, Tiannan Zhou, Wangchunshu Wang, Shenzhi Bu, Xingyuan Liu, Minghao Wang, Guoyin Zhang, Ge Lin, Chenghua Computation and Language Aligning large language models (LLMs) with human preferences has achieved remarkable success. However, existing Chinese preference datasets are limited by small scale, narrow domain coverage, and lack of rigorous data validation. Additionally, the reliance on human annotators for instruction and response labeling significantly constrains the scalability of human preference datasets. To address these challenges, we design an LLM-based Chinese preference dataset annotation pipeline with no human intervention. Specifically, we crawled and carefully filtered 92k high-quality Chinese queries and employed 15 mainstream LLMs to generate and score chosen-rejected response pairs. Based on it, we introduce COIG-P (Chinese Open Instruction Generalist - Preference), a high-quality, large-scale Chinese preference dataset, comprises 1,009k Chinese preference pairs spanning 6 diverse domains: Chat, Code, Math, Logic, Novel, and Role. Building upon COIG-P, to reduce the overhead of using LLMs for scoring, we trained a 8B-sized Chinese Reward Model (CRM) and meticulously constructed a Chinese Reward Benchmark (CRBench). Evaluation results based on AlignBench \citep{liu2024alignbenchbenchmarkingchinesealignment} show that that COIG-P significantly outperforms other Chinese preference datasets, and it brings significant performance improvements ranging from 2% to 12% for the Qwen2/2.5 and Infinity-Instruct-3M-0625 model series, respectively. The results on CRBench demonstrate that our CRM has a strong and robust scoring ability. We apply it to filter chosen-rejected response pairs in a test split of COIG-P, and our experiments show that it is comparable to GPT-4o in identifying low-quality samples while maintaining efficiency and cost-effectiveness. Our codes and data are released in https://github.com/multimodal-art-projection/COIG-P. |
| title | COIG-P: A High-Quality and Large-Scale Chinese Preference Dataset for Alignment with Human Values |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.05535 |