Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.03524 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918046034034688 |
|---|---|
| author | Seed, ByteDance Zhang, Yuyu Su, Jing Sun, Yifan Xi, Chenguang Xiao, Xia Zheng, Shen Zhang, Anxiang Liu, Kaibo Zan, Daoguang Sun, Tao Zhu, Jinhua Xin, Shulin Huang, Dong Bai, Yetao Dong, Lixin Li, Chao Chen, Jianchong Zhou, Hanzhi Huang, Yifan Ning, Guanghan Song, Xierui Chen, Jiaze Liu, Siyao Shen, Kai Xiang, Liang Wu, Yonghui |
| author_facet | Seed, ByteDance Zhang, Yuyu Su, Jing Sun, Yifan Xi, Chenguang Xiao, Xia Zheng, Shen Zhang, Anxiang Liu, Kaibo Zan, Daoguang Sun, Tao Zhu, Jinhua Xin, Shulin Huang, Dong Bai, Yetao Dong, Lixin Li, Chao Chen, Jianchong Zhou, Hanzhi Huang, Yifan Ning, Guanghan Song, Xierui Chen, Jiaze Liu, Siyao Shen, Kai Xiang, Liang Wu, Yonghui |
| contents | Code data in large language model (LLM) pretraining is recognized crucial not only for code-related tasks but also for enhancing general intelligence of LLMs. Current open-source LLMs often heavily rely on human effort to produce their code pretraining data, such as employing hand-crafted filtering rules tailored to individual programming languages, or using human-annotated data to train quality filters. However, these approaches are inherently limited in scalability, prone to subjective biases, and costly to extend and maintain across diverse programming languages. To address these challenges, we introduce Seed-Coder, a series of open-source LLMs comprising base, instruct and reasoning models of 8B size, minimizing human involvement in data construction. Our code pretraining data is produced by a model-centric data pipeline, which predominantly leverages LLMs for scoring and filtering code data. The instruct model is further trained via supervised fine-tuning and preference optimization, and the reasoning model leverages Long-Chain-of-Thought (LongCoT) reinforcement learning to improve multi-step code reasoning. Seed-Coder achieves state-of-the-art results among open-source models of similar size and even surpasses some much larger models, demonstrating superior performance in code generation, code completion, code editing, code reasoning, and software engineering tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_03524 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Seed-Coder: Let the Code Model Curate Data for Itself Seed, ByteDance Zhang, Yuyu Su, Jing Sun, Yifan Xi, Chenguang Xiao, Xia Zheng, Shen Zhang, Anxiang Liu, Kaibo Zan, Daoguang Sun, Tao Zhu, Jinhua Xin, Shulin Huang, Dong Bai, Yetao Dong, Lixin Li, Chao Chen, Jianchong Zhou, Hanzhi Huang, Yifan Ning, Guanghan Song, Xierui Chen, Jiaze Liu, Siyao Shen, Kai Xiang, Liang Wu, Yonghui Computation and Language Software Engineering Code data in large language model (LLM) pretraining is recognized crucial not only for code-related tasks but also for enhancing general intelligence of LLMs. Current open-source LLMs often heavily rely on human effort to produce their code pretraining data, such as employing hand-crafted filtering rules tailored to individual programming languages, or using human-annotated data to train quality filters. However, these approaches are inherently limited in scalability, prone to subjective biases, and costly to extend and maintain across diverse programming languages. To address these challenges, we introduce Seed-Coder, a series of open-source LLMs comprising base, instruct and reasoning models of 8B size, minimizing human involvement in data construction. Our code pretraining data is produced by a model-centric data pipeline, which predominantly leverages LLMs for scoring and filtering code data. The instruct model is further trained via supervised fine-tuning and preference optimization, and the reasoning model leverages Long-Chain-of-Thought (LongCoT) reinforcement learning to improve multi-step code reasoning. Seed-Coder achieves state-of-the-art results among open-source models of similar size and even surpasses some much larger models, demonstrating superior performance in code generation, code completion, code editing, code reasoning, and software engineering tasks. |
| title | Seed-Coder: Let the Code Model Curate Data for Itself |
| topic | Computation and Language Software Engineering |
| url | https://arxiv.org/abs/2506.03524 |