OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915434101473280 |
|---|---|
| author | Ahmad, Wasi Uddin Ficek, Aleksander Samadi, Mehrzad Huang, Jocelyn Noroozi, Vahid Majumdar, Somshubra Ginsburg, Boris |
| author_facet | Ahmad, Wasi Uddin Ficek, Aleksander Samadi, Mehrzad Huang, Jocelyn Noroozi, Vahid Majumdar, Somshubra Ginsburg, Boris |
| contents | Large Language Models (LLMs) have transformed software development by enabling code generation, automated debugging, and complex reasoning. However, their continued advancement is constrained by the scarcity of high-quality, publicly available supervised fine-tuning (SFT) datasets tailored for coding tasks. To bridge this gap, we introduce OpenCodeInstruct, the largest open-access instruction tuning dataset, comprising 5 million diverse samples. Each sample includes a programming question, solution, test cases, execution feedback, and LLM-generated quality assessments. We fine-tune various base models, including LLaMA and Qwen, across multiple scales (1B+, 3B+, and 7B+) using our dataset. Comprehensive evaluations on popular benchmarks (HumanEval, MBPP, LiveCodeBench, and BigCodeBench) demonstrate substantial performance improvements achieved by SFT with OpenCodeInstruct. We also present a detailed methodology encompassing seed data curation, synthetic instruction and solution generation, and filtering. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_04030 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs Ahmad, Wasi Uddin Ficek, Aleksander Samadi, Mehrzad Huang, Jocelyn Noroozi, Vahid Majumdar, Somshubra Ginsburg, Boris Software Engineering Computation and Language Large Language Models (LLMs) have transformed software development by enabling code generation, automated debugging, and complex reasoning. However, their continued advancement is constrained by the scarcity of high-quality, publicly available supervised fine-tuning (SFT) datasets tailored for coding tasks. To bridge this gap, we introduce OpenCodeInstruct, the largest open-access instruction tuning dataset, comprising 5 million diverse samples. Each sample includes a programming question, solution, test cases, execution feedback, and LLM-generated quality assessments. We fine-tune various base models, including LLaMA and Qwen, across multiple scales (1B+, 3B+, and 7B+) using our dataset. Comprehensive evaluations on popular benchmarks (HumanEval, MBPP, LiveCodeBench, and BigCodeBench) demonstrate substantial performance improvements achieved by SFT with OpenCodeInstruct. We also present a detailed methodology encompassing seed data curation, synthetic instruction and solution generation, and filtering. |
| title | OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs |
| topic | Software Engineering Computation and Language |
| url | https://arxiv.org/abs/2504.04030 |