StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916114453233664 |
|---|---|
| author | Dou, Shihan Liu, Yan Jia, Haoxiang Xiong, Limao Zhou, Enyu Shen, Wei Shan, Junjie Huang, Caishuang Wang, Xiao Fan, Xiaoran Xi, Zhiheng Zhou, Yuhao Ji, Tao Zheng, Rui Zhang, Qi Huang, Xuanjing Gui, Tao |
| author_facet | Dou, Shihan Liu, Yan Jia, Haoxiang Xiong, Limao Zhou, Enyu Shen, Wei Shan, Junjie Huang, Caishuang Wang, Xiao Fan, Xiaoran Xi, Zhiheng Zhou, Yuhao Ji, Tao Zheng, Rui Zhang, Qi Huang, Xuanjing Gui, Tao |
| contents | The advancement of large language models (LLMs) has significantly propelled the field of code generation. Previous work integrated reinforcement learning (RL) with compiler feedback for exploring the output space of LLMs to enhance code generation quality. However, the lengthy code generated by LLMs in response to complex human requirements makes RL exploration a challenge. Also, since the unit tests may not cover the complicated code, optimizing LLMs by using these unexecuted code snippets is ineffective. To tackle these challenges, we introduce StepCoder, a novel RL framework for code generation, consisting of two main components: CCCS addresses the exploration challenge by breaking the long sequences code generation task into a Curriculum of Code Completion Subtasks, while FGO only optimizes the model by masking the unexecuted code segments to provide Fine-Grained Optimization. In addition, we furthermore construct the APPS+ dataset for RL training, which is manually verified to ensure the correctness of unit tests. Experimental results show that our method improves the ability to explore the output space and outperforms state-of-the-art approaches in corresponding benchmarks. Our dataset APPS+ and StepCoder are available online. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_01391 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback Dou, Shihan Liu, Yan Jia, Haoxiang Xiong, Limao Zhou, Enyu Shen, Wei Shan, Junjie Huang, Caishuang Wang, Xiao Fan, Xiaoran Xi, Zhiheng Zhou, Yuhao Ji, Tao Zheng, Rui Zhang, Qi Huang, Xuanjing Gui, Tao Software Engineering Computation and Language The advancement of large language models (LLMs) has significantly propelled the field of code generation. Previous work integrated reinforcement learning (RL) with compiler feedback for exploring the output space of LLMs to enhance code generation quality. However, the lengthy code generated by LLMs in response to complex human requirements makes RL exploration a challenge. Also, since the unit tests may not cover the complicated code, optimizing LLMs by using these unexecuted code snippets is ineffective. To tackle these challenges, we introduce StepCoder, a novel RL framework for code generation, consisting of two main components: CCCS addresses the exploration challenge by breaking the long sequences code generation task into a Curriculum of Code Completion Subtasks, while FGO only optimizes the model by masking the unexecuted code segments to provide Fine-Grained Optimization. In addition, we furthermore construct the APPS+ dataset for RL training, which is manually verified to ensure the correctness of unit tests. Experimental results show that our method improves the ability to explore the output space and outperforms state-of-the-art approaches in corresponding benchmarks. Our dataset APPS+ and StepCoder are available online. |
| title | StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback |
| topic | Software Engineering Computation and Language |
| url | https://arxiv.org/abs/2402.01391 |