COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912230200573952 |
|---|---|
| author | Yang, Weiqing Wang, Hanbin Liu, Zhenghao Li, Xinze Yan, Yukun Wang, Shuo Gu, Yu Yu, Minghe Liu, Zhiyuan Yu, Ge |
| author_facet | Yang, Weiqing Wang, Hanbin Liu, Zhenghao Li, Xinze Yan, Yukun Wang, Shuo Gu, Yu Yu, Minghe Liu, Zhiyuan Yu, Ge |
| contents | Code debugging is a vital stage of software development, essential for ensuring the reliability and performance of Large Language Models (LLMs) in the code generation task. Human debugging typically follows a multi-stage process, which includes Bug Localization, Bug Identification, Code Repair, and Code Recognition. However, existing code debugging benchmarks predominantly focus on the Code Repair stage, which offers only a limited perspective on evaluating the debugging capabilities of LLMs. In this paper, we introduce DEBUGEVAL, a comprehensive benchmark for evaluating the debugging abilities of LLMs by emulating the multi-stage human debugging process. Through evaluating on DEBUGEVAL, we observe that 7B-scale models consistently underperform compared to their larger counterparts, highlighting their limitations in comprehending code semantics. In this case, we propose the COmmunicative Agent-based data SynThesis (COAST) framework, which employs a multi-agent system to generate high-quality training data for supervised fine-tuning (SFT). Experimental results demonstrate that COAST-generated data outperform human-curated and GPT-4-generated data, enabling 7B-scale LLMs to achieve debugging performance comparable to GPT-3.5. All data and codes are available at https://github.com/NEUIR/COAST. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_05006 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis Yang, Weiqing Wang, Hanbin Liu, Zhenghao Li, Xinze Yan, Yukun Wang, Shuo Gu, Yu Yu, Minghe Liu, Zhiyuan Yu, Ge Software Engineering Artificial Intelligence Code debugging is a vital stage of software development, essential for ensuring the reliability and performance of Large Language Models (LLMs) in the code generation task. Human debugging typically follows a multi-stage process, which includes Bug Localization, Bug Identification, Code Repair, and Code Recognition. However, existing code debugging benchmarks predominantly focus on the Code Repair stage, which offers only a limited perspective on evaluating the debugging capabilities of LLMs. In this paper, we introduce DEBUGEVAL, a comprehensive benchmark for evaluating the debugging abilities of LLMs by emulating the multi-stage human debugging process. Through evaluating on DEBUGEVAL, we observe that 7B-scale models consistently underperform compared to their larger counterparts, highlighting their limitations in comprehending code semantics. In this case, we propose the COmmunicative Agent-based data SynThesis (COAST) framework, which employs a multi-agent system to generate high-quality training data for supervised fine-tuning (SFT). Experimental results demonstrate that COAST-generated data outperform human-curated and GPT-4-generated data, enabling 7B-scale LLMs to achieve debugging performance comparable to GPT-3.5. All data and codes are available at https://github.com/NEUIR/COAST. |
| title | COAST: Enhancing the Code Debugging Ability of LLMs through Communicative Agent Based Data Synthesis |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2408.05006 |