What's Wrong with Your Code Generated by Large Language Models? An Extensive Study
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914098799706112 |
|---|---|
| author | Dou, Shihan Jia, Haoxiang Wu, Shenxi Zheng, Huiyuan Wu, Muling Tao, Yunbo Zhang, Ming Chai, Mingxu Fan, Jessica Xi, Zhiheng Zheng, Rui Wu, Yueming Wen, Ming Gui, Tao Zhang, Qi Qiu, Xipeng Huang, Xuanjing |
| author_facet | Dou, Shihan Jia, Haoxiang Wu, Shenxi Zheng, Huiyuan Wu, Muling Tao, Yunbo Zhang, Ming Chai, Mingxu Fan, Jessica Xi, Zhiheng Zheng, Rui Wu, Yueming Wen, Ming Gui, Tao Zhang, Qi Qiu, Xipeng Huang, Xuanjing |
| contents | The increasing development of LLMs in code generation has drawn significant attention among researchers. To enhance LLM-based code generation ability, current efforts are predominantly directed towards collecting high-quality datasets and leveraging diverse training technologies. However, there is a notable lack of comprehensive studies examining the limitations and boundaries of existing methods. To bridge this gap, we conducted an extensive empirical study evaluating the performance of three leading closed-source LLMs and six popular open-source LLMs on three commonly used benchmarks. Our investigation, which evaluated the length, cyclomatic complexity and API number of the generated code, revealed that these LLMs face challenges in generating successful code for more complex problems, and tend to produce code that is shorter yet more complicated as compared to canonical solutions. Additionally, we developed a taxonomy of bugs for incorrect codes that includes three categories and ten sub-categories, and analyzed the root cause for common bug types. To better understand the performance of LLMs in real-world projects, we also manually created a real-world benchmark RWPB. We analyzed bugs on RWPB to highlight distinct differences in bug distributions between actual scenarios and existing benchmarks. Finally, we propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code based on bug types and compiler feedback. Our comprehensive and extensive study provides insights into the current limitations of LLM-based code generation and opportunities for enhancing the accuracy and quality of the generated code. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_06153 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | What's Wrong with Your Code Generated by Large Language Models? An Extensive Study Dou, Shihan Jia, Haoxiang Wu, Shenxi Zheng, Huiyuan Wu, Muling Tao, Yunbo Zhang, Ming Chai, Mingxu Fan, Jessica Xi, Zhiheng Zheng, Rui Wu, Yueming Wen, Ming Gui, Tao Zhang, Qi Qiu, Xipeng Huang, Xuanjing Software Engineering Computation and Language The increasing development of LLMs in code generation has drawn significant attention among researchers. To enhance LLM-based code generation ability, current efforts are predominantly directed towards collecting high-quality datasets and leveraging diverse training technologies. However, there is a notable lack of comprehensive studies examining the limitations and boundaries of existing methods. To bridge this gap, we conducted an extensive empirical study evaluating the performance of three leading closed-source LLMs and six popular open-source LLMs on three commonly used benchmarks. Our investigation, which evaluated the length, cyclomatic complexity and API number of the generated code, revealed that these LLMs face challenges in generating successful code for more complex problems, and tend to produce code that is shorter yet more complicated as compared to canonical solutions. Additionally, we developed a taxonomy of bugs for incorrect codes that includes three categories and ten sub-categories, and analyzed the root cause for common bug types. To better understand the performance of LLMs in real-world projects, we also manually created a real-world benchmark RWPB. We analyzed bugs on RWPB to highlight distinct differences in bug distributions between actual scenarios and existing benchmarks. Finally, we propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code based on bug types and compiler feedback. Our comprehensive and extensive study provides insights into the current limitations of LLM-based code generation and opportunities for enhancing the accuracy and quality of the generated code. |
| title | What's Wrong with Your Code Generated by Large Language Models? An Extensive Study |
| topic | Software Engineering Computation and Language |
| url | https://arxiv.org/abs/2407.06153 |