DevEval: Evaluating Code Generation in Practical Software Projects
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910354671403008 |
|---|---|
| author | Li, Jia Li, Ge Zhao, Yunfei Li, Yongmin Jin, Zhi Zhu, Hao Liu, Huanyu Liu, Kaibo Wang, Lecheng Fang, Zheng Wang, Lanshen Ding, Jiazheng Zhang, Xuanming Dong, Yihong Zhu, Yuqi Gu, Bin Yang, Mengfei |
| author_facet | Li, Jia Li, Ge Zhao, Yunfei Li, Yongmin Jin, Zhi Zhu, Hao Liu, Huanyu Liu, Kaibo Wang, Lecheng Fang, Zheng Wang, Lanshen Ding, Jiazheng Zhang, Xuanming Dong, Yihong Zhu, Yuqi Gu, Bin Yang, Mengfei |
| contents | How to evaluate Large Language Models (LLMs) in code generation is an open question. Many benchmarks have been proposed but are inconsistent with practical software projects, e.g., unreal program distributions, insufficient dependencies, and small-scale project contexts. Thus, the capabilities of LLMs in practical projects are still unclear. In this paper, we propose a new benchmark named DevEval, aligned with Developers' experiences in practical projects. DevEval is collected through a rigorous pipeline, containing 2,690 samples from 119 practical projects and covering 10 domains. Compared to previous benchmarks, DevEval aligns to practical projects in multiple dimensions, e.g., real program distributions, sufficient dependencies, and enough-scale project contexts. We assess five popular LLMs on DevEval (e.g., gpt-4, gpt-3.5-turbo, CodeLLaMa, and StarCoder) and reveal their actual abilities in code generation. For instance, the highest Pass@1 of gpt-3.5-turbo only is 42 in our experiments. We also discuss the challenges and future directions of code generation in practical projects. We open-source DevEval and hope it can facilitate the development of code generation in practical projects. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_06401 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DevEval: Evaluating Code Generation in Practical Software Projects Li, Jia Li, Ge Zhao, Yunfei Li, Yongmin Jin, Zhi Zhu, Hao Liu, Huanyu Liu, Kaibo Wang, Lecheng Fang, Zheng Wang, Lanshen Ding, Jiazheng Zhang, Xuanming Dong, Yihong Zhu, Yuqi Gu, Bin Yang, Mengfei Software Engineering Artificial Intelligence Computation and Language How to evaluate Large Language Models (LLMs) in code generation is an open question. Many benchmarks have been proposed but are inconsistent with practical software projects, e.g., unreal program distributions, insufficient dependencies, and small-scale project contexts. Thus, the capabilities of LLMs in practical projects are still unclear. In this paper, we propose a new benchmark named DevEval, aligned with Developers' experiences in practical projects. DevEval is collected through a rigorous pipeline, containing 2,690 samples from 119 practical projects and covering 10 domains. Compared to previous benchmarks, DevEval aligns to practical projects in multiple dimensions, e.g., real program distributions, sufficient dependencies, and enough-scale project contexts. We assess five popular LLMs on DevEval (e.g., gpt-4, gpt-3.5-turbo, CodeLLaMa, and StarCoder) and reveal their actual abilities in code generation. For instance, the highest Pass@1 of gpt-3.5-turbo only is 42 in our experiments. We also discuss the challenges and future directions of code generation in practical projects. We open-source DevEval and hope it can facilitate the development of code generation in practical projects. |
| title | DevEval: Evaluating Code Generation in Practical Software Projects |
| topic | Software Engineering Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2401.06401 |