DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909213299572736 |
|---|---|
| author | Li, Jia Li, Ge Zhao, Yunfei Li, Yongmin Liu, Huanyu Zhu, Hao Wang, Lecheng Liu, Kaibo Fang, Zheng Wang, Lanshen Ding, Jiazheng Zhang, Xuanming Zhu, Yuqi Dong, Yihong Jin, Zhi Li, Binhua Huang, Fei Li, Yongbin |
| author_facet | Li, Jia Li, Ge Zhao, Yunfei Li, Yongmin Liu, Huanyu Zhu, Hao Wang, Lecheng Liu, Kaibo Fang, Zheng Wang, Lanshen Ding, Jiazheng Zhang, Xuanming Zhu, Yuqi Dong, Yihong Jin, Zhi Li, Binhua Huang, Fei Li, Yongbin |
| contents | How to evaluate the coding abilities of Large Language Models (LLMs) remains an open question. We find that existing benchmarks are poorly aligned with real-world code repositories and are insufficient to evaluate the coding abilities of LLMs.
To address the knowledge gap, we propose a new benchmark named DevEval, which has three advances. (1) DevEval aligns with real-world repositories in multiple dimensions, e.g., code distributions and dependency distributions. (2) DevEval is annotated by 13 developers and contains comprehensive annotations (e.g., requirements, original repositories, reference code, and reference dependencies). (3) DevEval comprises 1,874 testing samples from 117 repositories, covering 10 popular domains (e.g., Internet, Database). Based on DevEval, we propose repository-level code generation and evaluate 8 popular LLMs on DevEval (e.g., gpt-4, gpt-3.5, StarCoder 2, DeepSeek Coder, CodeLLaMa). Our experiments reveal these LLMs' coding abilities in real-world code repositories. For example, in our experiments, the highest Pass@1 of gpt-4-turbo is only 53.04%. We also analyze LLMs' failed cases and summarize their shortcomings. We hope DevEval can facilitate the development of LLMs in real code repositories. DevEval, prompts, and LLMs' predictions have been released. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_19856 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories Li, Jia Li, Ge Zhao, Yunfei Li, Yongmin Liu, Huanyu Zhu, Hao Wang, Lecheng Liu, Kaibo Fang, Zheng Wang, Lanshen Ding, Jiazheng Zhang, Xuanming Zhu, Yuqi Dong, Yihong Jin, Zhi Li, Binhua Huang, Fei Li, Yongbin Computation and Language Software Engineering How to evaluate the coding abilities of Large Language Models (LLMs) remains an open question. We find that existing benchmarks are poorly aligned with real-world code repositories and are insufficient to evaluate the coding abilities of LLMs. To address the knowledge gap, we propose a new benchmark named DevEval, which has three advances. (1) DevEval aligns with real-world repositories in multiple dimensions, e.g., code distributions and dependency distributions. (2) DevEval is annotated by 13 developers and contains comprehensive annotations (e.g., requirements, original repositories, reference code, and reference dependencies). (3) DevEval comprises 1,874 testing samples from 117 repositories, covering 10 popular domains (e.g., Internet, Database). Based on DevEval, we propose repository-level code generation and evaluate 8 popular LLMs on DevEval (e.g., gpt-4, gpt-3.5, StarCoder 2, DeepSeek Coder, CodeLLaMa). Our experiments reveal these LLMs' coding abilities in real-world code repositories. For example, in our experiments, the highest Pass@1 of gpt-4-turbo is only 53.04%. We also analyze LLMs' failed cases and summarize their shortcomings. We hope DevEval can facilitate the development of LLMs in real code repositories. DevEval, prompts, and LLMs' predictions have been released. |
| title | DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories |
| topic | Computation and Language Software Engineering |
| url | https://arxiv.org/abs/2405.19856 |