NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Prompts
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929337236717568 |
|---|---|
| author | Zhang, Shudan Zhao, Hanlin Liu, Xiao Zheng, Qinkai Qi, Zehan Gu, Xiaotao Zhang, Xiaohan Dong, Yuxiao Tang, Jie |
| author_facet | Zhang, Shudan Zhao, Hanlin Liu, Xiao Zheng, Qinkai Qi, Zehan Gu, Xiaotao Zhang, Xiaohan Dong, Yuxiao Tang, Jie |
| contents | Large language models (LLMs) have manifested strong ability to generate codes for productive activities. However, current benchmarks for code synthesis, such as HumanEval, MBPP, and DS-1000, are predominantly oriented towards introductory tasks on algorithm and data science, insufficiently satisfying challenging requirements prevalent in real-world coding. To fill this gap, we propose NaturalCodeBench (NCB), a challenging code benchmark designed to mirror the complexity and variety of scenarios in real coding tasks. NCB comprises 402 high-quality problems in Python and Java, meticulously selected from natural user queries from online coding services, covering 6 different domains. Noting the extraordinary difficulty in creating testing cases for real-world queries, we also introduce a semi-automated pipeline to enhance the efficiency of test case construction. Comparing with manual solutions, it achieves an efficiency increase of more than 4 times. Our systematic experiments on 39 LLMs find that performance gaps on NCB between models with close HumanEval scores could still be significant, indicating a lack of focus on practical code synthesis scenarios or over-specified optimization on HumanEval. On the other hand, even the best-performing GPT-4 is still far from satisfying on NCB. The evaluation toolkit and development set are available at https://github.com/THUDM/NaturalCodeBench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_04520 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Prompts Zhang, Shudan Zhao, Hanlin Liu, Xiao Zheng, Qinkai Qi, Zehan Gu, Xiaotao Zhang, Xiaohan Dong, Yuxiao Tang, Jie Computation and Language Machine Learning Software Engineering Large language models (LLMs) have manifested strong ability to generate codes for productive activities. However, current benchmarks for code synthesis, such as HumanEval, MBPP, and DS-1000, are predominantly oriented towards introductory tasks on algorithm and data science, insufficiently satisfying challenging requirements prevalent in real-world coding. To fill this gap, we propose NaturalCodeBench (NCB), a challenging code benchmark designed to mirror the complexity and variety of scenarios in real coding tasks. NCB comprises 402 high-quality problems in Python and Java, meticulously selected from natural user queries from online coding services, covering 6 different domains. Noting the extraordinary difficulty in creating testing cases for real-world queries, we also introduce a semi-automated pipeline to enhance the efficiency of test case construction. Comparing with manual solutions, it achieves an efficiency increase of more than 4 times. Our systematic experiments on 39 LLMs find that performance gaps on NCB between models with close HumanEval scores could still be significant, indicating a lack of focus on practical code synthesis scenarios or over-specified optimization on HumanEval. On the other hand, even the best-performing GPT-4 is still far from satisfying on NCB. The evaluation toolkit and development set are available at https://github.com/THUDM/NaturalCodeBench. |
| title | NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Prompts |
| topic | Computation and Language Machine Learning Software Engineering |
| url | https://arxiv.org/abs/2405.04520 |