CodeEditorBench: Evaluating Code Editing Capability of Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910905929826304 |
|---|---|
| author | Guo, Jiawei Li, Ziming Liu, Xueling Ma, Kaijing Zheng, Tianyu Yu, Zhouliang Pan, Ding LI, Yizhi Liu, Ruibo Wang, Yue Guo, Shuyue Qu, Xingwei Yue, Xiang Zhang, Ge Chen, Wenhu Fu, Jie |
| author_facet | Guo, Jiawei Li, Ziming Liu, Xueling Ma, Kaijing Zheng, Tianyu Yu, Zhouliang Pan, Ding LI, Yizhi Liu, Ruibo Wang, Yue Guo, Shuyue Qu, Xingwei Yue, Xiang Zhang, Ge Chen, Wenhu Fu, Jie |
| contents | Large Language Models (LLMs) for code are rapidly evolving, with code editing emerging as a critical capability. We introduce CodeEditorBench, an evaluation framework designed to rigorously assess the performance of LLMs in code editing tasks, including debugging, translating, polishing, and requirement switching. Unlike existing benchmarks focusing solely on code generation, CodeEditorBench emphasizes real-world scenarios and practical aspects of software development. We curate diverse coding challenges and scenarios from five sources, covering various programming languages, complexity levels, and editing tasks. Evaluation of 19 LLMs reveals that closed-source models (particularly Gemini-Ultra and GPT-4), outperform open-source models in CodeEditorBench, highlighting differences in model performance based on problem types and prompt sensitivities. CodeEditorBench aims to catalyze advancements in LLMs by providing a robust platform for assessing code editing capabilities. We will release all prompts and datasets to enable the community to expand the dataset and benchmark emerging LLMs. By introducing CodeEditorBench, we contribute to the advancement of LLMs in code editing and provide a valuable resource for researchers and practitioners. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_03543 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CodeEditorBench: Evaluating Code Editing Capability of Large Language Models Guo, Jiawei Li, Ziming Liu, Xueling Ma, Kaijing Zheng, Tianyu Yu, Zhouliang Pan, Ding LI, Yizhi Liu, Ruibo Wang, Yue Guo, Shuyue Qu, Xingwei Yue, Xiang Zhang, Ge Chen, Wenhu Fu, Jie Software Engineering Artificial Intelligence Computation and Language Machine Learning Large Language Models (LLMs) for code are rapidly evolving, with code editing emerging as a critical capability. We introduce CodeEditorBench, an evaluation framework designed to rigorously assess the performance of LLMs in code editing tasks, including debugging, translating, polishing, and requirement switching. Unlike existing benchmarks focusing solely on code generation, CodeEditorBench emphasizes real-world scenarios and practical aspects of software development. We curate diverse coding challenges and scenarios from five sources, covering various programming languages, complexity levels, and editing tasks. Evaluation of 19 LLMs reveals that closed-source models (particularly Gemini-Ultra and GPT-4), outperform open-source models in CodeEditorBench, highlighting differences in model performance based on problem types and prompt sensitivities. CodeEditorBench aims to catalyze advancements in LLMs by providing a robust platform for assessing code editing capabilities. We will release all prompts and datasets to enable the community to expand the dataset and benchmark emerging LLMs. By introducing CodeEditorBench, we contribute to the advancement of LLMs in code editing and provide a valuable resource for researchers and practitioners. |
| title | CodeEditorBench: Evaluating Code Editing Capability of Large Language Models |
| topic | Software Engineering Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2404.03543 |