Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2309.01940 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916153705627648 |
|---|---|
| author | Fu, Lingyue Chai, Huacan Luo, Shuang Du, Kounianhua Zhang, Weiming Fan, Longteng Lei, Jiayi Rui, Renting Lin, Jianghao Fang, Yuchen Liu, Yifan Wang, Jingkuan Qi, Siyuan Zhang, Kangning Zhang, Weinan Yu, Yong |
| author_facet | Fu, Lingyue Chai, Huacan Luo, Shuang Du, Kounianhua Zhang, Weiming Fan, Longteng Lei, Jiayi Rui, Renting Lin, Jianghao Fang, Yuchen Liu, Yifan Wang, Jingkuan Qi, Siyuan Zhang, Kangning Zhang, Weinan Yu, Yong |
| contents | With the emergence of Large Language Models (LLMs), there has been a significant improvement in the programming capabilities of models, attracting growing attention from researchers. Evaluating the programming capabilities of LLMs is crucial as it reflects the multifaceted abilities of LLMs, and it has numerous downstream applications. In this paper, we propose CodeApex, a bilingual benchmark dataset focusing on the programming comprehension, code generation, and code correction abilities of LLMs. Programming comprehension task tests LLMs on multiple-choice exam questions covering conceptual understanding, commonsense reasoning, and multi-hop reasoning. The code generation task evaluates LLMs through completing C++ functions based on provided descriptions and prototypes. The code correction task asks LLMs to fix real-world erroneous code segments with different error messages. We evaluate 12 widely used LLMs, including both general-purpose and specialized models. GPT-4 exhibits the best programming capabilities, achieving approximate accuracy of 69%, 54%, and 66% on the three tasks, respectively. Compared to human performance, there is still significant room for improvement in LLM programming. We hope that CodeApex can serve as a reference for evaluating the coding capabilities of LLMs, further promoting their development and growth. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_01940 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | CodeApex: A Bilingual Programming Evaluation Benchmark for Large Language Models Fu, Lingyue Chai, Huacan Luo, Shuang Du, Kounianhua Zhang, Weiming Fan, Longteng Lei, Jiayi Rui, Renting Lin, Jianghao Fang, Yuchen Liu, Yifan Wang, Jingkuan Qi, Siyuan Zhang, Kangning Zhang, Weinan Yu, Yong Computation and Language Artificial Intelligence With the emergence of Large Language Models (LLMs), there has been a significant improvement in the programming capabilities of models, attracting growing attention from researchers. Evaluating the programming capabilities of LLMs is crucial as it reflects the multifaceted abilities of LLMs, and it has numerous downstream applications. In this paper, we propose CodeApex, a bilingual benchmark dataset focusing on the programming comprehension, code generation, and code correction abilities of LLMs. Programming comprehension task tests LLMs on multiple-choice exam questions covering conceptual understanding, commonsense reasoning, and multi-hop reasoning. The code generation task evaluates LLMs through completing C++ functions based on provided descriptions and prototypes. The code correction task asks LLMs to fix real-world erroneous code segments with different error messages. We evaluate 12 widely used LLMs, including both general-purpose and specialized models. GPT-4 exhibits the best programming capabilities, achieving approximate accuracy of 69%, 54%, and 66% on the three tasks, respectively. Compared to human performance, there is still significant room for improvement in LLM programming. We hope that CodeApex can serve as a reference for evaluating the coding capabilities of LLMs, further promoting their development and growth. |
| title | CodeApex: A Bilingual Programming Evaluation Benchmark for Large Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2309.01940 |