ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913502024695808 |
|---|---|
| author | Feng, Jia Liu, Jiachen Gao, Cuiyun Chong, Chun Yong Wang, Chaozheng Gao, Shan Xia, Xin |
| author_facet | Feng, Jia Liu, Jiachen Gao, Cuiyun Chong, Chun Yong Wang, Chaozheng Gao, Shan Xia, Xin |
| contents | In recent years, the application of large language models (LLMs) to code-related tasks has gained significant attention. However, existing evaluation benchmarks often focus on limited scenarios, such as code generation or completion, which do not reflect the diverse challenges developers face in real-world contexts. To address this, we introduce ComplexCodeEval, a benchmark designed to assess LCMs in various development tasks, including code generation, completion, API recommendation, and test case generation. It includes 3,897 Java samples and 7,184 Python samples from high-star GitHub repositories, each annotated with function signatures, docstrings, and API references to simulate real development environments. Our experiments across ten LCMs reveal that context improves performance and that data leakage can lead to overestimation, highlighting the need for more accurate evaluations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_10280 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code Feng, Jia Liu, Jiachen Gao, Cuiyun Chong, Chun Yong Wang, Chaozheng Gao, Shan Xia, Xin Software Engineering In recent years, the application of large language models (LLMs) to code-related tasks has gained significant attention. However, existing evaluation benchmarks often focus on limited scenarios, such as code generation or completion, which do not reflect the diverse challenges developers face in real-world contexts. To address this, we introduce ComplexCodeEval, a benchmark designed to assess LCMs in various development tasks, including code generation, completion, API recommendation, and test case generation. It includes 3,897 Java samples and 7,184 Python samples from high-star GitHub repositories, each annotated with function signatures, docstrings, and API references to simulate real development environments. Our experiments across ten LCMs reveal that context improves performance and that data leakage can lead to overestimation, highlighting the need for more accurate evaluations. |
| title | ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2409.10280 |