CoCo-Bench: A Comprehensive Code Benchmark For Multi-task Large Language Model Evaluation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916712528478208 |
|---|---|
| author | Yin, Wenjing Sun, Tianze Yu, Yijiong Fang, Jiawei Su, Guangyao Wang, Jiancheng Wang, Zekun Wang, Wei Chen, Ran Dai, Ziyun Yuan, Shuai Dong, Menghang Luo, Peng Cao, Dong Lei, Da Zhang, Yajun Chen, Hao Ma, Xiang Liu, Yong Liu, Weifeng Xu, Yuanjian Pei, Ji |
| author_facet | Yin, Wenjing Sun, Tianze Yu, Yijiong Fang, Jiawei Su, Guangyao Wang, Jiancheng Wang, Zekun Wang, Wei Chen, Ran Dai, Ziyun Yuan, Shuai Dong, Menghang Luo, Peng Cao, Dong Lei, Da Zhang, Yajun Chen, Hao Ma, Xiang Liu, Yong Liu, Weifeng Xu, Yuanjian Pei, Ji |
| contents | Large language models (LLMs) play a crucial role in software engineering, excelling in tasks like code generation and maintenance. However, existing benchmarks are often narrow in scope, focusing on a specific task and lack a comprehensive evaluation framework that reflects real-world applications. To address these gaps, we introduce CoCo-Bench (Comprehensive Code Benchmark), designed to evaluate LLMs across four critical dimensions: code understanding, code generation, code modification, and code review. These dimensions capture essential developer needs, ensuring a more systematic and representative evaluation. CoCo-Bench includes multiple programming languages and varying task difficulties, with rigorous manual review to ensure data quality and accuracy. Empirical results show that CoCo-Bench aligns with existing benchmarks while uncovering significant variations in model performance, effectively highlighting strengths and weaknesses. By offering a holistic and objective evaluation, CoCo-Bench provides valuable insights to guide future research and technological advancements in code-oriented LLMs, establishing a reliable benchmark for the field. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_20673 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CoCo-Bench: A Comprehensive Code Benchmark For Multi-task Large Language Model Evaluation Yin, Wenjing Sun, Tianze Yu, Yijiong Fang, Jiawei Su, Guangyao Wang, Jiancheng Wang, Zekun Wang, Wei Chen, Ran Dai, Ziyun Yuan, Shuai Dong, Menghang Luo, Peng Cao, Dong Lei, Da Zhang, Yajun Chen, Hao Ma, Xiang Liu, Yong Liu, Weifeng Xu, Yuanjian Pei, Ji Software Engineering Artificial Intelligence Large language models (LLMs) play a crucial role in software engineering, excelling in tasks like code generation and maintenance. However, existing benchmarks are often narrow in scope, focusing on a specific task and lack a comprehensive evaluation framework that reflects real-world applications. To address these gaps, we introduce CoCo-Bench (Comprehensive Code Benchmark), designed to evaluate LLMs across four critical dimensions: code understanding, code generation, code modification, and code review. These dimensions capture essential developer needs, ensuring a more systematic and representative evaluation. CoCo-Bench includes multiple programming languages and varying task difficulties, with rigorous manual review to ensure data quality and accuracy. Empirical results show that CoCo-Bench aligns with existing benchmarks while uncovering significant variations in model performance, effectively highlighting strengths and weaknesses. By offering a holistic and objective evaluation, CoCo-Bench provides valuable insights to guide future research and technological advancements in code-oriented LLMs, establishing a reliable benchmark for the field. |
| title | CoCo-Bench: A Comprehensive Code Benchmark For Multi-task Large Language Model Evaluation |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2504.20673 |