CodeArena: A Collective Evaluation Platform for LLM Code Generation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912256010223616 |
|---|---|
| author | Du, Mingzhe Luu, Anh Tuan Ji, Bin Wu, Xiaobao Huang, Dong Zhuo, Terry Yue Liu, Qian Ng, See-Kiong |
| author_facet | Du, Mingzhe Luu, Anh Tuan Ji, Bin Wu, Xiaobao Huang, Dong Zhuo, Terry Yue Liu, Qian Ng, See-Kiong |
| contents | Large Language Models (LLMs) have reshaped code generation by synergizing their exceptional comprehension of natural language and programming syntax, thereby substantially boosting developer productivity. These advancements have prompted numerous efforts to quantitatively evaluate their coding capabilities. However, persistent challenges, such as benchmark leakage, data dissipation, and limited system accessibility, continue to impede a timely and accurate assessment. To address these limitations, we introduce CodeArena, an online evaluation framework tailored for LLM code generation. The key innovation is a collective evaluation mechanism, which dynamically recalibrates individual model scores based on the holistic performance of all participating models, mitigating score biases caused by widespread benchmark leakage. In addition, CodeArena ensures open access to all submitted solutions and test cases and provides automation-friendly APIs to streamline the code evaluation workflow. Our main contributions are: (1) a collective evaluation system for unbiased assessment, (2) a public repository of solutions and test cases, and (3) automation-ready APIs for seamless integration. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01295 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CodeArena: A Collective Evaluation Platform for LLM Code Generation Du, Mingzhe Luu, Anh Tuan Ji, Bin Wu, Xiaobao Huang, Dong Zhuo, Terry Yue Liu, Qian Ng, See-Kiong Software Engineering Large Language Models (LLMs) have reshaped code generation by synergizing their exceptional comprehension of natural language and programming syntax, thereby substantially boosting developer productivity. These advancements have prompted numerous efforts to quantitatively evaluate their coding capabilities. However, persistent challenges, such as benchmark leakage, data dissipation, and limited system accessibility, continue to impede a timely and accurate assessment. To address these limitations, we introduce CodeArena, an online evaluation framework tailored for LLM code generation. The key innovation is a collective evaluation mechanism, which dynamically recalibrates individual model scores based on the holistic performance of all participating models, mitigating score biases caused by widespread benchmark leakage. In addition, CodeArena ensures open access to all submitted solutions and test cases and provides automation-friendly APIs to streamline the code evaluation workflow. Our main contributions are: (1) a collective evaluation system for unbiased assessment, (2) a public repository of solutions and test cases, and (3) automation-ready APIs for seamless integration. |
| title | CodeArena: A Collective Evaluation Platform for LLM Code Generation |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2503.01295 |