LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916276676329472 |
|---|---|
| author | Jain, Naman Han, King Gu, Alex Li, Wen-Ding Yan, Fanjia Zhang, Tianjun Wang, Sida Solar-Lezama, Armando Sen, Koushik Stoica, Ion |
| author_facet | Jain, Naman Han, King Gu, Alex Li, Wen-Ding Yan, Fanjia Zhang, Tianjun Wang, Sida Solar-Lezama, Armando Sen, Koushik Stoica, Ion |
| contents | Large Language Models (LLMs) applied to code-related applications have emerged as a prominent field, attracting significant interest from both academia and industry. However, as new and improved LLMs are developed, existing evaluation benchmarks (e.g., HumanEval, MBPP) are no longer sufficient for assessing their capabilities. In this work, we propose LiveCodeBench, a comprehensive and contamination-free evaluation of LLMs for code, which continuously collects new problems over time from contests across three competition platforms, namely LeetCode, AtCoder, and CodeForces. Notably, our benchmark also focuses on a broader range of code related capabilities, such as self-repair, code execution, and test output prediction, beyond just code generation. Currently, LiveCodeBench hosts four hundred high-quality coding problems that were published between May 2023 and May 2024. We have evaluated 18 base LLMs and 34 instruction-tuned LLMs on LiveCodeBench. We present empirical findings on contamination, holistic performance comparisons, potential overfitting in existing benchmarks as well as individual model comparisons. We will release all prompts and model completions for further community analysis, along with a general toolkit for adding new scenarios and model |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_07974 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code Jain, Naman Han, King Gu, Alex Li, Wen-Ding Yan, Fanjia Zhang, Tianjun Wang, Sida Solar-Lezama, Armando Sen, Koushik Stoica, Ion Software Engineering Computation and Language Machine Learning Large Language Models (LLMs) applied to code-related applications have emerged as a prominent field, attracting significant interest from both academia and industry. However, as new and improved LLMs are developed, existing evaluation benchmarks (e.g., HumanEval, MBPP) are no longer sufficient for assessing their capabilities. In this work, we propose LiveCodeBench, a comprehensive and contamination-free evaluation of LLMs for code, which continuously collects new problems over time from contests across three competition platforms, namely LeetCode, AtCoder, and CodeForces. Notably, our benchmark also focuses on a broader range of code related capabilities, such as self-repair, code execution, and test output prediction, beyond just code generation. Currently, LiveCodeBench hosts four hundred high-quality coding problems that were published between May 2023 and May 2024. We have evaluated 18 base LLMs and 34 instruction-tuned LLMs on LiveCodeBench. We present empirical findings on contamination, holistic performance comparisons, potential overfitting in existing benchmarks as well as individual model comparisons. We will release all prompts and model completions for further community analysis, along with a general toolkit for adding new scenarios and model |
| title | LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code |
| topic | Software Engineering Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2403.07974 |