Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2506.11928 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908407337844736 |
|---|---|
| author | Zheng, Zihan Cheng, Zerui Shen, Zeyu Zhou, Shang Liu, Kaiyuan He, Hansen Li, Dongruixuan Wei, Stanley Hao, Hangyi Yao, Jianzhu Sheng, Peiyao Wang, Zixuan Chai, Wenhao Korolova, Aleksandra Henderson, Peter Arora, Sanjeev Viswanath, Pramod Shang, Jingbo Xie, Saining |
| author_facet | Zheng, Zihan Cheng, Zerui Shen, Zeyu Zhou, Shang Liu, Kaiyuan He, Hansen Li, Dongruixuan Wei, Stanley Hao, Hangyi Yao, Jianzhu Sheng, Peiyao Wang, Zixuan Chai, Wenhao Korolova, Aleksandra Henderson, Peter Arora, Sanjeev Viswanath, Pramod Shang, Jingbo Xie, Saining |
| contents | Recent reports claim that large language models (LLMs) now outperform elite humans in competitive programming. Drawing on knowledge from a group of medalists in international algorithmic contests, we revisit this claim, examining how LLMs differ from human experts and where limitations still remain. We introduce LiveCodeBench Pro, a benchmark composed of problems from Codeforces, ICPC, and IOI that are continuously updated to reduce the likelihood of data contamination. A team of Olympiad medalists annotates every problem for algorithmic categories and conducts a line-by-line analysis of failed model-generated submissions. Using this new data and benchmark, we find that frontier models still have significant limitations: without external tools, the best model achieves only 53% pass@1 on medium-difficulty problems and 0% on hard problems, domains where expert humans still excel. We also find that LLMs succeed at implementation-heavy problems but struggle with nuanced algorithmic reasoning and complex case analysis, often generating confidently incorrect justifications. High performance appears largely driven by implementation precision and tool augmentation, not superior reasoning. LiveCodeBench Pro thus highlights the significant gap to human grandmaster levels, while offering fine-grained diagnostics to steer future improvements in code-centric LLM reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_11928 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? Zheng, Zihan Cheng, Zerui Shen, Zeyu Zhou, Shang Liu, Kaiyuan He, Hansen Li, Dongruixuan Wei, Stanley Hao, Hangyi Yao, Jianzhu Sheng, Peiyao Wang, Zixuan Chai, Wenhao Korolova, Aleksandra Henderson, Peter Arora, Sanjeev Viswanath, Pramod Shang, Jingbo Xie, Saining Software Engineering Artificial Intelligence Computation and Language Machine Learning Recent reports claim that large language models (LLMs) now outperform elite humans in competitive programming. Drawing on knowledge from a group of medalists in international algorithmic contests, we revisit this claim, examining how LLMs differ from human experts and where limitations still remain. We introduce LiveCodeBench Pro, a benchmark composed of problems from Codeforces, ICPC, and IOI that are continuously updated to reduce the likelihood of data contamination. A team of Olympiad medalists annotates every problem for algorithmic categories and conducts a line-by-line analysis of failed model-generated submissions. Using this new data and benchmark, we find that frontier models still have significant limitations: without external tools, the best model achieves only 53% pass@1 on medium-difficulty problems and 0% on hard problems, domains where expert humans still excel. We also find that LLMs succeed at implementation-heavy problems but struggle with nuanced algorithmic reasoning and complex case analysis, often generating confidently incorrect justifications. High performance appears largely driven by implementation precision and tool augmentation, not superior reasoning. LiveCodeBench Pro thus highlights the significant gap to human grandmaster levels, while offering fine-grained diagnostics to steer future improvements in code-centric LLM reasoning. |
| title | LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? |
| topic | Software Engineering Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2506.11928 |