Humanity's Last Code Exam: Can Advanced LLMs Conquer Human's Hardest Code Competition?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912656372269056 |
|---|---|
| author | Li, Xiangyang Li, Xiaopeng Dong, Kuicai Zhang, Quanhu Ruan, Rongju Dai, Xinyi Liu, Xiaoshuang Xu, Shengchun Wang, Yasheng Tang, Ruiming |
| author_facet | Li, Xiangyang Li, Xiaopeng Dong, Kuicai Zhang, Quanhu Ruan, Rongju Dai, Xinyi Liu, Xiaoshuang Xu, Shengchun Wang, Yasheng Tang, Ruiming |
| contents | Code generation is a core capability of large language models (LLMs), yet mainstream benchmarks (e.g., APPs and LiveCodeBench) contain questions with medium-level difficulty and pose no challenge to advanced LLMs. To better reflected the advanced reasoning and code generation ability, We introduce Humanity's Last Code Exam (HLCE), comprising 235 most challenging problems from the International Collegiate Programming Contest (ICPC World Finals) and the International Olympiad in Informatics (IOI) spanning 2010 - 2024. As part of HLCE, we design a harmonized online-offline sandbox that guarantees fully reproducible evaluation. Through our comprehensive evaluation, we observe that even the strongest reasoning LLMs: o4-mini(high) and Gemini-2.5 Pro, achieve pass@1 rates of only 15.9% and 11.4%, respectively. Meanwhile, we propose a novel "self-recognition" task to measure LLMs' awareness of their own capabilities. Results indicate that LLMs' self-recognition abilities are not proportionally correlated with their code generation performance. Finally, our empirical validation of test-time scaling laws reveals that current advanced LLMs have substantial room for improvement on complex programming tasks. We expect HLCE to become a milestone challenge for code generation and to catalyze advances in high-performance reasoning and human-AI collaborative programming. Our code and dataset are also public available(https://github.com/Humanity-s-Last-Code-Exam/HLCE). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_12713 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Humanity's Last Code Exam: Can Advanced LLMs Conquer Human's Hardest Code Competition? Li, Xiangyang Li, Xiaopeng Dong, Kuicai Zhang, Quanhu Ruan, Rongju Dai, Xinyi Liu, Xiaoshuang Xu, Shengchun Wang, Yasheng Tang, Ruiming Software Engineering Computation and Language Code generation is a core capability of large language models (LLMs), yet mainstream benchmarks (e.g., APPs and LiveCodeBench) contain questions with medium-level difficulty and pose no challenge to advanced LLMs. To better reflected the advanced reasoning and code generation ability, We introduce Humanity's Last Code Exam (HLCE), comprising 235 most challenging problems from the International Collegiate Programming Contest (ICPC World Finals) and the International Olympiad in Informatics (IOI) spanning 2010 - 2024. As part of HLCE, we design a harmonized online-offline sandbox that guarantees fully reproducible evaluation. Through our comprehensive evaluation, we observe that even the strongest reasoning LLMs: o4-mini(high) and Gemini-2.5 Pro, achieve pass@1 rates of only 15.9% and 11.4%, respectively. Meanwhile, we propose a novel "self-recognition" task to measure LLMs' awareness of their own capabilities. Results indicate that LLMs' self-recognition abilities are not proportionally correlated with their code generation performance. Finally, our empirical validation of test-time scaling laws reveals that current advanced LLMs have substantial room for improvement on complex programming tasks. We expect HLCE to become a milestone challenge for code generation and to catalyze advances in high-performance reasoning and human-AI collaborative programming. Our code and dataset are also public available(https://github.com/Humanity-s-Last-Code-Exam/HLCE). |
| title | Humanity's Last Code Exam: Can Advanced LLMs Conquer Human's Hardest Code Competition? |
| topic | Software Engineering Computation and Language |
| url | https://arxiv.org/abs/2506.12713 |