Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.16402 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908499244482560 |
|---|---|
| author | Wang, Zihan Chen, Jiaze Liu, Zhicheng Mak, Markus Du, Yidi Moon, Geonsik Xu, Luoqi Tua, Aaron Peng, Kunshuo Lu, Jiayi Xia, Mingfei Zou, Boqian Ran, Chenyang Tian, Guang Zhu, Shoutai Duan, Yeheng Kang, Zhenghui Lin, Zhenxing Li, Shangshu Luo, Qiang Long, Qingshen Chen, Zhiyong Xiao, Yihan Wu, Yurong Zan, Daoguang Fu, Yuyi Wang, Mingxuan Ding, Ming |
| author_facet | Wang, Zihan Chen, Jiaze Liu, Zhicheng Mak, Markus Du, Yidi Moon, Geonsik Xu, Luoqi Tua, Aaron Peng, Kunshuo Lu, Jiayi Xia, Mingfei Zou, Boqian Ran, Chenyang Tian, Guang Zhu, Shoutai Duan, Yeheng Kang, Zhenghui Lin, Zhenxing Li, Shangshu Luo, Qiang Long, Qingshen Chen, Zhiyong Xiao, Yihan Wu, Yurong Zan, Daoguang Fu, Yuyi Wang, Mingxuan Ding, Ming |
| contents | Competitive programming has emerged as a critical benchmark for evaluating the reasoning and coding capabilities of Large Language Models (LLMs). Despite impressive progress on existing benchmarks, we argue that current evaluations overstate model proficiency, masking a substantial gap between LLMs and elite human programmers. This gap arises from two key limitations: insufficient difficulty and scope of benchmark problems, and evaluation bias from low-quality test cases. To address these shortcomings, we present AetherCode, a new benchmark that draws problems from premier programming competitions such as IOI and ICPC, offering broader coverage and higher difficulty. AetherCode further incorporates comprehensive, expert-validated test suites built through a hybrid of automated generation and human curation, ensuring rigorous and reliable assessment. By combining challenging problem design with robust evaluation, AetherCode provides a more faithful measure of LLM capabilities and sets a new standard for future research in code reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_16402 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AetherCode: Evaluating LLMs' Ability to Win In Premier Programming Competitions Wang, Zihan Chen, Jiaze Liu, Zhicheng Mak, Markus Du, Yidi Moon, Geonsik Xu, Luoqi Tua, Aaron Peng, Kunshuo Lu, Jiayi Xia, Mingfei Zou, Boqian Ran, Chenyang Tian, Guang Zhu, Shoutai Duan, Yeheng Kang, Zhenghui Lin, Zhenxing Li, Shangshu Luo, Qiang Long, Qingshen Chen, Zhiyong Xiao, Yihan Wu, Yurong Zan, Daoguang Fu, Yuyi Wang, Mingxuan Ding, Ming Software Engineering Computation and Language Competitive programming has emerged as a critical benchmark for evaluating the reasoning and coding capabilities of Large Language Models (LLMs). Despite impressive progress on existing benchmarks, we argue that current evaluations overstate model proficiency, masking a substantial gap between LLMs and elite human programmers. This gap arises from two key limitations: insufficient difficulty and scope of benchmark problems, and evaluation bias from low-quality test cases. To address these shortcomings, we present AetherCode, a new benchmark that draws problems from premier programming competitions such as IOI and ICPC, offering broader coverage and higher difficulty. AetherCode further incorporates comprehensive, expert-validated test suites built through a hybrid of automated generation and human curation, ensuring rigorous and reliable assessment. By combining challenging problem design with robust evaluation, AetherCode provides a more faithful measure of LLM capabilities and sets a new standard for future research in code reasoning. |
| title | AetherCode: Evaluating LLMs' Ability to Win In Premier Programming Competitions |
| topic | Software Engineering Computation and Language |
| url | https://arxiv.org/abs/2508.16402 |