Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2512.15699 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918252810076160 |
|---|---|
| author | Mang, Qiuyang Chai, Wenhao Li, Zhifei Mao, Huanzhi Zhou, Shang Du, Alexander Li, Hanchen Liu, Shu Chen, Edwin Wang, Yichuan Chu, Xieting Cheng, Zerui Xu, Yuan Xia, Tian Wang, Zirui Shi, Tianneng Yao, Jianzhu Zhao, Yilong Zhang, Qizheng Ruan, Charlie Shen, Zeyu Liu, Kaiyuan He, Runyuan Xing, Dong Li, Zerui Zeng, Zirong Jiang, Yige Cheng, Lufeng Zhao, Ziyi Sun, Youran Zheng, Wesley Zhang, Meiyuwang Ji, Ruyi Tu, Xuechang Zheng, Zihan Chen, Zexing Zhou, Kangyang Wang, Zhaozi Chen, Jingbang Korolova, Aleksandra Henderson, Peter Viswanath, Pramod Ganesh, Vijay Xie, Saining Liu, Zhuang Song, Dawn Min, Sewon Stoica, Ion Gonzalez, Joseph E. Shang, Jingbo Cheung, Alvin |
| author_facet | Mang, Qiuyang Chai, Wenhao Li, Zhifei Mao, Huanzhi Zhou, Shang Du, Alexander Li, Hanchen Liu, Shu Chen, Edwin Wang, Yichuan Chu, Xieting Cheng, Zerui Xu, Yuan Xia, Tian Wang, Zirui Shi, Tianneng Yao, Jianzhu Zhao, Yilong Zhang, Qizheng Ruan, Charlie Shen, Zeyu Liu, Kaiyuan He, Runyuan Xing, Dong Li, Zerui Zeng, Zirong Jiang, Yige Cheng, Lufeng Zhao, Ziyi Sun, Youran Zheng, Wesley Zhang, Meiyuwang Ji, Ruyi Tu, Xuechang Zheng, Zihan Chen, Zexing Zhou, Kangyang Wang, Zhaozi Chen, Jingbang Korolova, Aleksandra Henderson, Peter Viswanath, Pramod Ganesh, Vijay Xie, Saining Liu, Zhuang Song, Dawn Min, Sewon Stoica, Ion Gonzalez, Joseph E. Shang, Jingbo Cheung, Alvin |
| contents | We introduce FrontierCS, a benchmark of 156 open-ended problems across diverse areas of computer science, designed and reviewed by experts, including CS PhDs and top-tier competitive programming participants and problem setters. Unlike existing benchmarks that focus on tasks with known optimal solutions, FrontierCS targets problems where the optimal solution is unknown, but the quality of a solution can be objectively evaluated. Models solve these tasks by implementing executable programs rather than outputting a direct answer. FrontierCS includes algorithmic problems, which are often NP-hard variants of competitive programming problems with objective partial scoring, and research problems with the same property. For each problem we provide an expert reference solution and an automatic evaluator. Combining open-ended design, measurable progress, and expert curation, FrontierCS provides a benchmark at the frontier of computer-science difficulty. Empirically, we find that frontier reasoning models still lag far behind human experts on both the algorithmic and research tracks, that increasing reasoning budgets alone does not close this gap, and that models often over-optimize for generating merely workable code instead of discovering high-quality algorithms and system designs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_15699 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FrontierCS: Evolving Challenges for Evolving Intelligence Mang, Qiuyang Chai, Wenhao Li, Zhifei Mao, Huanzhi Zhou, Shang Du, Alexander Li, Hanchen Liu, Shu Chen, Edwin Wang, Yichuan Chu, Xieting Cheng, Zerui Xu, Yuan Xia, Tian Wang, Zirui Shi, Tianneng Yao, Jianzhu Zhao, Yilong Zhang, Qizheng Ruan, Charlie Shen, Zeyu Liu, Kaiyuan He, Runyuan Xing, Dong Li, Zerui Zeng, Zirong Jiang, Yige Cheng, Lufeng Zhao, Ziyi Sun, Youran Zheng, Wesley Zhang, Meiyuwang Ji, Ruyi Tu, Xuechang Zheng, Zihan Chen, Zexing Zhou, Kangyang Wang, Zhaozi Chen, Jingbang Korolova, Aleksandra Henderson, Peter Viswanath, Pramod Ganesh, Vijay Xie, Saining Liu, Zhuang Song, Dawn Min, Sewon Stoica, Ion Gonzalez, Joseph E. Shang, Jingbo Cheung, Alvin Machine Learning Software Engineering We introduce FrontierCS, a benchmark of 156 open-ended problems across diverse areas of computer science, designed and reviewed by experts, including CS PhDs and top-tier competitive programming participants and problem setters. Unlike existing benchmarks that focus on tasks with known optimal solutions, FrontierCS targets problems where the optimal solution is unknown, but the quality of a solution can be objectively evaluated. Models solve these tasks by implementing executable programs rather than outputting a direct answer. FrontierCS includes algorithmic problems, which are often NP-hard variants of competitive programming problems with objective partial scoring, and research problems with the same property. For each problem we provide an expert reference solution and an automatic evaluator. Combining open-ended design, measurable progress, and expert curation, FrontierCS provides a benchmark at the frontier of computer-science difficulty. Empirically, we find that frontier reasoning models still lag far behind human experts on both the algorithmic and research tracks, that increasing reasoning budgets alone does not close this gap, and that models often over-optimize for generating merely workable code instead of discovering high-quality algorithms and system designs. |
| title | FrontierCS: Evolving Challenges for Evolving Intelligence |
| topic | Machine Learning Software Engineering |
| url | https://arxiv.org/abs/2512.15699 |