OIBench: Benchmarking Strong Reasoning Models with Olympiad in Informatics
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918056736849920 |
|---|---|
| author | Zhu, Yaoming Wang, Junxin Li, Yiyang Qiu, Lin Wang, ZongYu Xu, Jun Cao, Xuezhi Wei, Yuhuai Wang, Mingshi Cai, Xunliang Ma, Rong |
| author_facet | Zhu, Yaoming Wang, Junxin Li, Yiyang Qiu, Lin Wang, ZongYu Xu, Jun Cao, Xuezhi Wei, Yuhuai Wang, Mingshi Cai, Xunliang Ma, Rong |
| contents | As models become increasingly sophisticated, conventional algorithm benchmarks are increasingly saturated, underscoring the need for more challenging benchmarks to guide future improvements in algorithmic reasoning. This paper introduces OIBench, a high-quality, private, and challenging olympiad-level informatics dataset comprising 250 carefully curated original problems. We detail the construction methodology of the benchmark, ensuring a comprehensive assessment across various programming paradigms and complexities, and we demonstrate its contamination-resistant properties via experiments. We propose Time/Space Completion Curves for finer-grained efficiency analysis and enable direct human-model comparisons through high-level participant evaluations. Our experiments reveal that while open-source models lag behind closed-source counterparts, current SOTA models already outperform most human participants in both correctness and efficiency, while still being suboptimal compared to the canonical solutions. By releasing OIBench as a fully open-source resource (https://huggingface.co/datasets/AGI-Eval/OIBench), we hope this benchmark will contribute to advancing code reasoning capabilities for future LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_10481 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | OIBench: Benchmarking Strong Reasoning Models with Olympiad in Informatics Zhu, Yaoming Wang, Junxin Li, Yiyang Qiu, Lin Wang, ZongYu Xu, Jun Cao, Xuezhi Wei, Yuhuai Wang, Mingshi Cai, Xunliang Ma, Rong Artificial Intelligence As models become increasingly sophisticated, conventional algorithm benchmarks are increasingly saturated, underscoring the need for more challenging benchmarks to guide future improvements in algorithmic reasoning. This paper introduces OIBench, a high-quality, private, and challenging olympiad-level informatics dataset comprising 250 carefully curated original problems. We detail the construction methodology of the benchmark, ensuring a comprehensive assessment across various programming paradigms and complexities, and we demonstrate its contamination-resistant properties via experiments. We propose Time/Space Completion Curves for finer-grained efficiency analysis and enable direct human-model comparisons through high-level participant evaluations. Our experiments reveal that while open-source models lag behind closed-source counterparts, current SOTA models already outperform most human participants in both correctness and efficiency, while still being suboptimal compared to the canonical solutions. By releasing OIBench as a fully open-source resource (https://huggingface.co/datasets/AGI-Eval/OIBench), we hope this benchmark will contribute to advancing code reasoning capabilities for future LLMs. |
| title | OIBench: Benchmarking Strong Reasoning Models with Olympiad in Informatics |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2506.10481 |