Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2506.09050 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918154987372544 |
|---|---|
| author | Imajuku, Yuki Horie, Kohki Iwata, Yoichi Aoki, Kensho Takahashi, Naohiro Akiba, Takuya |
| author_facet | Imajuku, Yuki Horie, Kohki Iwata, Yoichi Aoki, Kensho Takahashi, Naohiro Akiba, Takuya |
| contents | How well do AI systems perform in algorithm engineering for hard optimization problems in domains such as package-delivery routing, crew scheduling, factory production planning, and power-grid balancing? We introduce ALE-Bench, a new benchmark for evaluating AI systems on score-based algorithmic programming contests. Drawing on real tasks from the AtCoder Heuristic Contests, ALE-Bench presents optimization problems that are computationally hard and admit no known exact solution. Unlike short-duration, pass/fail coding benchmarks, ALE-Bench encourages iterative solution refinement over long time horizons. Our software framework supports interactive agent architectures that leverage test-run feedback and visualizations. Our evaluation of frontier LLMs revealed that while they demonstrate high performance on specific problems, a notable gap remains compared to humans in terms of consistency across problems and long-horizon problem-solving capabilities. This highlights the need for this benchmark to foster future AI advancements. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_09050 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineering Imajuku, Yuki Horie, Kohki Iwata, Yoichi Aoki, Kensho Takahashi, Naohiro Akiba, Takuya Artificial Intelligence How well do AI systems perform in algorithm engineering for hard optimization problems in domains such as package-delivery routing, crew scheduling, factory production planning, and power-grid balancing? We introduce ALE-Bench, a new benchmark for evaluating AI systems on score-based algorithmic programming contests. Drawing on real tasks from the AtCoder Heuristic Contests, ALE-Bench presents optimization problems that are computationally hard and admit no known exact solution. Unlike short-duration, pass/fail coding benchmarks, ALE-Bench encourages iterative solution refinement over long time horizons. Our software framework supports interactive agent architectures that leverage test-run feedback and visualizations. Our evaluation of frontier LLMs revealed that while they demonstrate high performance on specific problems, a notable gap remains compared to humans in terms of consistency across problems and long-horizon problem-solving capabilities. This highlights the need for this benchmark to foster future AI advancements. |
| title | ALE-Bench: A Benchmark for Long-Horizon Objective-Driven Algorithm Engineering |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2506.09050 |