AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914110737743872 |
|---|---|
| author | Press, Ori Amos, Brandon Zhao, Haoyu Wu, Yikai Ainsworth, Samuel K. Krupke, Dominik Kidger, Patrick Sajed, Touqir Stellato, Bartolomeo Park, Jisun Bosch, Nathanael Meril, Eli Steppi, Albert Zharmagambetov, Arman Zhang, Fangzhao Perez-Pineiro, David Mercurio, Alberto Zhan, Ni Abramovich, Talor Lieret, Kilian Zhang, Hanlin Huang, Shirley Bethge, Matthias Press, Ofir |
| author_facet | Press, Ori Amos, Brandon Zhao, Haoyu Wu, Yikai Ainsworth, Samuel K. Krupke, Dominik Kidger, Patrick Sajed, Touqir Stellato, Bartolomeo Park, Jisun Bosch, Nathanael Meril, Eli Steppi, Albert Zharmagambetov, Arman Zhang, Fangzhao Perez-Pineiro, David Mercurio, Alberto Zhan, Ni Abramovich, Talor Lieret, Kilian Zhang, Hanlin Huang, Shirley Bethge, Matthias Press, Ofir |
| contents | Despite progress in language model (LM) capabilities, evaluations have thus far focused on models' performance on tasks that humans have previously solved, including in programming (Jimenez et al., 2024) and mathematics (Glazer et al., 2024). We therefore propose testing models' ability to design and implement algorithms in an open-ended benchmark: We task LMs with writing code that efficiently solves computationally challenging problems in computer science, physics, and mathematics. Our AlgoTune benchmark consists of 154 coding tasks collected from domain experts and a framework for validating and timing LM-synthesized solution code, which is compared to reference implementations from popular open-source packages. In addition, we develop a baseline LM agent, AlgoTuner, and evaluate its performance across a suite of frontier models. AlgoTuner uses a simple, budgeted loop that edits code, compiles and runs it, profiles performance, verifies correctness on tests, and selects the fastest valid version. AlgoTuner achieves an average 1.72x speedup against our reference solvers, which use libraries such as SciPy, sk-learn and CVXPY. However, we find that current models fail to discover algorithmic innovations, instead preferring surface-level optimizations. We hope that AlgoTune catalyzes the development of LM agents exhibiting creative problem solving beyond state-of-the-art human performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_15887 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs? Press, Ori Amos, Brandon Zhao, Haoyu Wu, Yikai Ainsworth, Samuel K. Krupke, Dominik Kidger, Patrick Sajed, Touqir Stellato, Bartolomeo Park, Jisun Bosch, Nathanael Meril, Eli Steppi, Albert Zharmagambetov, Arman Zhang, Fangzhao Perez-Pineiro, David Mercurio, Alberto Zhan, Ni Abramovich, Talor Lieret, Kilian Zhang, Hanlin Huang, Shirley Bethge, Matthias Press, Ofir Software Engineering Artificial Intelligence Computation and Language Machine Learning Despite progress in language model (LM) capabilities, evaluations have thus far focused on models' performance on tasks that humans have previously solved, including in programming (Jimenez et al., 2024) and mathematics (Glazer et al., 2024). We therefore propose testing models' ability to design and implement algorithms in an open-ended benchmark: We task LMs with writing code that efficiently solves computationally challenging problems in computer science, physics, and mathematics. Our AlgoTune benchmark consists of 154 coding tasks collected from domain experts and a framework for validating and timing LM-synthesized solution code, which is compared to reference implementations from popular open-source packages. In addition, we develop a baseline LM agent, AlgoTuner, and evaluate its performance across a suite of frontier models. AlgoTuner uses a simple, budgeted loop that edits code, compiles and runs it, profiles performance, verifies correctness on tests, and selects the fastest valid version. AlgoTuner achieves an average 1.72x speedup against our reference solvers, which use libraries such as SciPy, sk-learn and CVXPY. However, we find that current models fail to discover algorithmic innovations, instead preferring surface-level optimizations. We hope that AlgoTune catalyzes the development of LM agents exhibiting creative problem solving beyond state-of-the-art human performance. |
| title | AlgoTune: Can Language Models Speed Up General-Purpose Numerical Programs? |
| topic | Software Engineering Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2507.15887 |