Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.06855 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912907459035136 |
|---|---|
| author | Lupidi, Alisia Gauri, Bhavul Foster, Thomas Simon Omari, Bassel Al Magka, Despoina Pepe, Alberto Audran-Reiss, Alexis Aghamelu, Muna Baldwin, Nicolas Cipolina-Kun, Lucia Gagnon-Audet, Jean-Christophe Leow, Chee Hau Lefdal, Sandra Mossalam, Hossam Moudgil, Abhinav Nazir, Saba Tewolde, Emanuel Urrego, Isabel Estape, Jordi Armengol Budhiraja, Amar Chaurasia, Gaurav Charnalia, Abhishek Dunfield, Derek Hambardzumyan, Karen Izcovich, Daniel Josifoski, Martin Mediratta, Ishita Niu, Kelvin Pathak, Parth Shvartsman, Michael Toledo, Edan Protopopov, Anton Raileanu, Roberta Miller, Alexander Shavrina, Tatiana Foerster, Jakob Bachrach, Yoram |
| author_facet | Lupidi, Alisia Gauri, Bhavul Foster, Thomas Simon Omari, Bassel Al Magka, Despoina Pepe, Alberto Audran-Reiss, Alexis Aghamelu, Muna Baldwin, Nicolas Cipolina-Kun, Lucia Gagnon-Audet, Jean-Christophe Leow, Chee Hau Lefdal, Sandra Mossalam, Hossam Moudgil, Abhinav Nazir, Saba Tewolde, Emanuel Urrego, Isabel Estape, Jordi Armengol Budhiraja, Amar Chaurasia, Gaurav Charnalia, Abhishek Dunfield, Derek Hambardzumyan, Karen Izcovich, Daniel Josifoski, Martin Mediratta, Ishita Niu, Kelvin Pathak, Parth Shvartsman, Michael Toledo, Edan Protopopov, Anton Raileanu, Roberta Miller, Alexander Shavrina, Tatiana Foerster, Jakob Bachrach, Yoram |
| contents | LLM agents hold significant promise for advancing scientific research. To accelerate this progress, we introduce AIRS-Bench (the AI Research Science Benchmark), a suite of 20 tasks sourced from state-of-the-art machine learning papers. These tasks span diverse domains, including language modeling, mathematics, bioinformatics, and time series forecasting. AIRS-Bench tasks assess agentic capabilities over the full research lifecycle -- including idea generation, experiment analysis and iterative refinement -- without providing baseline code. The AIRS-Bench task format is versatile, enabling easy integration of new tasks and rigorous comparison across different agentic frameworks. We establish baselines using frontier models paired with both sequential and parallel scaffolds. Our results show that agents exceed human SOTA in four tasks but fail to match it in sixteen others. Even when agents surpass human benchmarks, they do not reach the theoretical performance ceiling for the underlying tasks. These findings indicate that AIRS-Bench is far from saturated and offers substantial room for improvement. We open-source the AIRS-Bench task definitions and evaluation code to catalyze further development in autonomous scientific research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_06855 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents Lupidi, Alisia Gauri, Bhavul Foster, Thomas Simon Omari, Bassel Al Magka, Despoina Pepe, Alberto Audran-Reiss, Alexis Aghamelu, Muna Baldwin, Nicolas Cipolina-Kun, Lucia Gagnon-Audet, Jean-Christophe Leow, Chee Hau Lefdal, Sandra Mossalam, Hossam Moudgil, Abhinav Nazir, Saba Tewolde, Emanuel Urrego, Isabel Estape, Jordi Armengol Budhiraja, Amar Chaurasia, Gaurav Charnalia, Abhishek Dunfield, Derek Hambardzumyan, Karen Izcovich, Daniel Josifoski, Martin Mediratta, Ishita Niu, Kelvin Pathak, Parth Shvartsman, Michael Toledo, Edan Protopopov, Anton Raileanu, Roberta Miller, Alexander Shavrina, Tatiana Foerster, Jakob Bachrach, Yoram Artificial Intelligence LLM agents hold significant promise for advancing scientific research. To accelerate this progress, we introduce AIRS-Bench (the AI Research Science Benchmark), a suite of 20 tasks sourced from state-of-the-art machine learning papers. These tasks span diverse domains, including language modeling, mathematics, bioinformatics, and time series forecasting. AIRS-Bench tasks assess agentic capabilities over the full research lifecycle -- including idea generation, experiment analysis and iterative refinement -- without providing baseline code. The AIRS-Bench task format is versatile, enabling easy integration of new tasks and rigorous comparison across different agentic frameworks. We establish baselines using frontier models paired with both sequential and parallel scaffolds. Our results show that agents exceed human SOTA in four tasks but fail to match it in sixteen others. Even when agents surpass human benchmarks, they do not reach the theoretical performance ceiling for the underlying tasks. These findings indicate that AIRS-Bench is far from saturated and offers substantial room for improvement. We open-source the AIRS-Bench task definitions and evaluation code to catalyze further development in autonomous scientific research. |
| title | AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2602.06855 |