COMPOSITE-Stem
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911600316776448 |
|---|---|
| author | Waters, Kyle Nuzzi, Lucas Looram, Tadhg Tomasiello, Alessandro Kamdoum, Ariel Ghislain Kemogne Li, Bikun Sileo, Damien Kretov, Egor Fournier-Facio, Francesco Soloupis, Georgios Kassahun, Haile Wolff, Hew Cai, Jiaqi Li, Lianghui Roth, Marc Naiya, Mohinder Guo, Naixu Tang, Qicheng Wheeler, Richard Sala, Samuele Popov, Serguei Dillmann, Steven Li, Yuqi |
| author_facet | Waters, Kyle Nuzzi, Lucas Looram, Tadhg Tomasiello, Alessandro Kamdoum, Ariel Ghislain Kemogne Li, Bikun Sileo, Damien Kretov, Egor Fournier-Facio, Francesco Soloupis, Georgios Kassahun, Haile Wolff, Hew Cai, Jiaqi Li, Lianghui Roth, Marc Naiya, Mohinder Guo, Naixu Tang, Qicheng Wheeler, Richard Sala, Samuele Popov, Serguei Dillmann, Steven Li, Yuqi |
| contents | AI agents hold growing promise for accelerating scientific discovery; yet, a lack of frontier evaluations hinders adoption into real workflows. Expert-written benchmarks have proven effective at measuring AI reasoning, but most at this stage have become saturated and only measure performance on constrained outputs. To help address this gap, we introduce COMPOSITE-STEM, a benchmark of 70 expert-written tasks in physics, biology, chemistry, and mathematics, curated by doctoral-level researchers. Our benchmark combines exact-match grading and criterion-based rubrics with an LLM-as-a-jury grading protocol, allowing more flexible assessment of scientifically meaningful outputs. Using an adapted multimodal Terminus-2 agent harness within the Harbor agentic evaluation framework, we evaluate four frontier models. The top-performing model achieves 21%, demonstrating that COMPOSITE-STEM captures capabilities beyond current agent reach. All tasks are open-sourced with contributor permission to support reproducibility and to promote additional research towards AI's acceleration of scientific progress in these domains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_09836 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | COMPOSITE-Stem Waters, Kyle Nuzzi, Lucas Looram, Tadhg Tomasiello, Alessandro Kamdoum, Ariel Ghislain Kemogne Li, Bikun Sileo, Damien Kretov, Egor Fournier-Facio, Francesco Soloupis, Georgios Kassahun, Haile Wolff, Hew Cai, Jiaqi Li, Lianghui Roth, Marc Naiya, Mohinder Guo, Naixu Tang, Qicheng Wheeler, Richard Sala, Samuele Popov, Serguei Dillmann, Steven Li, Yuqi Artificial Intelligence Computation and Language Machine Learning AI agents hold growing promise for accelerating scientific discovery; yet, a lack of frontier evaluations hinders adoption into real workflows. Expert-written benchmarks have proven effective at measuring AI reasoning, but most at this stage have become saturated and only measure performance on constrained outputs. To help address this gap, we introduce COMPOSITE-STEM, a benchmark of 70 expert-written tasks in physics, biology, chemistry, and mathematics, curated by doctoral-level researchers. Our benchmark combines exact-match grading and criterion-based rubrics with an LLM-as-a-jury grading protocol, allowing more flexible assessment of scientifically meaningful outputs. Using an adapted multimodal Terminus-2 agent harness within the Harbor agentic evaluation framework, we evaluate four frontier models. The top-performing model achieves 21%, demonstrating that COMPOSITE-STEM captures capabilities beyond current agent reach. All tasks are open-sourced with contributor permission to support reproducibility and to promote additional research towards AI's acceleration of scientific progress in these domains. |
| title | COMPOSITE-Stem |
| topic | Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2604.09836 |