Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.24785 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916771648241664 |
|---|---|
| author | Kon, Patrick Tser Jern Liu, Jiachen Zhu, Xinyi Ding, Qiuyi Peng, Jingjia Xing, Jiarong Huang, Yibo Qiu, Yiming Srinivasa, Jayanth Lee, Myungjin Chowdhury, Mosharaf Zaharia, Matei Chen, Ang |
| author_facet | Kon, Patrick Tser Jern Liu, Jiachen Zhu, Xinyi Ding, Qiuyi Peng, Jingjia Xing, Jiarong Huang, Yibo Qiu, Yiming Srinivasa, Jayanth Lee, Myungjin Chowdhury, Mosharaf Zaharia, Matei Chen, Ang |
| contents | Automating AI research holds immense potential for accelerating scientific progress, yet current AI agents struggle with the complexities of rigorous, end-to-end experimentation. We introduce EXP-Bench, a novel benchmark designed to systematically evaluate AI agents on complete research experiments sourced from influential AI publications. Given a research question and incomplete starter code, EXP-Bench challenges AI agents to formulate hypotheses, design and implement experimental procedures, execute them, and analyze results. To enable the creation of such intricate and authentic tasks with high-fidelity, we design a semi-autonomous pipeline to extract and structure crucial experimental details from these research papers and their associated open-source code. With the pipeline, EXP-Bench curated 461 AI research tasks from 51 top-tier AI research papers. Evaluations of leading LLM-based agents, such as OpenHands and IterativeAgent on EXP-Bench demonstrate partial capabilities: while scores on individual experimental aspects such as design or implementation correctness occasionally reach 20-35%, the success rate for complete, executable experiments was a mere 0.5%. By identifying these bottlenecks and providing realistic step-by-step experiment procedures, EXP-Bench serves as a vital tool for future AI agents to improve their ability to conduct AI research experiments. EXP-Bench is open-sourced at https://github.com/Just-Curieous/Curie/tree/main/benchmark/exp_bench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_24785 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EXP-Bench: Can AI Conduct AI Research Experiments? Kon, Patrick Tser Jern Liu, Jiachen Zhu, Xinyi Ding, Qiuyi Peng, Jingjia Xing, Jiarong Huang, Yibo Qiu, Yiming Srinivasa, Jayanth Lee, Myungjin Chowdhury, Mosharaf Zaharia, Matei Chen, Ang Artificial Intelligence Automating AI research holds immense potential for accelerating scientific progress, yet current AI agents struggle with the complexities of rigorous, end-to-end experimentation. We introduce EXP-Bench, a novel benchmark designed to systematically evaluate AI agents on complete research experiments sourced from influential AI publications. Given a research question and incomplete starter code, EXP-Bench challenges AI agents to formulate hypotheses, design and implement experimental procedures, execute them, and analyze results. To enable the creation of such intricate and authentic tasks with high-fidelity, we design a semi-autonomous pipeline to extract and structure crucial experimental details from these research papers and their associated open-source code. With the pipeline, EXP-Bench curated 461 AI research tasks from 51 top-tier AI research papers. Evaluations of leading LLM-based agents, such as OpenHands and IterativeAgent on EXP-Bench demonstrate partial capabilities: while scores on individual experimental aspects such as design or implementation correctness occasionally reach 20-35%, the success rate for complete, executable experiments was a mere 0.5%. By identifying these bottlenecks and providing realistic step-by-step experiment procedures, EXP-Bench serves as a vital tool for future AI agents to improve their ability to conduct AI research experiments. EXP-Bench is open-sourced at https://github.com/Just-Curieous/Curie/tree/main/benchmark/exp_bench. |
| title | EXP-Bench: Can AI Conduct AI Research Experiments? |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2505.24785 |