Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2602.02905 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914303588696064 |
|---|---|
| author | Wang, Zhen Bai, Fan Luo, Zhongyan Su, Jinyan Sun, Kaiser Yu, Xinle Liu, Jieyuan Zhou, Kun Cardie, Claire Dredze, Mark Xing, Eric P. Hu, Zhiting |
| author_facet | Wang, Zhen Bai, Fan Luo, Zhongyan Su, Jinyan Sun, Kaiser Yu, Xinle Liu, Jieyuan Zhou, Kun Cardie, Claire Dredze, Mark Xing, Eric P. Hu, Zhiting |
| contents | Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge. Existing benchmarks face a trade-off: they either heavily rely on LLM-as-judge evaluations of automatically generated research outputs or optimize convenient yet isolated performance metrics that provide coarse proxies for scientific insight. To address this gap, we introduce FIRE-Bench (Full-cycle Insight Rediscovery Evaluation), a benchmark that evaluates agents through the rediscovery of established findings from recent, high-impact machine learning research. Agents are given only a high-level research question extracted from a published, verified study and must autonomously explore ideas, design experiments, implement code, execute their plans, and derive conclusions supported by empirical evidence. We evaluate a range of state-of-the-art agents with frontier LLMs backbones like gpt-5 on FIRE-Bench. Our results show that full-cycle scientific research remains challenging for current agent systems: even the strongest agents achieve limited rediscovery success (<50 F1), exhibit high variance across runs, and display recurring failure modes in experimental design, execution, and evidence-based reasoning. FIRE-Bench provides a rigorous and diagnostic framework for measuring progress toward reliable agent-driven scientific discovery. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_02905 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights Wang, Zhen Bai, Fan Luo, Zhongyan Su, Jinyan Sun, Kaiser Yu, Xinle Liu, Jieyuan Zhou, Kun Cardie, Claire Dredze, Mark Xing, Eric P. Hu, Zhiting Artificial Intelligence Autonomous agents powered by large language models (LLMs) promise to accelerate scientific discovery end-to-end, but rigorously evaluating their capacity for verifiable discovery remains a central challenge. Existing benchmarks face a trade-off: they either heavily rely on LLM-as-judge evaluations of automatically generated research outputs or optimize convenient yet isolated performance metrics that provide coarse proxies for scientific insight. To address this gap, we introduce FIRE-Bench (Full-cycle Insight Rediscovery Evaluation), a benchmark that evaluates agents through the rediscovery of established findings from recent, high-impact machine learning research. Agents are given only a high-level research question extracted from a published, verified study and must autonomously explore ideas, design experiments, implement code, execute their plans, and derive conclusions supported by empirical evidence. We evaluate a range of state-of-the-art agents with frontier LLMs backbones like gpt-5 on FIRE-Bench. Our results show that full-cycle scientific research remains challenging for current agent systems: even the strongest agents achieve limited rediscovery success (<50 F1), exhibit high variance across runs, and display recurring failure modes in experimental design, execution, and evidence-based reasoning. FIRE-Bench provides a rigorous and diagnostic framework for measuring progress toward reliable agent-driven scientific discovery. |
| title | FIRE-Bench: Evaluating Agents on the Rediscovery of Scientific Insights |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2602.02905 |