FML-bench: Benchmarking Machine Learning Agents for Scientific Research
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911466655842304 |
|---|---|
| author | Zou, Qiran Lam, Hou Hei Zhao, Wenhao Tang, Yiming Chen, Tingting Yu, Samson Zhang, Tianyi Liu, Chang Ji, Xiangyang Liu, Dianbo |
| author_facet | Zou, Qiran Lam, Hou Hei Zhao, Wenhao Tang, Yiming Chen, Tingting Yu, Samson Zhang, Tianyi Liu, Chang Ji, Xiangyang Liu, Dianbo |
| contents | Large language models (LLMs) have sparked growing interest in machine learning research agents that can autonomously propose ideas and conduct experiments. However, existing benchmarks predominantly adopt an engineering-oriented perspective: they emphasize application-oriented tasks and evaluate primarily on final performance and computational cost, overlooking agents' research processes and limiting assessment of their capabilities in scientific research settings. To more comprehensively evaluate agents in scientific research settings, we introduce FML-bench, a benchmark comprising 8 diverse and fundamental ML research tasks, and further propose complementary metrics, notably Exploration Diversity, which quantifies the variance of proposals across iterations and reveals how exploration patterns influence research outcomes. We evaluate state-of-the-art research agents on FML-bench, showing that agents employing broad exploration strategies exhibit higher exploration diversity and achieve superior performance, and that exploration diversity positively correlates with performance improvements across multiple tasks. We hope these findings and our benchmark inform future agent design and support the community in further investigating agent behavior. Our benchmark is available at https://github.com/qrzou/FML-bench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_10472 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FML-bench: Benchmarking Machine Learning Agents for Scientific Research Zou, Qiran Lam, Hou Hei Zhao, Wenhao Tang, Yiming Chen, Tingting Yu, Samson Zhang, Tianyi Liu, Chang Ji, Xiangyang Liu, Dianbo Computation and Language Artificial Intelligence Large language models (LLMs) have sparked growing interest in machine learning research agents that can autonomously propose ideas and conduct experiments. However, existing benchmarks predominantly adopt an engineering-oriented perspective: they emphasize application-oriented tasks and evaluate primarily on final performance and computational cost, overlooking agents' research processes and limiting assessment of their capabilities in scientific research settings. To more comprehensively evaluate agents in scientific research settings, we introduce FML-bench, a benchmark comprising 8 diverse and fundamental ML research tasks, and further propose complementary metrics, notably Exploration Diversity, which quantifies the variance of proposals across iterations and reveals how exploration patterns influence research outcomes. We evaluate state-of-the-art research agents on FML-bench, showing that agents employing broad exploration strategies exhibit higher exploration diversity and achieve superior performance, and that exploration diversity positively correlates with performance improvements across multiple tasks. We hope these findings and our benchmark inform future agent design and support the community in further investigating agent behavior. Our benchmark is available at https://github.com/qrzou/FML-bench. |
| title | FML-bench: Benchmarking Machine Learning Agents for Scientific Research |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.10472 |