ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908978797084672 |
|---|---|
| author | Sun, Qiushi Liu, Zhoumianze Ma, Chang Ding, Zichen Xu, Fangzhi Yin, Zhangyue Zhao, Haiteng Wu, Zhenyu Cheng, Kanzhi Liu, Zhaoyang Wang, Jianing Li, Qintong Tang, Xiangru Xie, Tianbao Feng, Xiachong Li, Xiang Kao, Ben Wang, Wenhai Qi, Biqing Kong, Lingpeng Wu, Zhiyong |
| author_facet | Sun, Qiushi Liu, Zhoumianze Ma, Chang Ding, Zichen Xu, Fangzhi Yin, Zhangyue Zhao, Haiteng Wu, Zhenyu Cheng, Kanzhi Liu, Zhaoyang Wang, Jianing Li, Qintong Tang, Xiangru Xie, Tianbao Feng, Xiachong Li, Xiang Kao, Ben Wang, Wenhai Qi, Biqing Kong, Lingpeng Wu, Zhiyong |
| contents | Large Language Models (LLMs) have extended their impact beyond Natural Language Processing, substantially fostering the development of interdisciplinary research. Recently, various LLM-based agents have been developed to assist scientific discovery progress across multiple aspects and domains. Among these, computer-using agents, capable of interacting with operating systems as humans do, are paving the way to automated scientific problem-solving and addressing routines in researchers' workflows. Recognizing the transformative potential of these agents, we introduce ScienceBoard, which encompasses two complementary contributions: (i) a realistic, multi-domain environment featuring dynamic and visually rich scientific workflows with integrated professional software, where agents can autonomously interact via different interfaces to accelerate complex research tasks and experiments; and (ii) a challenging benchmark of 169 high-quality, rigorously validated real-world tasks curated by humans, spanning scientific-discovery workflows in domains such as biochemistry, astronomy, and geoinformatics. Extensive evaluations of agents with state-of-the-art backbones (e.g., GPT-4o, Claude 3.7, UI-TARS) show that, despite some promising results, they still fall short of reliably assisting scientists in complex workflows, achieving only a 15% overall success rate. In-depth analysis further provides valuable insights for addressing current agent limitations and more effective design principles, paving the way to build more capable agents for scientific discovery. Our code, environment, and benchmark are at https://qiushisun.github.io/ScienceBoard-Home/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_19897 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows Sun, Qiushi Liu, Zhoumianze Ma, Chang Ding, Zichen Xu, Fangzhi Yin, Zhangyue Zhao, Haiteng Wu, Zhenyu Cheng, Kanzhi Liu, Zhaoyang Wang, Jianing Li, Qintong Tang, Xiangru Xie, Tianbao Feng, Xiachong Li, Xiang Kao, Ben Wang, Wenhai Qi, Biqing Kong, Lingpeng Wu, Zhiyong Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Human-Computer Interaction Large Language Models (LLMs) have extended their impact beyond Natural Language Processing, substantially fostering the development of interdisciplinary research. Recently, various LLM-based agents have been developed to assist scientific discovery progress across multiple aspects and domains. Among these, computer-using agents, capable of interacting with operating systems as humans do, are paving the way to automated scientific problem-solving and addressing routines in researchers' workflows. Recognizing the transformative potential of these agents, we introduce ScienceBoard, which encompasses two complementary contributions: (i) a realistic, multi-domain environment featuring dynamic and visually rich scientific workflows with integrated professional software, where agents can autonomously interact via different interfaces to accelerate complex research tasks and experiments; and (ii) a challenging benchmark of 169 high-quality, rigorously validated real-world tasks curated by humans, spanning scientific-discovery workflows in domains such as biochemistry, astronomy, and geoinformatics. Extensive evaluations of agents with state-of-the-art backbones (e.g., GPT-4o, Claude 3.7, UI-TARS) show that, despite some promising results, they still fall short of reliably assisting scientists in complex workflows, achieving only a 15% overall success rate. In-depth analysis further provides valuable insights for addressing current agent limitations and more effective design principles, paving the way to build more capable agents for scientific discovery. Our code, environment, and benchmark are at https://qiushisun.github.io/ScienceBoard-Home/. |
| title | ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflows |
| topic | Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Human-Computer Interaction |
| url | https://arxiv.org/abs/2505.19897 |