Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.11606 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918159638855680 |
|---|---|
| author | Xu, Yicheng Wu, Yue Yu, Jiashuo Yan, Ziang Jiang, Tianxiang He, Yinan Zhao, Qingsong Chen, Kai Qiao, Yu Wang, Limin Okumura, Manabu Wang, Yi |
| author_facet | Xu, Yicheng Wu, Yue Yu, Jiashuo Yan, Ziang Jiang, Tianxiang He, Yinan Zhao, Qingsong Chen, Kai Qiao, Yu Wang, Limin Okumura, Manabu Wang, Yi |
| contents | Multimodal Large Language Models (MLLMs) hold promise for accelerating scientific discovery by interpreting complex experimental procedures. However, their true capabilities are poorly understood, as existing benchmarks neglect the fine-grained and long-horizon nature of authentic laboratory work, especially in wet-lab settings. To bridge this gap, we introduce ExpVid, the first benchmark designed to systematically evaluate MLLMs on scientific experiment videos. Curated from peer-reviewed video publications, ExpVid features a new three-level task hierarchy that mirrors the scientific process: (1) Fine-grained Perception of tools, materials, and actions; (2) Procedural Understanding of step order and completeness; and (3) Scientific Reasoning that connects the full experiment to its published conclusions. Our vision-centric annotation pipeline, combining automated generation with multi-disciplinary expert validation, ensures that tasks require visual grounding. We evaluate 19 leading MLLMs on ExpVid and find that while they excel at coarse-grained recognition, they struggle with disambiguating fine details, tracking state changes over time, and linking experimental procedures to scientific outcomes. Our results reveal a notable performance gap between proprietary and open-source models, particularly in high-order reasoning. ExpVid not only provides a diagnostic tool but also charts a roadmap for developing MLLMs capable of becoming trustworthy partners in scientific experimentation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_11606 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ExpVid: A Benchmark for Experiment Video Understanding & Reasoning Xu, Yicheng Wu, Yue Yu, Jiashuo Yan, Ziang Jiang, Tianxiang He, Yinan Zhao, Qingsong Chen, Kai Qiao, Yu Wang, Limin Okumura, Manabu Wang, Yi Computer Vision and Pattern Recognition Multimodal Large Language Models (MLLMs) hold promise for accelerating scientific discovery by interpreting complex experimental procedures. However, their true capabilities are poorly understood, as existing benchmarks neglect the fine-grained and long-horizon nature of authentic laboratory work, especially in wet-lab settings. To bridge this gap, we introduce ExpVid, the first benchmark designed to systematically evaluate MLLMs on scientific experiment videos. Curated from peer-reviewed video publications, ExpVid features a new three-level task hierarchy that mirrors the scientific process: (1) Fine-grained Perception of tools, materials, and actions; (2) Procedural Understanding of step order and completeness; and (3) Scientific Reasoning that connects the full experiment to its published conclusions. Our vision-centric annotation pipeline, combining automated generation with multi-disciplinary expert validation, ensures that tasks require visual grounding. We evaluate 19 leading MLLMs on ExpVid and find that while they excel at coarse-grained recognition, they struggle with disambiguating fine details, tracking state changes over time, and linking experimental procedures to scientific outcomes. Our results reveal a notable performance gap between proprietary and open-source models, particularly in high-order reasoning. ExpVid not only provides a diagnostic tool but also charts a roadmap for developing MLLMs capable of becoming trustworthy partners in scientific experimentation. |
| title | ExpVid: A Benchmark for Experiment Video Understanding & Reasoning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2510.11606 |