Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2407.04903 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915162183696384 |
|---|---|
| author | Li, Zekun Yang, Xianjun Choi, Kyuri Zhu, Wanrong Hsieh, Ryan Kim, HyeonJung Lim, Jin Hyuk Ji, Sungyoung Lee, Byungju Yan, Xifeng Petzold, Linda Ruth Wilson, Stephen D. Lim, Woosang Wang, William Yang |
| author_facet | Li, Zekun Yang, Xianjun Choi, Kyuri Zhu, Wanrong Hsieh, Ryan Kim, HyeonJung Lim, Jin Hyuk Ji, Sungyoung Lee, Byungju Yan, Xifeng Petzold, Linda Ruth Wilson, Stephen D. Lim, Woosang Wang, William Yang |
| contents | Scientific figure interpretation is a crucial capability for AI-driven scientific assistants built on advanced Large Vision Language Models. However, current datasets and benchmarks primarily focus on simple charts or other relatively straightforward figures from limited science domains. To address this gap, we present a comprehensive dataset compiled from peer-reviewed Nature Communications articles covering 72 scientific fields, encompassing complex visualizations such as schematic diagrams, microscopic images, and experimental data which require graduate-level expertise to interpret. We evaluated 19 proprietary and open-source models on two benchmark tasks, figure captioning and multiple-choice, and conducted human expert annotation. Our analysis revealed significant task challenges and performance gaps among models. Beyond serving as a benchmark, this dataset serves as a valuable resource for large-scale training. Fine-tuning Qwen2-VL-7B with our task-specific data achieved better performance than GPT-4o and even human experts in multiple-choice evaluations. Furthermore, continuous pre-training on our interleaved article and figure data substantially enhanced the model's downstream task performance in materials science. We have released our dataset to support further research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_04903 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding Li, Zekun Yang, Xianjun Choi, Kyuri Zhu, Wanrong Hsieh, Ryan Kim, HyeonJung Lim, Jin Hyuk Ji, Sungyoung Lee, Byungju Yan, Xifeng Petzold, Linda Ruth Wilson, Stephen D. Lim, Woosang Wang, William Yang Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Scientific figure interpretation is a crucial capability for AI-driven scientific assistants built on advanced Large Vision Language Models. However, current datasets and benchmarks primarily focus on simple charts or other relatively straightforward figures from limited science domains. To address this gap, we present a comprehensive dataset compiled from peer-reviewed Nature Communications articles covering 72 scientific fields, encompassing complex visualizations such as schematic diagrams, microscopic images, and experimental data which require graduate-level expertise to interpret. We evaluated 19 proprietary and open-source models on two benchmark tasks, figure captioning and multiple-choice, and conducted human expert annotation. Our analysis revealed significant task challenges and performance gaps among models. Beyond serving as a benchmark, this dataset serves as a valuable resource for large-scale training. Fine-tuning Qwen2-VL-7B with our task-specific data achieved better performance than GPT-4o and even human experts in multiple-choice evaluations. Furthermore, continuous pre-training on our interleaved article and figure data substantially enhanced the model's downstream task performance in materials science. We have released our dataset to support further research. |
| title | MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.04903 |