Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.08807 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912144380919808 |
|---|---|
| author | Roberts, Jonathan Han, Kai Houlsby, Neil Albanie, Samuel |
| author_facet | Roberts, Jonathan Han, Kai Houlsby, Neil Albanie, Samuel |
| contents | Large multimodal models (LMMs) have proven flexible and generalisable across many tasks and fields. Although they have strong potential to aid scientific research, their capabilities in this domain are not well characterised. A key aspect of scientific research is the ability to understand and interpret figures, which serve as a rich, compressed source of complex information. In this work, we present SciFIBench, a scientific figure interpretation benchmark consisting of 2000 questions split between two tasks across 8 categories. The questions are curated from arXiv paper figures and captions, using adversarial filtering to find hard negatives and human verification for quality control. We evaluate 28 LMMs on SciFIBench, finding it to be a challenging benchmark. Finally, we investigate the alignment and reasoning faithfulness of the LMMs on augmented question sets from our benchmark. We release SciFIBench to encourage progress in this domain. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_08807 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation Roberts, Jonathan Han, Kai Houlsby, Neil Albanie, Samuel Computer Vision and Pattern Recognition Large multimodal models (LMMs) have proven flexible and generalisable across many tasks and fields. Although they have strong potential to aid scientific research, their capabilities in this domain are not well characterised. A key aspect of scientific research is the ability to understand and interpret figures, which serve as a rich, compressed source of complex information. In this work, we present SciFIBench, a scientific figure interpretation benchmark consisting of 2000 questions split between two tasks across 8 categories. The questions are curated from arXiv paper figures and captions, using adversarial filtering to find hard negatives and human verification for quality control. We evaluate 28 LMMs on SciFIBench, finding it to be a challenging benchmark. Finally, we investigate the alignment and reasoning faithfulness of the LMMs on augmented question sets from our benchmark. We release SciFIBench to encourage progress in this domain. |
| title | SciFIBench: Benchmarking Large Multimodal Models for Scientific Figure Interpretation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.08807 |