MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909237868756992 |
|---|---|
| author | Wang, Fei Fu, Xingyu Huang, James Y. Li, Zekun Liu, Qin Liu, Xiaogeng Ma, Mingyu Derek Xu, Nan Zhou, Wenxuan Zhang, Kai Yan, Tianyi Lorena Mo, Wenjie Jacky Liu, Hsiang-Hui Lu, Pan Li, Chunyuan Xiao, Chaowei Chang, Kai-Wei Roth, Dan Zhang, Sheng Poon, Hoifung Chen, Muhao |
| author_facet | Wang, Fei Fu, Xingyu Huang, James Y. Li, Zekun Liu, Qin Liu, Xiaogeng Ma, Mingyu Derek Xu, Nan Zhou, Wenxuan Zhang, Kai Yan, Tianyi Lorena Mo, Wenjie Jacky Liu, Hsiang-Hui Lu, Pan Li, Chunyuan Xiao, Chaowei Chang, Kai-Wei Roth, Dan Zhang, Sheng Poon, Hoifung Chen, Muhao |
| contents | We introduce MuirBench, a comprehensive benchmark that focuses on robust multi-image understanding capabilities of multimodal LLMs. MuirBench consists of 12 diverse multi-image tasks (e.g., scene understanding, ordering) that involve 10 categories of multi-image relations (e.g., multiview, temporal relations). Comprising 11,264 images and 2,600 multiple-choice questions, MuirBench is created in a pairwise manner, where each standard instance is paired with an unanswerable variant that has minimal semantic differences, in order for a reliable assessment. Evaluated upon 20 recent multi-modal LLMs, our results reveal that even the best-performing models like GPT-4o and Gemini Pro find it challenging to solve MuirBench, achieving 68.0% and 49.3% in accuracy. Open-source multimodal LLMs trained on single images can hardly generalize to multi-image questions, hovering below 33.3% in accuracy. These results highlight the importance of MuirBench in encouraging the community to develop multimodal LLMs that can look beyond a single image, suggesting potential pathways for future improvements. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_09411 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding Wang, Fei Fu, Xingyu Huang, James Y. Li, Zekun Liu, Qin Liu, Xiaogeng Ma, Mingyu Derek Xu, Nan Zhou, Wenxuan Zhang, Kai Yan, Tianyi Lorena Mo, Wenjie Jacky Liu, Hsiang-Hui Lu, Pan Li, Chunyuan Xiao, Chaowei Chang, Kai-Wei Roth, Dan Zhang, Sheng Poon, Hoifung Chen, Muhao Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language We introduce MuirBench, a comprehensive benchmark that focuses on robust multi-image understanding capabilities of multimodal LLMs. MuirBench consists of 12 diverse multi-image tasks (e.g., scene understanding, ordering) that involve 10 categories of multi-image relations (e.g., multiview, temporal relations). Comprising 11,264 images and 2,600 multiple-choice questions, MuirBench is created in a pairwise manner, where each standard instance is paired with an unanswerable variant that has minimal semantic differences, in order for a reliable assessment. Evaluated upon 20 recent multi-modal LLMs, our results reveal that even the best-performing models like GPT-4o and Gemini Pro find it challenging to solve MuirBench, achieving 68.0% and 49.3% in accuracy. Open-source multimodal LLMs trained on single images can hardly generalize to multi-image questions, hovering below 33.3% in accuracy. These results highlight the importance of MuirBench in encouraging the community to develop multimodal LLMs that can look beyond a single image, suggesting potential pathways for future improvements. |
| title | MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2406.09411 |