Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910307590340608 |
|---|---|
| author | Wang, Xiyao Zhou, Yuhang Liu, Xiaoyu Lu, Hongjin Xu, Yuancheng He, Feihong Yoon, Jaehong Lu, Taixi Bertasius, Gedas Bansal, Mohit Yao, Huaxiu Huang, Furong |
| author_facet | Wang, Xiyao Zhou, Yuhang Liu, Xiaoyu Lu, Hongjin Xu, Yuancheng He, Feihong Yoon, Jaehong Lu, Taixi Bertasius, Gedas Bansal, Mohit Yao, Huaxiu Huang, Furong |
| contents | Multimodal Large Language Models (MLLMs) have demonstrated proficiency in handling a variety of visual-language tasks. However, current MLLM benchmarks are predominantly designed to evaluate reasoning based on static information about a single image, and the ability of modern MLLMs to extrapolate from image sequences, which is essential for understanding our ever-changing world, has been less investigated. To address this challenge, this paper introduces Mementos, a new benchmark designed to assess MLLMs' sequential image reasoning abilities. Mementos features 4,761 diverse image sequences with varying lengths. We also employ a GPT-4 assisted method to evaluate MLLM reasoning performance. Through a careful evaluation of nine recent MLLMs on Mementos, including GPT-4V and Gemini, we find that they struggle to accurately describe dynamic information about given image sequences, often leading to hallucinations/misrepresentations of objects and their corresponding behaviors. Our quantitative analysis and case studies identify three key factors impacting MLLMs' sequential image reasoning: the correlation between object and behavioral hallucinations, the influence of cooccurring behaviors, and the compounding impact of behavioral hallucinations. Our dataset is available at https://github.com/umd-huang-lab/Mementos. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_10529 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences Wang, Xiyao Zhou, Yuhang Liu, Xiaoyu Lu, Hongjin Xu, Yuancheng He, Feihong Yoon, Jaehong Lu, Taixi Bertasius, Gedas Bansal, Mohit Yao, Huaxiu Huang, Furong Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning Multimodal Large Language Models (MLLMs) have demonstrated proficiency in handling a variety of visual-language tasks. However, current MLLM benchmarks are predominantly designed to evaluate reasoning based on static information about a single image, and the ability of modern MLLMs to extrapolate from image sequences, which is essential for understanding our ever-changing world, has been less investigated. To address this challenge, this paper introduces Mementos, a new benchmark designed to assess MLLMs' sequential image reasoning abilities. Mementos features 4,761 diverse image sequences with varying lengths. We also employ a GPT-4 assisted method to evaluate MLLM reasoning performance. Through a careful evaluation of nine recent MLLMs on Mementos, including GPT-4V and Gemini, we find that they struggle to accurately describe dynamic information about given image sequences, often leading to hallucinations/misrepresentations of objects and their corresponding behaviors. Our quantitative analysis and case studies identify three key factors impacting MLLMs' sequential image reasoning: the correlation between object and behavioral hallucinations, the influence of cooccurring behaviors, and the compounding impact of behavioral hallucinations. Our dataset is available at https://github.com/umd-huang-lab/Mementos. |
| title | Mementos: A Comprehensive Benchmark for Multimodal Large Language Model Reasoning over Image Sequences |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2401.10529 |