II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910781602267136 |
|---|---|
| author | Liu, Ziqiang Fang, Feiteng Feng, Xi Du, Xinrun Zhang, Chenhao Wang, Zekun Bai, Yuelin Zhao, Qixuan Fan, Liyang Gan, Chengguang Lin, Hongquan Li, Jiaming Ni, Yuansheng Wu, Haihong Narsupalli, Yaswanth Zheng, Zhigang Li, Chengming Hu, Xiping Xu, Ruifeng Chen, Xiaojun Yang, Min Liu, Jiaheng Liu, Ruibo Huang, Wenhao Zhang, Ge Ni, Shiwen |
| author_facet | Liu, Ziqiang Fang, Feiteng Feng, Xi Du, Xinrun Zhang, Chenhao Wang, Zekun Bai, Yuelin Zhao, Qixuan Fan, Liyang Gan, Chengguang Lin, Hongquan Li, Jiaming Ni, Yuansheng Wu, Haihong Narsupalli, Yaswanth Zheng, Zhigang Li, Chengming Hu, Xiping Xu, Ruifeng Chen, Xiaojun Yang, Min Liu, Jiaheng Liu, Ruibo Huang, Wenhao Zhang, Ge Ni, Shiwen |
| contents | The rapid advancements in the development of multimodal large language models (MLLMs) have consistently led to new breakthroughs on various benchmarks. In response, numerous challenging and comprehensive benchmarks have been proposed to more accurately assess the capabilities of MLLMs. However, there is a dearth of exploration of the higher-order perceptual capabilities of MLLMs. To fill this gap, we propose the Image Implication understanding Benchmark, II-Bench, which aims to evaluate the model's higher-order perception of images. Through extensive experiments on II-Bench across multiple MLLMs, we have made significant findings. Initially, a substantial gap is observed between the performance of MLLMs and humans on II-Bench. The pinnacle accuracy of MLLMs attains 74.8%, whereas human accuracy averages 90%, peaking at an impressive 98%. Subsequently, MLLMs perform worse on abstract and complex images, suggesting limitations in their ability to understand high-level semantics and capture image details. Finally, it is observed that most models exhibit enhanced accuracy when image sentiment polarity hints are incorporated into the prompts. This observation underscores a notable deficiency in their inherent understanding of image sentiment. We believe that II-Bench will inspire the community to develop the next generation of MLLMs, advancing the journey towards expert artificial general intelligence (AGI). II-Bench is publicly available at https://huggingface.co/datasets/m-a-p/II-Bench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_05862 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models Liu, Ziqiang Fang, Feiteng Feng, Xi Du, Xinrun Zhang, Chenhao Wang, Zekun Bai, Yuelin Zhao, Qixuan Fan, Liyang Gan, Chengguang Lin, Hongquan Li, Jiaming Ni, Yuansheng Wu, Haihong Narsupalli, Yaswanth Zheng, Zhigang Li, Chengming Hu, Xiping Xu, Ruifeng Chen, Xiaojun Yang, Min Liu, Jiaheng Liu, Ruibo Huang, Wenhao Zhang, Ge Ni, Shiwen Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition The rapid advancements in the development of multimodal large language models (MLLMs) have consistently led to new breakthroughs on various benchmarks. In response, numerous challenging and comprehensive benchmarks have been proposed to more accurately assess the capabilities of MLLMs. However, there is a dearth of exploration of the higher-order perceptual capabilities of MLLMs. To fill this gap, we propose the Image Implication understanding Benchmark, II-Bench, which aims to evaluate the model's higher-order perception of images. Through extensive experiments on II-Bench across multiple MLLMs, we have made significant findings. Initially, a substantial gap is observed between the performance of MLLMs and humans on II-Bench. The pinnacle accuracy of MLLMs attains 74.8%, whereas human accuracy averages 90%, peaking at an impressive 98%. Subsequently, MLLMs perform worse on abstract and complex images, suggesting limitations in their ability to understand high-level semantics and capture image details. Finally, it is observed that most models exhibit enhanced accuracy when image sentiment polarity hints are incorporated into the prompts. This observation underscores a notable deficiency in their inherent understanding of image sentiment. We believe that II-Bench will inspire the community to develop the next generation of MLLMs, advancing the journey towards expert artificial general intelligence (AGI). II-Bench is publicly available at https://huggingface.co/datasets/m-a-p/II-Bench. |
| title | II-Bench: An Image Implication Understanding Benchmark for Multimodal Large Language Models |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.05862 |