Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2502.13059 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917927516635136 |
|---|---|
| author | Cheng, Xianfu Zhang, Wei Zhang, Shiwei Yang, Jian Guan, Xiangyuan Wu, Xianjie Li, Xiang Zhang, Ge Liu, Jiaheng Mai, Yuying Zeng, Yutao Wen, Zhoufutu Jin, Ke Wang, Baorui Zhou, Weixiao Lu, Yunhong Li, Tongliang Huang, Wenhao Li, Zhoujun |
| author_facet | Cheng, Xianfu Zhang, Wei Zhang, Shiwei Yang, Jian Guan, Xiangyuan Wu, Xianjie Li, Xiang Zhang, Ge Liu, Jiaheng Mai, Yuying Zeng, Yutao Wen, Zhoufutu Jin, Ke Wang, Baorui Zhou, Weixiao Lu, Yunhong Li, Tongliang Huang, Wenhao Li, Zhoujun |
| contents | The increasing application of multi-modal large language models (MLLMs) across various sectors have spotlighted the essence of their output reliability and accuracy, particularly their ability to produce content grounded in factual information (e.g. common and domain-specific knowledge). In this work, we introduce SimpleVQA, the first comprehensive multi-modal benchmark to evaluate the factuality ability of MLLMs to answer natural language short questions. SimpleVQA is characterized by six key features: it covers multiple tasks and multiple scenarios, ensures high quality and challenging queries, maintains static and timeless reference answers, and is straightforward to evaluate. Our approach involves categorizing visual question-answering items into 9 different tasks around objective events or common knowledge and situating these within 9 topics. Rigorous quality control processes are implemented to guarantee high-quality, concise, and clear answers, facilitating evaluation with minimal variance via an LLM-as-a-judge scoring system. Using SimpleVQA, we perform a comprehensive assessment of leading 18 MLLMs and 8 text-only LLMs, delving into their image comprehension and text generation abilities by identifying and analyzing error cases. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_13059 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Models Cheng, Xianfu Zhang, Wei Zhang, Shiwei Yang, Jian Guan, Xiangyuan Wu, Xianjie Li, Xiang Zhang, Ge Liu, Jiaheng Mai, Yuying Zeng, Yutao Wen, Zhoufutu Jin, Ke Wang, Baorui Zhou, Weixiao Lu, Yunhong Li, Tongliang Huang, Wenhao Li, Zhoujun Computation and Language The increasing application of multi-modal large language models (MLLMs) across various sectors have spotlighted the essence of their output reliability and accuracy, particularly their ability to produce content grounded in factual information (e.g. common and domain-specific knowledge). In this work, we introduce SimpleVQA, the first comprehensive multi-modal benchmark to evaluate the factuality ability of MLLMs to answer natural language short questions. SimpleVQA is characterized by six key features: it covers multiple tasks and multiple scenarios, ensures high quality and challenging queries, maintains static and timeless reference answers, and is straightforward to evaluate. Our approach involves categorizing visual question-answering items into 9 different tasks around objective events or common knowledge and situating these within 9 topics. Rigorous quality control processes are implemented to guarantee high-quality, concise, and clear answers, facilitating evaluation with minimal variance via an LLM-as-a-judge scoring system. Using SimpleVQA, we perform a comprehensive assessment of leading 18 MLLMs and 8 text-only LLMs, delving into their image comprehension and text generation abilities by identifying and analyzing error cases. |
| title | SimpleVQA: Multimodal Factuality Evaluation for Multimodal Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.13059 |