"See the World, Discover Knowledge": A Chinese Factuality Evaluation for Large Vision Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908384602619904 |
|---|---|
| author | Gu, Jihao Wang, Yingyao Bu, Pi Wang, Chen Wang, Ziming Song, Tengtao Wei, Donglai Yuan, Jiale Zhao, Yingxiu He, Yancheng Li, Shilong Liu, Jiaheng Cao, Meng Song, Jun Tan, Yingshui Li, Xiang Su, Wenbo Zheng, Zhicheng Zhu, Xiaoyong Zheng, Bo |
| author_facet | Gu, Jihao Wang, Yingyao Bu, Pi Wang, Chen Wang, Ziming Song, Tengtao Wei, Donglai Yuan, Jiale Zhao, Yingxiu He, Yancheng Li, Shilong Liu, Jiaheng Cao, Meng Song, Jun Tan, Yingshui Li, Xiang Su, Wenbo Zheng, Zhicheng Zhu, Xiaoyong Zheng, Bo |
| contents | The evaluation of factual accuracy in large vision language models (LVLMs) has lagged behind their rapid development, making it challenging to fully reflect these models' knowledge capacity and reliability. In this paper, we introduce the first factuality-based visual question-answering benchmark in Chinese, named ChineseSimpleVQA, aimed at assessing the visual factuality of LVLMs across 8 major topics and 56 subtopics. The key features of this benchmark include a focus on the Chinese language, diverse knowledge types, a multi-hop question construction, high-quality data, static consistency, and easy-to-evaluate through short answers. Moreover, we contribute a rigorous data construction pipeline and decouple the visual factuality into two parts: seeing the world (i.e., object recognition) and discovering knowledge. This decoupling allows us to analyze the capability boundaries and execution mechanisms of LVLMs. Subsequently, we evaluate 34 advanced open-source and closed-source models, revealing critical performance gaps within this field. Our evaluation-friendly code and data have already been open-sourced. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_11718 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | "See the World, Discover Knowledge": A Chinese Factuality Evaluation for Large Vision Language Models Gu, Jihao Wang, Yingyao Bu, Pi Wang, Chen Wang, Ziming Song, Tengtao Wei, Donglai Yuan, Jiale Zhao, Yingxiu He, Yancheng Li, Shilong Liu, Jiaheng Cao, Meng Song, Jun Tan, Yingshui Li, Xiang Su, Wenbo Zheng, Zhicheng Zhu, Xiaoyong Zheng, Bo Computation and Language Computer Vision and Pattern Recognition The evaluation of factual accuracy in large vision language models (LVLMs) has lagged behind their rapid development, making it challenging to fully reflect these models' knowledge capacity and reliability. In this paper, we introduce the first factuality-based visual question-answering benchmark in Chinese, named ChineseSimpleVQA, aimed at assessing the visual factuality of LVLMs across 8 major topics and 56 subtopics. The key features of this benchmark include a focus on the Chinese language, diverse knowledge types, a multi-hop question construction, high-quality data, static consistency, and easy-to-evaluate through short answers. Moreover, we contribute a rigorous data construction pipeline and decouple the visual factuality into two parts: seeing the world (i.e., object recognition) and discovering knowledge. This decoupling allows us to analyze the capability boundaries and execution mechanisms of LVLMs. Subsequently, we evaluate 34 advanced open-source and closed-source models, revealing critical performance gaps within this field. Our evaluation-friendly code and data have already been open-sourced. |
| title | "See the World, Discover Knowledge": A Chinese Factuality Evaluation for Large Vision Language Models |
| topic | Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2502.11718 |