MULTI: Multimodal Understanding Leaderboard with Text and Images
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911211347509248 |
|---|---|
| author | Zhu, Zichen Xu, Yang Chen, Lu Yang, Jingkai Ma, Yichuan Sun, Yiming Wen, Hailin Liu, Jiaqi Cai, Jinyu Ma, Yingzi Zhang, Situo Zhao, Zihan Sun, Liangtai Yu, Kai |
| author_facet | Zhu, Zichen Xu, Yang Chen, Lu Yang, Jingkai Ma, Yichuan Sun, Yiming Wen, Hailin Liu, Jiaqi Cai, Jinyu Ma, Yingzi Zhang, Situo Zhao, Zihan Sun, Liangtai Yu, Kai |
| contents | The rapid development of multimodal large language models (MLLMs) raises the question of how they compare to human performance. While existing datasets often feature synthetic or overly simplistic tasks, some models have already surpassed human expert baselines. In this paper, we present MULTI, a Chinese multimodal dataset derived from authentic examination questions. Comprising over 18,000 carefully selected and refined questions, MULTI evaluates models using real-world examination standards, encompassing image-text comprehension, complex reasoning, and knowledge recall. Additionally, We also introduce MULTI-Elite, a 500-question selected hard subset, and MULTI-Extend with more than 4,500 external knowledge context pieces for testing in-context learning capabilities. Our evaluation highlights substantial room for MLLM advancement, with Qwen2-VL-72B achieving a 76.9% accuracy on MULTI and 53.1% on MULTI-Elite leading 25 evaluated models, compared to human expert baselines of 86.1% and 73.1%. MULTI serves not only as a robust evaluation platform but also paves the way for the development of expert-level AI. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_03173 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MULTI: Multimodal Understanding Leaderboard with Text and Images Zhu, Zichen Xu, Yang Chen, Lu Yang, Jingkai Ma, Yichuan Sun, Yiming Wen, Hailin Liu, Jiaqi Cai, Jinyu Ma, Yingzi Zhang, Situo Zhao, Zihan Sun, Liangtai Yu, Kai Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition The rapid development of multimodal large language models (MLLMs) raises the question of how they compare to human performance. While existing datasets often feature synthetic or overly simplistic tasks, some models have already surpassed human expert baselines. In this paper, we present MULTI, a Chinese multimodal dataset derived from authentic examination questions. Comprising over 18,000 carefully selected and refined questions, MULTI evaluates models using real-world examination standards, encompassing image-text comprehension, complex reasoning, and knowledge recall. Additionally, We also introduce MULTI-Elite, a 500-question selected hard subset, and MULTI-Extend with more than 4,500 external knowledge context pieces for testing in-context learning capabilities. Our evaluation highlights substantial room for MLLM advancement, with Qwen2-VL-72B achieving a 76.9% accuracy on MULTI and 53.1% on MULTI-Elite leading 25 evaluated models, compared to human expert baselines of 86.1% and 73.1%. MULTI serves not only as a robust evaluation platform but also paves the way for the development of expert-level AI. |
| title | MULTI: Multimodal Understanding Leaderboard with Text and Images |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2402.03173 |