GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929551285682176 |
|---|---|
| author | Chen, Pengcheng Ye, Jin Wang, Guoan Li, Yanjun Deng, Zhongying Li, Wei Li, Tianbin Duan, Haodong Huang, Ziyan Su, Yanzhou Wang, Benyou Zhang, Shaoting Fu, Bin Cai, Jianfei Zhuang, Bohan Seibel, Eric J He, Junjun Qiao, Yu |
| author_facet | Chen, Pengcheng Ye, Jin Wang, Guoan Li, Yanjun Deng, Zhongying Li, Wei Li, Tianbin Duan, Haodong Huang, Ziyan Su, Yanzhou Wang, Benyou Zhang, Shaoting Fu, Bin Cai, Jianfei Zhuang, Bohan Seibel, Eric J He, Junjun Qiao, Yu |
| contents | Large Vision-Language Models (LVLMs) are capable of handling diverse data types such as imaging, text, and physiological signals, and can be applied in various fields. In the medical field, LVLMs have a high potential to offer substantial assistance for diagnosis and treatment. Before that, it is crucial to develop benchmarks to evaluate LVLMs' effectiveness in various medical applications. Current benchmarks are often built upon specific academic literature, mainly focusing on a single domain, and lacking varying perceptual granularities. Thus, they face specific challenges, including limited clinical relevance, incomplete evaluations, and insufficient guidance for interactive LVLMs. To address these limitations, we developed the GMAI-MMBench, the most comprehensive general medical AI benchmark with well-categorized data structure and multi-perceptual granularity to date. It is constructed from 284 datasets across 38 medical image modalities, 18 clinical-related tasks, 18 departments, and 4 perceptual granularities in a Visual Question Answering (VQA) format. Additionally, we implemented a lexical tree structure that allows users to customize evaluation tasks, accommodating various assessment needs and substantially supporting medical AI research and applications. We evaluated 50 LVLMs, and the results show that even the advanced GPT-4o only achieves an accuracy of 53.96%, indicating significant room for improvement. Moreover, we identified five key insufficiencies in current cutting-edge LVLMs that need to be addressed to advance the development of better medical applications. We believe that GMAI-MMBench will stimulate the community to build the next generation of LVLMs toward GMAI. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_03361 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI Chen, Pengcheng Ye, Jin Wang, Guoan Li, Yanjun Deng, Zhongying Li, Wei Li, Tianbin Duan, Haodong Huang, Ziyan Su, Yanzhou Wang, Benyou Zhang, Shaoting Fu, Bin Cai, Jianfei Zhuang, Bohan Seibel, Eric J He, Junjun Qiao, Yu Image and Video Processing Computer Vision and Pattern Recognition Large Vision-Language Models (LVLMs) are capable of handling diverse data types such as imaging, text, and physiological signals, and can be applied in various fields. In the medical field, LVLMs have a high potential to offer substantial assistance for diagnosis and treatment. Before that, it is crucial to develop benchmarks to evaluate LVLMs' effectiveness in various medical applications. Current benchmarks are often built upon specific academic literature, mainly focusing on a single domain, and lacking varying perceptual granularities. Thus, they face specific challenges, including limited clinical relevance, incomplete evaluations, and insufficient guidance for interactive LVLMs. To address these limitations, we developed the GMAI-MMBench, the most comprehensive general medical AI benchmark with well-categorized data structure and multi-perceptual granularity to date. It is constructed from 284 datasets across 38 medical image modalities, 18 clinical-related tasks, 18 departments, and 4 perceptual granularities in a Visual Question Answering (VQA) format. Additionally, we implemented a lexical tree structure that allows users to customize evaluation tasks, accommodating various assessment needs and substantially supporting medical AI research and applications. We evaluated 50 LVLMs, and the results show that even the advanced GPT-4o only achieves an accuracy of 53.96%, indicating significant room for improvement. Moreover, we identified five key insufficiencies in current cutting-edge LVLMs that need to be addressed to advance the development of better medical applications. We believe that GMAI-MMBench will stimulate the community to build the next generation of LVLMs toward GMAI. |
| title | GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI |
| topic | Image and Video Processing Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.03361 |