MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911053186596864 |
|---|---|
| author | Chen, Jiacheng Liang, Tianhao Siu, Sherman Wang, Zhengqing Wang, Kai Wang, Yubo Ni, Yuansheng Zhu, Wang Jiang, Ziyan Lyu, Bohan Jiang, Dongfu He, Xuan Liu, Yuan Hu, Hexiang Yue, Xiang Chen, Wenhu |
| author_facet | Chen, Jiacheng Liang, Tianhao Siu, Sherman Wang, Zhengqing Wang, Kai Wang, Yubo Ni, Yuansheng Zhu, Wang Jiang, Ziyan Lyu, Bohan Jiang, Dongfu He, Xuan Liu, Yuan Hu, Hexiang Yue, Xiang Chen, Wenhu |
| contents | We present MEGA-Bench, an evaluation suite that scales multimodal evaluation to over 500 real-world tasks, to address the highly heterogeneous daily use cases of end users. Our objective is to optimize for a set of high-quality data samples that cover a highly diverse and rich set of multimodal tasks, while enabling cost-effective and accurate model evaluation. In particular, we collected 505 realistic tasks encompassing over 8,000 samples from 16 expert annotators to extensively cover the multimodal task space. Instead of unifying these problems into standard multi-choice questions (like MMMU, MMBench, and MMT-Bench), we embrace a wide range of output formats like numbers, phrases, code, \LaTeX, coordinates, JSON, free-form, etc. To accommodate these formats, we developed over 40 metrics to evaluate these tasks. Unlike existing benchmarks, MEGA-Bench offers a fine-grained capability report across multiple dimensions (e.g., application, input type, output format, skill), allowing users to interact with and visualize model capabilities in depth. We evaluate a wide variety of frontier vision-language models on MEGA-Bench to understand their capabilities across these dimensions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_10563 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks Chen, Jiacheng Liang, Tianhao Siu, Sherman Wang, Zhengqing Wang, Kai Wang, Yubo Ni, Yuansheng Zhu, Wang Jiang, Ziyan Lyu, Bohan Jiang, Dongfu He, Xuan Liu, Yuan Hu, Hexiang Yue, Xiang Chen, Wenhu Computer Vision and Pattern Recognition We present MEGA-Bench, an evaluation suite that scales multimodal evaluation to over 500 real-world tasks, to address the highly heterogeneous daily use cases of end users. Our objective is to optimize for a set of high-quality data samples that cover a highly diverse and rich set of multimodal tasks, while enabling cost-effective and accurate model evaluation. In particular, we collected 505 realistic tasks encompassing over 8,000 samples from 16 expert annotators to extensively cover the multimodal task space. Instead of unifying these problems into standard multi-choice questions (like MMMU, MMBench, and MMT-Bench), we embrace a wide range of output formats like numbers, phrases, code, \LaTeX, coordinates, JSON, free-form, etc. To accommodate these formats, we developed over 40 metrics to evaluate these tasks. Unlike existing benchmarks, MEGA-Bench offers a fine-grained capability report across multiple dimensions (e.g., application, input type, output format, skill), allowing users to interact with and visualize model capabilities in depth. We evaluate a wide variety of frontier vision-language models on MEGA-Bench to understand their capabilities across these dimensions. |
| title | MEGA-Bench: Scaling Multimodal Evaluation to over 500 Real-World Tasks |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.10563 |