Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2503.10079 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929757650681856 |
|---|---|
| author | Li, Chunyi Li, Xiaozhe Zhang, Zicheng Tian, Yuan Jia, Ziheng Liu, Xiaohong Min, Xiongkuo Wang, Jia Duan, Haodong Chen, Kai Zhai, Guangtao |
| author_facet | Li, Chunyi Li, Xiaozhe Zhang, Zicheng Tian, Yuan Jia, Ziheng Liu, Xiaohong Min, Xiongkuo Wang, Jia Duan, Haodong Chen, Kai Zhai, Guangtao |
| contents | With the emergence of Multimodal Large Language Models (MLLMs), hundreds of benchmarks have been developed to ensure the reliability of MLLMs in downstream tasks. However, the evaluation mechanism itself may not be reliable. For developers of MLLMs, questions remain about which benchmark to use and whether the test results meet their requirements. Therefore, we propose a critical principle of Information Density, which examines how much insight a benchmark can provide for the development of MLLMs. We characterize it from four key dimensions: (1) Fallacy, (2) Difficulty, (3) Redundancy, (4) Diversity. Through a comprehensive analysis of more than 10,000 samples, we measured the information density of 19 MLLM benchmarks. Experiments show that using the latest benchmarks in testing can provide more insight compared to previous ones, but there is still room for improvement in their information density. We hope this principle can promote the development and application of future MLLM benchmarks. Project page: https://github.com/lcysyzxdxc/bench4bench |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_10079 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Information Density Principle for MLLM Benchmarks Li, Chunyi Li, Xiaozhe Zhang, Zicheng Tian, Yuan Jia, Ziheng Liu, Xiaohong Min, Xiongkuo Wang, Jia Duan, Haodong Chen, Kai Zhai, Guangtao Computation and Language With the emergence of Multimodal Large Language Models (MLLMs), hundreds of benchmarks have been developed to ensure the reliability of MLLMs in downstream tasks. However, the evaluation mechanism itself may not be reliable. For developers of MLLMs, questions remain about which benchmark to use and whether the test results meet their requirements. Therefore, we propose a critical principle of Information Density, which examines how much insight a benchmark can provide for the development of MLLMs. We characterize it from four key dimensions: (1) Fallacy, (2) Difficulty, (3) Redundancy, (4) Diversity. Through a comprehensive analysis of more than 10,000 samples, we measured the information density of 19 MLLM benchmarks. Experiments show that using the latest benchmarks in testing can provide more insight compared to previous ones, but there is still room for improvement in their information density. We hope this principle can promote the development and application of future MLLM benchmarks. Project page: https://github.com/lcysyzxdxc/bench4bench |
| title | Information Density Principle for MLLM Benchmarks |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.10079 |