Bridging vision language model (VLM) evaluation gaps with a framework for scalable and cost-effective benchmark generation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910839028580352 |
|---|---|
| author | Rädsch, Tim Mayer, Leon Pavicic, Simon Kavur, A. Emre Knopp, Marcel Öztürk, Barış Maier-Hein, Klaus Jaeger, Paul F. Isensee, Fabian Reinke, Annika Maier-Hein, Lena |
| author_facet | Rädsch, Tim Mayer, Leon Pavicic, Simon Kavur, A. Emre Knopp, Marcel Öztürk, Barış Maier-Hein, Klaus Jaeger, Paul F. Isensee, Fabian Reinke, Annika Maier-Hein, Lena |
| contents | Reliable evaluation of AI models is critical for scientific progress and practical application. While existing VLM benchmarks provide general insights into model capabilities, their heterogeneous designs and limited focus on a few imaging domains pose significant challenges for both cross-domain performance comparison and targeted domain-specific evaluation. To address this, we propose three key contributions: (1) a framework for the resource-efficient creation of domain-specific VLM benchmarks enabled by task augmentation for creating multiple diverse tasks from a single existing task, (2) the release of new VLM benchmarks for seven domains, created according to the same homogeneous protocol and including 162,946 thoroughly human-validated answers, and (3) an extensive benchmarking of 22 state-of-the-art VLMs on a total of 37,171 tasks, revealing performance variances across domains and tasks, thereby supporting the need for tailored VLM benchmarks. Adoption of our methodology will pave the way for the resource-efficient domain-specific selection of models and guide future research efforts toward addressing core open questions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_15563 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Bridging vision language model (VLM) evaluation gaps with a framework for scalable and cost-effective benchmark generation Rädsch, Tim Mayer, Leon Pavicic, Simon Kavur, A. Emre Knopp, Marcel Öztürk, Barış Maier-Hein, Klaus Jaeger, Paul F. Isensee, Fabian Reinke, Annika Maier-Hein, Lena Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning Reliable evaluation of AI models is critical for scientific progress and practical application. While existing VLM benchmarks provide general insights into model capabilities, their heterogeneous designs and limited focus on a few imaging domains pose significant challenges for both cross-domain performance comparison and targeted domain-specific evaluation. To address this, we propose three key contributions: (1) a framework for the resource-efficient creation of domain-specific VLM benchmarks enabled by task augmentation for creating multiple diverse tasks from a single existing task, (2) the release of new VLM benchmarks for seven domains, created according to the same homogeneous protocol and including 162,946 thoroughly human-validated answers, and (3) an extensive benchmarking of 22 state-of-the-art VLMs on a total of 37,171 tasks, revealing performance variances across domains and tasks, thereby supporting the need for tailored VLM benchmarks. Adoption of our methodology will pave the way for the resource-efficient domain-specific selection of models and guide future research efforts toward addressing core open questions. |
| title | Bridging vision language model (VLM) evaluation gaps with a framework for scalable and cost-effective benchmark generation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2502.15563 |