DatBench: Discriminative, Faithful, and Efficient VLM Evaluations
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914254363295744 |
|---|---|
| author | DatologyAI : Joshi, Siddharth Yin, Haoli Adiga, Rishabh Monti, Ricardo Carranza, Aldo Fang, Alex Deng, Alvin Abbas, Amro Larsen, Brett Blakeney, Cody Teh, Darren Schwab, David Pan, Fan Mongstad, Haakon Urbanek, Jack Lee, Jason Telanoff, Jason Wills, Josh Mentzer, Kaleigh Merrick, Luke Doshi, Parth Burstein, Paul Maini, Pratyush Loftin, Scott Das, Spandan Jiang, Tony Dorna, Vineeth Wang, Zhengping Gaza, Bogdan Morcos, Ari Leavitt, Matthew |
| author_facet | DatologyAI : Joshi, Siddharth Yin, Haoli Adiga, Rishabh Monti, Ricardo Carranza, Aldo Fang, Alex Deng, Alvin Abbas, Amro Larsen, Brett Blakeney, Cody Teh, Darren Schwab, David Pan, Fan Mongstad, Haakon Urbanek, Jack Lee, Jason Telanoff, Jason Wills, Josh Mentzer, Kaleigh Merrick, Luke Doshi, Parth Burstein, Paul Maini, Pratyush Loftin, Scott Das, Spandan Jiang, Tony Dorna, Vineeth Wang, Zhengping Gaza, Bogdan Morcos, Ari Leavitt, Matthew |
| contents | Empirical evaluation serves as the primary compass guiding research progress in foundation models. Despite a large body of work focused on training frontier vision-language models (VLMs), approaches to their evaluation remain nascent. To guide their maturation, we propose three desiderata that evaluations should satisfy: (1) faithfulness to the modality and application, (2) discriminability between models of varying quality, and (3) efficiency in compute. Through this lens, we identify critical failure modes that violate faithfulness and discriminability, misrepresenting model capabilities: (i) multiple-choice formats reward guessing, poorly reflect downstream use cases, and saturate early as models improve; (ii) blindly solvable questions, which can be answered without images, constitute up to 70% of some evaluations; and (iii) mislabeled or ambiguous samples compromise up to 42% of examples in certain datasets. Regarding efficiency, the computational burden of evaluating frontier models has become prohibitive: by some accounts, nearly 20% of development compute is devoted to evaluation alone. Rather than discarding existing benchmarks, we curate them via transformation and filtering to maximize fidelity and discriminability. We find that converting multiple-choice questions to generative tasks reveals sharp capability drops of up to 35%. In addition, filtering blindly solvable and mislabeled samples improves discriminative power while simultaneously reducing computational cost. We release DatBench-Full, a cleaned evaluation suite of 33 datasets spanning nine VLM capabilities, and DatBench, a discriminative subset that achieves 13x average speedup (up to 50x) while closely matching the discriminative power of the original datasets. Our work outlines a path toward evaluation practices that are both rigorous and sustainable as VLMs continue to scale. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_02316 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | DatBench: Discriminative, Faithful, and Efficient VLM Evaluations DatologyAI : Joshi, Siddharth Yin, Haoli Adiga, Rishabh Monti, Ricardo Carranza, Aldo Fang, Alex Deng, Alvin Abbas, Amro Larsen, Brett Blakeney, Cody Teh, Darren Schwab, David Pan, Fan Mongstad, Haakon Urbanek, Jack Lee, Jason Telanoff, Jason Wills, Josh Mentzer, Kaleigh Merrick, Luke Doshi, Parth Burstein, Paul Maini, Pratyush Loftin, Scott Das, Spandan Jiang, Tony Dorna, Vineeth Wang, Zhengping Gaza, Bogdan Morcos, Ari Leavitt, Matthew Machine Learning Artificial Intelligence Empirical evaluation serves as the primary compass guiding research progress in foundation models. Despite a large body of work focused on training frontier vision-language models (VLMs), approaches to their evaluation remain nascent. To guide their maturation, we propose three desiderata that evaluations should satisfy: (1) faithfulness to the modality and application, (2) discriminability between models of varying quality, and (3) efficiency in compute. Through this lens, we identify critical failure modes that violate faithfulness and discriminability, misrepresenting model capabilities: (i) multiple-choice formats reward guessing, poorly reflect downstream use cases, and saturate early as models improve; (ii) blindly solvable questions, which can be answered without images, constitute up to 70% of some evaluations; and (iii) mislabeled or ambiguous samples compromise up to 42% of examples in certain datasets. Regarding efficiency, the computational burden of evaluating frontier models has become prohibitive: by some accounts, nearly 20% of development compute is devoted to evaluation alone. Rather than discarding existing benchmarks, we curate them via transformation and filtering to maximize fidelity and discriminability. We find that converting multiple-choice questions to generative tasks reveals sharp capability drops of up to 35%. In addition, filtering blindly solvable and mislabeled samples improves discriminative power while simultaneously reducing computational cost. We release DatBench-Full, a cleaned evaluation suite of 33 datasets spanning nine VLM capabilities, and DatBench, a discriminative subset that achieves 13x average speedup (up to 50x) while closely matching the discriminative power of the original datasets. Our work outlines a path toward evaluation practices that are both rigorous and sustainable as VLMs continue to scale. |
| title | DatBench: Discriminative, Faithful, and Efficient VLM Evaluations |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2601.02316 |