Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.09672 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916880399204352 |
|---|---|
| author | Roger, Alexis Humane, Prateek Kaplan, Daniel Z. Gupta, Kshitij Sun, Qi Adamopoulos, George Lim, Jonathan Siu Chi Anthony, Quentin Fennell, Edwin Rish, Irina |
| author_facet | Roger, Alexis Humane, Prateek Kaplan, Daniel Z. Gupta, Kshitij Sun, Qi Adamopoulos, George Lim, Jonathan Siu Chi Anthony, Quentin Fennell, Edwin Rish, Irina |
| contents | The proliferation of Vision-Language Models (VLMs) in the past several years calls for rigorous and comprehensive evaluation methods and benchmarks. This work analyzes existing VLM evaluation techniques, including automated metrics, AI-based assessments, and human evaluations across diverse tasks. We first introduce Robin - a novel suite of VLMs that we built by combining Large Language Models (LLMs) and Vision Encoders (VEs) at multiple scales, and use Robin to identify shortcomings of current evaluation approaches across scales. Next, to overcome the identified limitations, we introduce CHIRP - a new long form response benchmark we developed for more robust and complete VLM evaluation. We provide open access to the Robin training code, model suite, and CHIRP benchmark to promote reproducibility and advance VLM research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_09672 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CHIRP: A Fine-Grained Benchmark for Open-Ended Response Evaluation in Vision-Language Models Roger, Alexis Humane, Prateek Kaplan, Daniel Z. Gupta, Kshitij Sun, Qi Adamopoulos, George Lim, Jonathan Siu Chi Anthony, Quentin Fennell, Edwin Rish, Irina Computer Vision and Pattern Recognition Artificial Intelligence The proliferation of Vision-Language Models (VLMs) in the past several years calls for rigorous and comprehensive evaluation methods and benchmarks. This work analyzes existing VLM evaluation techniques, including automated metrics, AI-based assessments, and human evaluations across diverse tasks. We first introduce Robin - a novel suite of VLMs that we built by combining Large Language Models (LLMs) and Vision Encoders (VEs) at multiple scales, and use Robin to identify shortcomings of current evaluation approaches across scales. Next, to overcome the identified limitations, we introduce CHIRP - a new long form response benchmark we developed for more robust and complete VLM evaluation. We provide open access to the Robin training code, model suite, and CHIRP benchmark to promote reproducibility and advance VLM research. |
| title | CHIRP: A Fine-Grained Benchmark for Open-Ended Response Evaluation in Vision-Language Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2501.09672 |