JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909453446545408 |
|---|---|
| author | Wang, Zhecan Liu, Junzhang Tang, Chia-Wei Alomari, Hani Sivakumar, Anushka Sun, Rui Li, Wenhao Atabuzzaman, Md. Ayyubi, Hammad You, Haoxuan Ishmam, Alvi Chang, Kai-Wei Chang, Shih-Fu Thomas, Chris |
| author_facet | Wang, Zhecan Liu, Junzhang Tang, Chia-Wei Alomari, Hani Sivakumar, Anushka Sun, Rui Li, Wenhao Atabuzzaman, Md. Ayyubi, Hammad You, Haoxuan Ishmam, Alvi Chang, Kai-Wei Chang, Shih-Fu Thomas, Chris |
| contents | Existing vision-language understanding benchmarks largely consist of images of objects in their usual contexts. As a consequence, recent multimodal large language models can perform well with only a shallow visual understanding by relying on background language biases. Thus, strong performance on these benchmarks does not necessarily correlate with strong visual understanding. In this paper, we release JourneyBench, a comprehensive human-annotated benchmark of generated images designed to assess the model's fine-grained multimodal reasoning abilities across five tasks: complementary multimodal chain of thought, multi-image VQA, imaginary image captioning, VQA with hallucination triggers, and fine-grained retrieval with sample-specific distractors. Unlike existing benchmarks, JourneyBench explicitly requires fine-grained multimodal reasoning in unusual imaginary scenarios where language bias and holistic image gist are insufficient. We benchmark state-of-the-art models on JourneyBench and analyze performance along a number of fine-grained dimensions. Results across all five tasks show that JourneyBench is exceptionally challenging for even the best models, indicating that models' visual reasoning abilities are not as strong as they first appear. We discuss the implications of our findings and propose avenues for further research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_12953 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images Wang, Zhecan Liu, Junzhang Tang, Chia-Wei Alomari, Hani Sivakumar, Anushka Sun, Rui Li, Wenhao Atabuzzaman, Md. Ayyubi, Hammad You, Haoxuan Ishmam, Alvi Chang, Kai-Wei Chang, Shih-Fu Thomas, Chris Computer Vision and Pattern Recognition Artificial Intelligence Existing vision-language understanding benchmarks largely consist of images of objects in their usual contexts. As a consequence, recent multimodal large language models can perform well with only a shallow visual understanding by relying on background language biases. Thus, strong performance on these benchmarks does not necessarily correlate with strong visual understanding. In this paper, we release JourneyBench, a comprehensive human-annotated benchmark of generated images designed to assess the model's fine-grained multimodal reasoning abilities across five tasks: complementary multimodal chain of thought, multi-image VQA, imaginary image captioning, VQA with hallucination triggers, and fine-grained retrieval with sample-specific distractors. Unlike existing benchmarks, JourneyBench explicitly requires fine-grained multimodal reasoning in unusual imaginary scenarios where language bias and holistic image gist are insufficient. We benchmark state-of-the-art models on JourneyBench and analyze performance along a number of fine-grained dimensions. Results across all five tasks show that JourneyBench is exceptionally challenging for even the best models, indicating that models' visual reasoning abilities are not as strong as they first appear. We discuss the implications of our findings and propose avenues for further research. |
| title | JourneyBench: A Challenging One-Stop Vision-Language Understanding Benchmark of Generated Images |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2409.12953 |