RBench-V: A Primary Assessment for Visual Reasoning Models with Multi-modal Outputs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916754138071040 |
|---|---|
| author | Guo, Meng-Hao Chu, Xuanyu Yang, Qianrui Mo, Zhe-Han Shen, Yiqing Li, Pei-lin Lin, Xinjie Zhang, Jinnian Chen, Xin-Sheng Zhang, Yi Nakayama, Kiyohiro Geng, Zhengyang Peng, Houwen Hu, Han Hu, Shi-Min |
| author_facet | Guo, Meng-Hao Chu, Xuanyu Yang, Qianrui Mo, Zhe-Han Shen, Yiqing Li, Pei-lin Lin, Xinjie Zhang, Jinnian Chen, Xin-Sheng Zhang, Yi Nakayama, Kiyohiro Geng, Zhengyang Peng, Houwen Hu, Han Hu, Shi-Min |
| contents | The rapid advancement of native multi-modal models and omni-models, exemplified by GPT-4o, Gemini, and o3, with their capability to process and generate content across modalities such as text and images, marks a significant milestone in the evolution of intelligence. Systematic evaluation of their multi-modal output capabilities in visual thinking processes (also known as multi-modal chain of thought, M-CoT) becomes critically important. However, existing benchmarks for evaluating multi-modal models primarily focus on assessing multi-modal inputs and text-only reasoning while neglecting the importance of reasoning through multi-modal outputs. In this paper, we present a benchmark, dubbed RBench-V, designed to assess models' vision-indispensable reasoning abilities. To construct RBench-V, we carefully hand-pick 803 questions covering math, physics, counting, and games. Unlike previous benchmarks that typically specify certain input modalities, RBench-V presents problems centered on multi-modal outputs, which require image manipulation such as generating novel images and constructing auxiliary lines to support the reasoning process. We evaluate numerous open- and closed-source models on RBench-V, including o3, Gemini 2.5 Pro, Qwen2.5-VL, etc. Even the best-performing model, o3, achieves only 25.8% accuracy on RBench-V, far below the human score of 82.3%, highlighting that current models struggle to leverage multi-modal reasoning. Data and code are available at https://evalmodels.github.io/rbenchv |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_16770 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RBench-V: A Primary Assessment for Visual Reasoning Models with Multi-modal Outputs Guo, Meng-Hao Chu, Xuanyu Yang, Qianrui Mo, Zhe-Han Shen, Yiqing Li, Pei-lin Lin, Xinjie Zhang, Jinnian Chen, Xin-Sheng Zhang, Yi Nakayama, Kiyohiro Geng, Zhengyang Peng, Houwen Hu, Han Hu, Shi-Min Computer Vision and Pattern Recognition The rapid advancement of native multi-modal models and omni-models, exemplified by GPT-4o, Gemini, and o3, with their capability to process and generate content across modalities such as text and images, marks a significant milestone in the evolution of intelligence. Systematic evaluation of their multi-modal output capabilities in visual thinking processes (also known as multi-modal chain of thought, M-CoT) becomes critically important. However, existing benchmarks for evaluating multi-modal models primarily focus on assessing multi-modal inputs and text-only reasoning while neglecting the importance of reasoning through multi-modal outputs. In this paper, we present a benchmark, dubbed RBench-V, designed to assess models' vision-indispensable reasoning abilities. To construct RBench-V, we carefully hand-pick 803 questions covering math, physics, counting, and games. Unlike previous benchmarks that typically specify certain input modalities, RBench-V presents problems centered on multi-modal outputs, which require image manipulation such as generating novel images and constructing auxiliary lines to support the reasoning process. We evaluate numerous open- and closed-source models on RBench-V, including o3, Gemini 2.5 Pro, Qwen2.5-VL, etc. Even the best-performing model, o3, achieves only 25.8% accuracy on RBench-V, far below the human score of 82.3%, highlighting that current models struggle to leverage multi-modal reasoning. Data and code are available at https://evalmodels.github.io/rbenchv |
| title | RBench-V: A Primary Assessment for Visual Reasoning Models with Multi-modal Outputs |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.16770 |