Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.02779 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914135832264704 |
|---|---|
| author | Zhou, Yiyang Tu, Haoqin Wang, Zijun Wang, Zeyu Muennighoff, Niklas Nie, Fan Choi, Yejin Zou, James Deng, Chaorui Yan, Shen Fan, Haoqi Xie, Cihang Yao, Huaxiu Ye, Qinghao |
| author_facet | Zhou, Yiyang Tu, Haoqin Wang, Zijun Wang, Zeyu Muennighoff, Niklas Nie, Fan Choi, Yejin Zou, James Deng, Chaorui Yan, Shen Fan, Haoqi Xie, Cihang Yao, Huaxiu Ye, Qinghao |
| contents | We propose MIRA, a new benchmark designed to evaluate models in scenarios where generating intermediate visual images is essential for successful reasoning. Unlike traditional CoT methods that rely solely on text, tasks in MIRA require models to generate and utilize intermediate images - such as sketches, structural diagrams, or path drawings - to guide their reasoning process. This setup closely mirrors how humans solve complex problems through "drawing to think". To solve this, MIRA focuses on tasks that are intrinsically challenging and involve complex structures, spatial relationships, or reasoning steps that are difficult to express through language alone. To ensure that our evaluation data is of high-quality, we include 546 multimodal problems, annotated with intermediate visual images and final answers. We also propose a unified evaluation protocol for MIRA that spans three levels of evaluation input: direct input with image and question only, text-only CoT input with image and thinking prompts, and Visual-CoT input with both annotated image clues and textual thinking prompts. To probe the upper bound of model capacity on our benchmark, we also report pass@k and majority voting accuracies under different k settings. Experimental results show that existing multimodal large language models, including strongest private models as well as strong open-weight models, perform poorly when relying solely on textual prompts. However, when intermediate visual cues are provided, model performance improves consistently, yielding an average relative gain of 33.7% across all models and tasks. We also probe the upper bound by expanding the search space and designing textual prompts aligned with Visual-CoT, but both yield only limited improvements compared to our Visual-CoT setting. These results underscore the critical role of imagined visual information in enabling successful reasoning on MIRA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_02779 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought Zhou, Yiyang Tu, Haoqin Wang, Zijun Wang, Zeyu Muennighoff, Niklas Nie, Fan Choi, Yejin Zou, James Deng, Chaorui Yan, Shen Fan, Haoqi Xie, Cihang Yao, Huaxiu Ye, Qinghao Computer Vision and Pattern Recognition We propose MIRA, a new benchmark designed to evaluate models in scenarios where generating intermediate visual images is essential for successful reasoning. Unlike traditional CoT methods that rely solely on text, tasks in MIRA require models to generate and utilize intermediate images - such as sketches, structural diagrams, or path drawings - to guide their reasoning process. This setup closely mirrors how humans solve complex problems through "drawing to think". To solve this, MIRA focuses on tasks that are intrinsically challenging and involve complex structures, spatial relationships, or reasoning steps that are difficult to express through language alone. To ensure that our evaluation data is of high-quality, we include 546 multimodal problems, annotated with intermediate visual images and final answers. We also propose a unified evaluation protocol for MIRA that spans three levels of evaluation input: direct input with image and question only, text-only CoT input with image and thinking prompts, and Visual-CoT input with both annotated image clues and textual thinking prompts. To probe the upper bound of model capacity on our benchmark, we also report pass@k and majority voting accuracies under different k settings. Experimental results show that existing multimodal large language models, including strongest private models as well as strong open-weight models, perform poorly when relying solely on textual prompts. However, when intermediate visual cues are provided, model performance improves consistently, yielding an average relative gain of 33.7% across all models and tasks. We also probe the upper bound by expanding the search space and designing textual prompts aligned with Visual-CoT, but both yield only limited improvements compared to our Visual-CoT setting. These results underscore the critical role of imagined visual information in enabling successful reasoning on MIRA. |
| title | When Visualizing is the First Step to Reasoning: MIRA, a Benchmark for Visual Chain-of-Thought |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2511.02779 |