ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911438863335424 |
|---|---|
| author | Tang, Liyan Kim, Grace Zhao, Xinyu Lake, Thom Ding, Wenxuan Yin, Fangcong Singhal, Prasann Wadhwa, Manya Liu, Zeyu Leo Sprague, Zayne Namuduri, Ramya Hu, Bodun Rodriguez, Juan Diego Peng, Puyuan Durrett, Greg |
| author_facet | Tang, Liyan Kim, Grace Zhao, Xinyu Lake, Thom Ding, Wenxuan Yin, Fangcong Singhal, Prasann Wadhwa, Manya Liu, Zeyu Leo Sprague, Zayne Namuduri, Ramya Hu, Bodun Rodriguez, Juan Diego Peng, Puyuan Durrett, Greg |
| contents | Chart understanding presents a unique challenge for large vision-language models (LVLMs), as it requires the integration of sophisticated textual and visual reasoning capabilities. However, current LVLMs exhibit a notable imbalance between these skills, falling short on visual reasoning that is difficult to perform in text. We conduct a case study using a synthetic dataset solvable only through visual reasoning and show that model performance degrades significantly with increasing visual complexity, while human performance remains robust. We then introduce ChartMuseum, a new Chart Question Answering (QA) benchmark containing 1,162 expert-annotated questions spanning multiple reasoning types, curated from real-world charts across 184 sources, specifically built to evaluate complex visual and textual reasoning. Unlike prior chart understanding benchmarks -- where frontier models perform similarly and near saturation -- our benchmark exposes a substantial gap between model and human performance, while effectively differentiating model capabilities: although humans achieve 93% accuracy, the best-performing model Gemini-2.5-Pro attains only 63.0%, and the leading open-source LVLM Qwen2.5-VL-72B-Instruct achieves only 38.5%. Moreover, on questions requiring primarily visual reasoning, all models experience a 35%-55% performance drop from text-reasoning-heavy question performance. Lastly, our qualitative error analysis reveals specific categories of visual reasoning that are challenging for current LVLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_13444 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models Tang, Liyan Kim, Grace Zhao, Xinyu Lake, Thom Ding, Wenxuan Yin, Fangcong Singhal, Prasann Wadhwa, Manya Liu, Zeyu Leo Sprague, Zayne Namuduri, Ramya Hu, Bodun Rodriguez, Juan Diego Peng, Puyuan Durrett, Greg Computation and Language Computer Vision and Pattern Recognition Chart understanding presents a unique challenge for large vision-language models (LVLMs), as it requires the integration of sophisticated textual and visual reasoning capabilities. However, current LVLMs exhibit a notable imbalance between these skills, falling short on visual reasoning that is difficult to perform in text. We conduct a case study using a synthetic dataset solvable only through visual reasoning and show that model performance degrades significantly with increasing visual complexity, while human performance remains robust. We then introduce ChartMuseum, a new Chart Question Answering (QA) benchmark containing 1,162 expert-annotated questions spanning multiple reasoning types, curated from real-world charts across 184 sources, specifically built to evaluate complex visual and textual reasoning. Unlike prior chart understanding benchmarks -- where frontier models perform similarly and near saturation -- our benchmark exposes a substantial gap between model and human performance, while effectively differentiating model capabilities: although humans achieve 93% accuracy, the best-performing model Gemini-2.5-Pro attains only 63.0%, and the leading open-source LVLM Qwen2.5-VL-72B-Instruct achieves only 38.5%. Moreover, on questions requiring primarily visual reasoning, all models experience a 35%-55% performance drop from text-reasoning-heavy question performance. Lastly, our qualitative error analysis reveals specific categories of visual reasoning that are challenging for current LVLMs. |
| title | ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models |
| topic | Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.13444 |