Chart-HQA: A Benchmark for Hypothetical Question Answering in Charts
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916645617795072 |
|---|---|
| author | Chen, Xiangnan Fang, Yuancheng Xiao, Qian Li, Juncheng Lin, Jun Tang, Siliang Yang, Yi Zhuang, Yueting |
| author_facet | Chen, Xiangnan Fang, Yuancheng Xiao, Qian Li, Juncheng Lin, Jun Tang, Siliang Yang, Yi Zhuang, Yueting |
| contents | Multimodal Large Language Models (MLLMs) have garnered significant attention for their strong visual-semantic understanding. Most existing chart benchmarks evaluate MLLMs' ability to parse information from charts to answer questions. However, they overlook the inherent output biases of MLLMs, where models rely on their parametric memory to answer questions rather than genuinely understanding the chart content. To address this limitation, we introduce a novel Chart Hypothetical Question Answering (HQA) task, which imposes assumptions on the same question to compel models to engage in counterfactual reasoning based on the chart content. Furthermore, we introduce HAI, a human-AI interactive data synthesis approach that leverages the efficient text-editing capabilities of LLMs alongside human expert knowledge to generate diverse and high-quality HQA data at a low cost. Using HAI, we construct Chart-HQA, a challenging benchmark synthesized from publicly available data sources. Evaluation results on 18 MLLMs of varying model sizes reveal that current models face significant generalization challenges and exhibit imbalanced reasoning performance on the HQA task. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_04095 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Chart-HQA: A Benchmark for Hypothetical Question Answering in Charts Chen, Xiangnan Fang, Yuancheng Xiao, Qian Li, Juncheng Lin, Jun Tang, Siliang Yang, Yi Zhuang, Yueting Computation and Language Artificial Intelligence Multimodal Large Language Models (MLLMs) have garnered significant attention for their strong visual-semantic understanding. Most existing chart benchmarks evaluate MLLMs' ability to parse information from charts to answer questions. However, they overlook the inherent output biases of MLLMs, where models rely on their parametric memory to answer questions rather than genuinely understanding the chart content. To address this limitation, we introduce a novel Chart Hypothetical Question Answering (HQA) task, which imposes assumptions on the same question to compel models to engage in counterfactual reasoning based on the chart content. Furthermore, we introduce HAI, a human-AI interactive data synthesis approach that leverages the efficient text-editing capabilities of LLMs alongside human expert knowledge to generate diverse and high-quality HQA data at a low cost. Using HAI, we construct Chart-HQA, a challenging benchmark synthesized from publicly available data sources. Evaluation results on 18 MLLMs of varying model sizes reveal that current models face significant generalization challenges and exhibit imbalanced reasoning performance on the HQA task. |
| title | Chart-HQA: A Benchmark for Hypothetical Question Answering in Charts |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.04095 |