Learning More from Less: Exploiting Counterfactuals for Data-Efficient Chart Understanding
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909033497100288 |
|---|---|
| author | Bao, Jianzhu Zhang, Haozhen Dong, Kuicai Wu, Bozhi Modi, Sarthak Ketanbhai Lim, Zi Pong Teo, Yon Shin Wang, Wenya |
| author_facet | Bao, Jianzhu Zhang, Haozhen Dong, Kuicai Wu, Bozhi Modi, Sarthak Ketanbhai Lim, Zi Pong Teo, Yon Shin Wang, Wenya |
| contents | Vision-Language Models (VLMs) have demonstrated remarkable progress in chart understanding, largely driven by supervised fine-tuning (SFT) on increasingly large synthetic datasets. However, scaling SFT data alone is inefficient and overlooks a key property of charts: charts are programmatically generated visual artifacts, where small, code-controlled visual changes can induce drastic shifts in semantics and correct answers. Learning this counterfactual sensitivity requires VLMs to discriminate fine-grained visual differences, yet standard SFT treats training instances independently and provides limited supervision to enforce this behavior. To address this, we introduce ChartCF, a data-efficient training framework designed to enhance counterfactual sensitivity. ChartCF consists of: (1) a counterfactual data synthesis pipeline via code modification, (2) a chart similarity-based data selection strategy that filters overly difficult samples for improved training efficiency, and (3) multimodal preference optimization across both textual and visual modalities. Experiments on five benchmarks show that ChartCF achieves superior or comparable performance to strong chart-specific VLMs while using significantly less training data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_10855 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Learning More from Less: Exploiting Counterfactuals for Data-Efficient Chart Understanding Bao, Jianzhu Zhang, Haozhen Dong, Kuicai Wu, Bozhi Modi, Sarthak Ketanbhai Lim, Zi Pong Teo, Yon Shin Wang, Wenya Computation and Language Vision-Language Models (VLMs) have demonstrated remarkable progress in chart understanding, largely driven by supervised fine-tuning (SFT) on increasingly large synthetic datasets. However, scaling SFT data alone is inefficient and overlooks a key property of charts: charts are programmatically generated visual artifacts, where small, code-controlled visual changes can induce drastic shifts in semantics and correct answers. Learning this counterfactual sensitivity requires VLMs to discriminate fine-grained visual differences, yet standard SFT treats training instances independently and provides limited supervision to enforce this behavior. To address this, we introduce ChartCF, a data-efficient training framework designed to enhance counterfactual sensitivity. ChartCF consists of: (1) a counterfactual data synthesis pipeline via code modification, (2) a chart similarity-based data selection strategy that filters overly difficult samples for improved training efficiency, and (3) multimodal preference optimization across both textual and visual modalities. Experiments on five benchmarks show that ChartCF achieves superior or comparable performance to strong chart-specific VLMs while using significantly less training data. |
| title | Learning More from Less: Exploiting Counterfactuals for Data-Efficient Chart Understanding |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.10855 |