Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.20912 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916032342392832 |
|---|---|
| author | Xu, Tianrun Jing, Haoda Li, Ye Wei, Yuquan Feng, Jun Chen, Guanyu Gao, Haichuan Zhang, Tianren Chen, Feng |
| author_facet | Xu, Tianrun Jing, Haoda Li, Ye Wei, Yuquan Feng, Jun Chen, Guanyu Gao, Haichuan Zhang, Tianren Chen, Feng |
| contents | Recent advances in multimodal language models (MLLMs) have made thinking with images a dominant paradigm for multimodal reasoning. However, existing methods still fail to ensure evidence-answer consistency, where correct answers must be supported by correct visual evidence. To address this issue, we propose DeFacto, a counterfactual reasoning framework that explicitly aligns visual evidence with final answers. Our approach integrates three complementary training paradigms: positive, counterfactual, and random-masking. We further develop a language-guided evidence construction pipeline that automatically localizes question-relevant regions and generates counterfactual variants, resulting in DeFacto-100K. Building on this dataset, we train MLLMs with GRPO-based reinforcement learning and design three complementary rewards to promote correct answering, structured reasoning, and consistent evidence selection. Moreover, we introduce DeFacto-1.5K, a human-annotated benchmark for systematically evaluating evidence-grounded consistency beyond answer accuracy. Experiments on diverse benchmarks demonstrate that DeFacto substantially improves both answer accuracy and evidence-answer consistency over strong baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_20912 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning Xu, Tianrun Jing, Haoda Li, Ye Wei, Yuquan Feng, Jun Chen, Guanyu Gao, Haichuan Zhang, Tianren Chen, Feng Artificial Intelligence Recent advances in multimodal language models (MLLMs) have made thinking with images a dominant paradigm for multimodal reasoning. However, existing methods still fail to ensure evidence-answer consistency, where correct answers must be supported by correct visual evidence. To address this issue, we propose DeFacto, a counterfactual reasoning framework that explicitly aligns visual evidence with final answers. Our approach integrates three complementary training paradigms: positive, counterfactual, and random-masking. We further develop a language-guided evidence construction pipeline that automatically localizes question-relevant regions and generates counterfactual variants, resulting in DeFacto-100K. Building on this dataset, we train MLLMs with GRPO-based reinforcement learning and design three complementary rewards to promote correct answering, structured reasoning, and consistent evidence selection. Moreover, we introduce DeFacto-1.5K, a human-annotated benchmark for systematically evaluating evidence-grounded consistency beyond answer accuracy. Experiments on diverse benchmarks demonstrate that DeFacto substantially improves both answer accuracy and evidence-answer consistency over strong baselines. |
| title | DeFacto: Counterfactual Thinking with Images for Enforcing Evidence-Grounded and Faithful Reasoning |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2509.20912 |