What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929314760491008 |
|---|---|
| author | Zhang, Letian Zhai, Xiaotong Zhao, Zhongkai Zong, Yongshuo Wen, Xin Zhao, Bingchen |
| author_facet | Zhang, Letian Zhai, Xiaotong Zhao, Zhongkai Zong, Yongshuo Wen, Xin Zhao, Bingchen |
| contents | Counterfactual reasoning, a fundamental aspect of human cognition, involves contemplating alternatives to established facts or past events, significantly enhancing our abilities in planning and decision-making. In light of the advancements in current multi-modal large language models, we explore their effectiveness in counterfactual reasoning. To facilitate this investigation, we introduce a novel dataset, C-VQA, specifically designed to test the counterfactual reasoning capabilities of modern multi-modal large language models. This dataset is constructed by infusing original questions with counterfactual presuppositions, spanning various types such as numerical and boolean queries. It encompasses a mix of real and synthetic data, representing a wide range of difficulty levels. Our thorough evaluations of contemporary vision-language models using this dataset have revealed substantial performance drops, with some models showing up to a 40% decrease, highlighting a significant gap between current models and human-like vision reasoning capabilities. We hope our dataset will serve as a vital benchmark for evaluating the counterfactual reasoning capabilities of models. Code and dataset are publicly available at https://bzhao.me/C-VQA/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_06627 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models Zhang, Letian Zhai, Xiaotong Zhao, Zhongkai Zong, Yongshuo Wen, Xin Zhao, Bingchen Computation and Language Computer Vision and Pattern Recognition Machine Learning Counterfactual reasoning, a fundamental aspect of human cognition, involves contemplating alternatives to established facts or past events, significantly enhancing our abilities in planning and decision-making. In light of the advancements in current multi-modal large language models, we explore their effectiveness in counterfactual reasoning. To facilitate this investigation, we introduce a novel dataset, C-VQA, specifically designed to test the counterfactual reasoning capabilities of modern multi-modal large language models. This dataset is constructed by infusing original questions with counterfactual presuppositions, spanning various types such as numerical and boolean queries. It encompasses a mix of real and synthetic data, representing a wide range of difficulty levels. Our thorough evaluations of contemporary vision-language models using this dataset have revealed substantial performance drops, with some models showing up to a 40% decrease, highlighting a significant gap between current models and human-like vision reasoning capabilities. We hope our dataset will serve as a vital benchmark for evaluating the counterfactual reasoning capabilities of models. Code and dataset are publicly available at https://bzhao.me/C-VQA/. |
| title | What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models |
| topic | Computation and Language Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2310.06627 |