Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2506.09943 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916790748053504 |
|---|---|
| author | Foss, Aaron Evans, Chloe Mitts, Sasha Sinha, Koustuv Rizvi, Ammar Kao, Justine T. |
| author_facet | Foss, Aaron Evans, Chloe Mitts, Sasha Sinha, Koustuv Rizvi, Ammar Kao, Justine T. |
| contents | We introduce CausalVQA, a benchmark dataset for video question answering (VQA) composed of question-answer pairs that probe models' understanding of causality in the physical world. Existing VQA benchmarks either tend to focus on surface perceptual understanding of real-world videos, or on narrow physical reasoning questions created using simulation environments. CausalVQA fills an important gap by presenting challenging questions that are grounded in real-world scenarios, while focusing on models' ability to predict the likely outcomes of different actions and events through five question types: counterfactual, hypothetical, anticipation, planning and descriptive. We designed quality control mechanisms that prevent models from exploiting trivial shortcuts, requiring models to base their answers on deep visual understanding instead of linguistic cues. We find that current frontier multimodal models fall substantially below human performance on the benchmark, especially on anticipation and hypothetical questions. This highlights a challenge for current systems to leverage spatial-temporal reasoning, understanding of physical principles, and comprehension of possible alternatives to make accurate predictions in real-world settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_09943 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Models Foss, Aaron Evans, Chloe Mitts, Sasha Sinha, Koustuv Rizvi, Ammar Kao, Justine T. Computer Vision and Pattern Recognition Artificial Intelligence I.2.10; I.4.8 We introduce CausalVQA, a benchmark dataset for video question answering (VQA) composed of question-answer pairs that probe models' understanding of causality in the physical world. Existing VQA benchmarks either tend to focus on surface perceptual understanding of real-world videos, or on narrow physical reasoning questions created using simulation environments. CausalVQA fills an important gap by presenting challenging questions that are grounded in real-world scenarios, while focusing on models' ability to predict the likely outcomes of different actions and events through five question types: counterfactual, hypothetical, anticipation, planning and descriptive. We designed quality control mechanisms that prevent models from exploiting trivial shortcuts, requiring models to base their answers on deep visual understanding instead of linguistic cues. We find that current frontier multimodal models fall substantially below human performance on the benchmark, especially on anticipation and hypothetical questions. This highlights a challenge for current systems to leverage spatial-temporal reasoning, understanding of physical principles, and comprehension of possible alternatives to make accurate predictions in real-world settings. |
| title | CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence I.2.10; I.4.8 |
| url | https://arxiv.org/abs/2506.09943 |