Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.15028 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915401222324224 |
|---|---|
| author | Zhang, Yuanhan Chew, Yunice Dong, Yuhao Leo, Aria Hu, Bo Liu, Ziwei |
| author_facet | Zhang, Yuanhan Chew, Yunice Dong, Yuhao Leo, Aria Hu, Bo Liu, Ziwei |
| contents | Human intelligence requires correctness and robustness, with the former being foundational for the latter. In video understanding, correctness ensures the accurate interpretation of visual content, and robustness maintains consistent performance in challenging conditions. Despite advances in video large language models (video LLMs), existing benchmarks inadequately reflect the gap between these models and human intelligence in maintaining correctness and robustness in video interpretation. We introduce the Video Thinking Test (Video-TT), to assess if video LLMs can interpret real-world videos as effectively as humans. Video-TT reflects genuine gaps in understanding complex visual narratives, and evaluates robustness against natural adversarial questions. Video-TT comprises 1,000 YouTube Shorts videos, each with one open-ended question and four adversarial questions that probe visual and narrative complexity. Our evaluation shows a significant gap between video LLMs and human performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_15028 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Video Thinking Test: A Holistic Benchmark for Advanced Video Reasoning and Understanding Zhang, Yuanhan Chew, Yunice Dong, Yuhao Leo, Aria Hu, Bo Liu, Ziwei Computer Vision and Pattern Recognition Human intelligence requires correctness and robustness, with the former being foundational for the latter. In video understanding, correctness ensures the accurate interpretation of visual content, and robustness maintains consistent performance in challenging conditions. Despite advances in video large language models (video LLMs), existing benchmarks inadequately reflect the gap between these models and human intelligence in maintaining correctness and robustness in video interpretation. We introduce the Video Thinking Test (Video-TT), to assess if video LLMs can interpret real-world videos as effectively as humans. Video-TT reflects genuine gaps in understanding complex visual narratives, and evaluates robustness against natural adversarial questions. Video-TT comprises 1,000 YouTube Shorts videos, each with one open-ended question and four adversarial questions that probe visual and narrative complexity. Our evaluation shows a significant gap between video LLMs and human performance. |
| title | Towards Video Thinking Test: A Holistic Benchmark for Advanced Video Reasoning and Understanding |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2507.15028 |