Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.09907 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912801377746944 |
|---|---|
| author | Zhang, Daoan Liu, Pai Zhou, Xiaofei Ge, Yuan Lan, Guangchen Bi, Jing Brinton, Christopher Hoque, Ehsan Luo, Jiebo |
| author_facet | Zhang, Daoan Liu, Pai Zhou, Xiaofei Ge, Yuan Lan, Guangchen Bi, Jing Brinton, Christopher Hoque, Ehsan Luo, Jiebo |
| contents | Vision-Language Models (VLMs) have achieved impressive progress in perceiving and describing visual environments. However, their ability to proactively reason and act based solely on visual inputs, without explicit textual prompts, remains underexplored. We introduce a new task, Visual Action Reasoning, and propose VisualActBench, a large-scale benchmark comprising 1,074 videos and 3,733 human-annotated actions across four real-world scenarios. Each action is labeled with an Action Prioritization Level (APL) and a proactive-reactive type to assess models' human-aligned reasoning and value sensitivity. We evaluate 29 VLMs on VisualActBench and find that while frontier models like GPT4o demonstrate relatively strong performance, a significant gap remains compared to human-level reasoning, particularly in generating proactive, high-priority actions. Our results highlight limitations in current VLMs' ability to interpret complex context, anticipate outcomes, and align with human decision-making frameworks. VisualActBench establishes a comprehensive foundation for assessing and improving the real-world readiness of proactive, vision-centric AI agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_09907 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VisualActBench: Can VLMs See and Act like a Human? Zhang, Daoan Liu, Pai Zhou, Xiaofei Ge, Yuan Lan, Guangchen Bi, Jing Brinton, Christopher Hoque, Ehsan Luo, Jiebo Computer Vision and Pattern Recognition Vision-Language Models (VLMs) have achieved impressive progress in perceiving and describing visual environments. However, their ability to proactively reason and act based solely on visual inputs, without explicit textual prompts, remains underexplored. We introduce a new task, Visual Action Reasoning, and propose VisualActBench, a large-scale benchmark comprising 1,074 videos and 3,733 human-annotated actions across four real-world scenarios. Each action is labeled with an Action Prioritization Level (APL) and a proactive-reactive type to assess models' human-aligned reasoning and value sensitivity. We evaluate 29 VLMs on VisualActBench and find that while frontier models like GPT4o demonstrate relatively strong performance, a significant gap remains compared to human-level reasoning, particularly in generating proactive, high-priority actions. Our results highlight limitations in current VLMs' ability to interpret complex context, anticipate outcomes, and align with human decision-making frameworks. VisualActBench establishes a comprehensive foundation for assessing and improving the real-world readiness of proactive, vision-centric AI agents. |
| title | VisualActBench: Can VLMs See and Act like a Human? |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2512.09907 |