Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910251149688832 |
|---|---|
| author | Ashraf, Tajamul Saqib, Amal Ghani, Hanan AlMahri, Muhra Li, Yuhao Ahsan, Noor Nawaz, Umair Lahoud, Jean Cholakkal, Hisham Shah, Mubarak Torr, Philip Khan, Fahad Shahbaz Anwer, Rao Muhammad Khan, Salman |
| author_facet | Ashraf, Tajamul Saqib, Amal Ghani, Hanan AlMahri, Muhra Li, Yuhao Ahsan, Noor Nawaz, Umair Lahoud, Jean Cholakkal, Hisham Shah, Mubarak Torr, Philip Khan, Fahad Shahbaz Anwer, Rao Muhammad Khan, Salman |
| contents | Deep reasoning is fundamental for solving complex tasks, especially in vision-centric scenarios that demand sequential, multimodal understanding. However, existing benchmarks typically evaluate agents with fully synthetic, single-turn queries, limited visual modalities, and lack a framework to assess reasoning quality over multiple steps as required in real-world settings. To address this, we introduce Agent-X, a large-scale benchmark for evaluating vision-centric agents multi-step and deep reasoning capabilities in real-world, multimodal settings. Agent- X features 828 agentic tasks with authentic visual contexts, including images, multi-image comparisons, videos, and instructional text. These tasks span six major agentic environments: general visual reasoning, web browsing, security and surveillance, autonomous driving, sports, and math reasoning. Our benchmark requires agents to integrate tool use with explicit, stepwise decision-making in these diverse settings. In addition, we propose a fine-grained, step-level evaluation framework that assesses the correctness and logical coherence of each reasoning step and the effectiveness of tool usage throughout the task. Our results reveal that even the best-performing models, including GPT, Gemini, and Qwen families, struggle to solve multi-step vision tasks, achieving less than 50% full-chain success. These findings highlight key bottlenecks in current LMM reasoning and tool-use capabilities and identify future research directions in vision-centric agentic reasoning models. Our data and code are publicly available at https://github.com/mbzuai-oryx/Agent-X |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_24876 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks Ashraf, Tajamul Saqib, Amal Ghani, Hanan AlMahri, Muhra Li, Yuhao Ahsan, Noor Nawaz, Umair Lahoud, Jean Cholakkal, Hisham Shah, Mubarak Torr, Philip Khan, Fahad Shahbaz Anwer, Rao Muhammad Khan, Salman Computer Vision and Pattern Recognition Computation and Language Deep reasoning is fundamental for solving complex tasks, especially in vision-centric scenarios that demand sequential, multimodal understanding. However, existing benchmarks typically evaluate agents with fully synthetic, single-turn queries, limited visual modalities, and lack a framework to assess reasoning quality over multiple steps as required in real-world settings. To address this, we introduce Agent-X, a large-scale benchmark for evaluating vision-centric agents multi-step and deep reasoning capabilities in real-world, multimodal settings. Agent- X features 828 agentic tasks with authentic visual contexts, including images, multi-image comparisons, videos, and instructional text. These tasks span six major agentic environments: general visual reasoning, web browsing, security and surveillance, autonomous driving, sports, and math reasoning. Our benchmark requires agents to integrate tool use with explicit, stepwise decision-making in these diverse settings. In addition, we propose a fine-grained, step-level evaluation framework that assesses the correctness and logical coherence of each reasoning step and the effectiveness of tool usage throughout the task. Our results reveal that even the best-performing models, including GPT, Gemini, and Qwen families, struggle to solve multi-step vision tasks, achieving less than 50% full-chain success. These findings highlight key bottlenecks in current LMM reasoning and tool-use capabilities and identify future research directions in vision-centric agentic reasoning models. Our data and code are publicly available at https://github.com/mbzuai-oryx/Agent-X |
| title | Agent-X: Evaluating Deep Multimodal Reasoning in Vision-Centric Agentic Tasks |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2505.24876 |