Guardado en:
| Autores principales: | , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2603.12180 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918399700893696 |
|---|---|
| author | Borchmann, Łukasz Van Landeghem, Jordy Turski, Michał Padarha, Shreyansh Kearns, Ryan Othniel Mahdi, Adam Rogge, Niels Fourrier, Clémentine Han, Siwei Yao, Huaxiu Llabrés, Artemis Xu, Yiming Karatzas, Dimosthenis Zhang, Hao Datta, Anupam |
| author_facet | Borchmann, Łukasz Van Landeghem, Jordy Turski, Michał Padarha, Shreyansh Kearns, Ryan Othniel Mahdi, Adam Rogge, Niels Fourrier, Clémentine Han, Siwei Yao, Huaxiu Llabrés, Artemis Xu, Yiming Karatzas, Dimosthenis Zhang, Hao Datta, Anupam |
| contents | Multimodal agents offer a promising path to automating complex document-intensive workflows. Yet, a critical question remains: do these agents demonstrate genuine strategic reasoning, or merely stochastic trial-and-error search? To address this, we introduce MADQA, a benchmark of 2,250 human-authored questions grounded in 800 heterogeneous PDF documents. Guided by Classical Test Theory, we design it to maximize discriminative power across varying levels of agentic abilities. To evaluate agentic behaviour, we introduce a novel evaluation protocol measuring the accuracy-effort trade-off. Using this framework, we show that while the best agents can match human searchers in raw accuracy, they succeed on largely different questions and rely on brute-force search to compensate for weak strategic planning. They fail to close the nearly 20% gap to oracle performance, persisting in unproductive loops. We release the dataset and evaluation harness to help facilitate the transition from brute-force retrieval to calibrated, efficient reasoning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_12180 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections Borchmann, Łukasz Van Landeghem, Jordy Turski, Michał Padarha, Shreyansh Kearns, Ryan Othniel Mahdi, Adam Rogge, Niels Fourrier, Clémentine Han, Siwei Yao, Huaxiu Llabrés, Artemis Xu, Yiming Karatzas, Dimosthenis Zhang, Hao Datta, Anupam Computation and Language Artificial Intelligence Multimodal agents offer a promising path to automating complex document-intensive workflows. Yet, a critical question remains: do these agents demonstrate genuine strategic reasoning, or merely stochastic trial-and-error search? To address this, we introduce MADQA, a benchmark of 2,250 human-authored questions grounded in 800 heterogeneous PDF documents. Guided by Classical Test Theory, we design it to maximize discriminative power across varying levels of agentic abilities. To evaluate agentic behaviour, we introduce a novel evaluation protocol measuring the accuracy-effort trade-off. Using this framework, we show that while the best agents can match human searchers in raw accuracy, they succeed on largely different questions and rely on brute-force search to compensate for weak strategic planning. They fail to close the nearly 20% gap to oracle performance, persisting in unproductive loops. We release the dataset and evaluation harness to help facilitate the transition from brute-force retrieval to calibrated, efficient reasoning. |
| title | Strategic Navigation or Stochastic Search? How Agents and Humans Reason Over Document Collections |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2603.12180 |