How Far Are We from Genuinely Useful Deep Research Agents?
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914201447956480 |
|---|---|
| author | Zhang, Dingling Zhu, He Ren, Jincheng Song, Kangqi Zhou, Xinran Feng, Boyu Liu, Shudong Luo, Jiabin Xie, Weihao Wang, Zhaohui Qin, Tianrui Zhu, King Wang, Yuqing Chen, Qianben Jiang, Yuchen Eleanor Wang, Wei Liu, Jiaheng Zhou, Wangchunshu |
| author_facet | Zhang, Dingling Zhu, He Ren, Jincheng Song, Kangqi Zhou, Xinran Feng, Boyu Liu, Shudong Luo, Jiabin Xie, Weihao Wang, Zhaohui Qin, Tianrui Zhu, King Wang, Yuqing Chen, Qianben Jiang, Yuchen Eleanor Wang, Wei Liu, Jiaheng Zhou, Wangchunshu |
| contents | Deep Research Agents (DRAs) aim to automatically produce analyst-level reports through iterative information retrieval and synthesis. However, most existing DRAs were validated on question-answering benchmarks, while research on generating comprehensive reports remains overlooked. Worse, current benchmarks for report synthesis suffer from task complexity and subjective metrics -- this fails to reflect user demands and limits the practical utility of generated reports. To address these gaps, we present Fine-grained DEepResearch bench (FINDER), an enhanced benchmark consisting of 100 human-curated research tasks with 419 structured checklist items that standardize report structure, analytical depth, and factual grounding. Based on approximately 1,000 reports produced by mainstream DRAs, we further propose Deep rEsearch Failure Taxonomy (DEFT), the first failure taxonomy for deep research agents. DEFT contains 14 fine-grained failure modes across reasoning, retrieval, and generation, and is built upon grounded theory with human-LLM co-annotating and inter-annotator reliability validation. Our experimental findings reveal that current DRAs struggle not with task comprehension but with evidence integration, verification, and reasoning-resilient planning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_01948 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | How Far Are We from Genuinely Useful Deep Research Agents? Zhang, Dingling Zhu, He Ren, Jincheng Song, Kangqi Zhou, Xinran Feng, Boyu Liu, Shudong Luo, Jiabin Xie, Weihao Wang, Zhaohui Qin, Tianrui Zhu, King Wang, Yuqing Chen, Qianben Jiang, Yuchen Eleanor Wang, Wei Liu, Jiaheng Zhou, Wangchunshu Computation and Language Deep Research Agents (DRAs) aim to automatically produce analyst-level reports through iterative information retrieval and synthesis. However, most existing DRAs were validated on question-answering benchmarks, while research on generating comprehensive reports remains overlooked. Worse, current benchmarks for report synthesis suffer from task complexity and subjective metrics -- this fails to reflect user demands and limits the practical utility of generated reports. To address these gaps, we present Fine-grained DEepResearch bench (FINDER), an enhanced benchmark consisting of 100 human-curated research tasks with 419 structured checklist items that standardize report structure, analytical depth, and factual grounding. Based on approximately 1,000 reports produced by mainstream DRAs, we further propose Deep rEsearch Failure Taxonomy (DEFT), the first failure taxonomy for deep research agents. DEFT contains 14 fine-grained failure modes across reasoning, retrieval, and generation, and is built upon grounded theory with human-LLM co-annotating and inter-annotator reliability validation. Our experimental findings reveal that current DRAs struggle not with task comprehension but with evidence integration, verification, and reasoning-resilient planning. |
| title | How Far Are We from Genuinely Useful Deep Research Agents? |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2512.01948 |