Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.02190 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915760985604096 |
|---|---|
| author | Yao, Yang Wang, Yixu Zhang, Yuxuan Lu, Yi Gu, Tianle Li, Lingyu Zhao, Dingyi Wu, Keming Wang, Haozhe Nie, Ping Teng, Yan Wang, Yingchun |
| author_facet | Yao, Yang Wang, Yixu Zhang, Yuxuan Lu, Yi Gu, Tianle Li, Lingyu Zhao, Dingyi Wu, Keming Wang, Haozhe Nie, Ping Teng, Yan Wang, Yingchun |
| contents | As an embodiment of intelligence evolution toward interconnected architectures, Deep Research Agents (DRAs) systematically exhibit the capabilities in task decomposition, cross-source retrieval, multi-stage reasoning, information integration, and structured output, which markedly enhance performance on complex and open-ended tasks. However, existing benchmarks remain deficient in evaluation dimensions, response format, and scoring mechanisms, limiting their effectiveness in assessing such agents. This paper introduces Dr. Bench, a multidimensional evaluation framework tailored to DRAs and long-form report-style responses. The benchmark comprises 214 expert-curated challenging tasks across 10 broad domains, each accompanied by manually constructed reference bundles to support composite evaluation. This framework incorporates metrics for semantic quality, topical focus, and retrieval trustworthiness, enabling a comprehensive evaluation of long reports generated by DRAs. Extensive experimentation confirms the superior performance of mainstream DRAs over web-search-tool-augmented reasoning models, yet reveals considerable scope for further improvement. This study provides a robust foundation for capability assessment, architectural refinement, and paradigm advancement of DRAs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_02190 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Dr. Bench: A Multidimensional Evaluation for Deep Research Agents, from Answers to Reports Yao, Yang Wang, Yixu Zhang, Yuxuan Lu, Yi Gu, Tianle Li, Lingyu Zhao, Dingyi Wu, Keming Wang, Haozhe Nie, Ping Teng, Yan Wang, Yingchun Artificial Intelligence Computation and Language As an embodiment of intelligence evolution toward interconnected architectures, Deep Research Agents (DRAs) systematically exhibit the capabilities in task decomposition, cross-source retrieval, multi-stage reasoning, information integration, and structured output, which markedly enhance performance on complex and open-ended tasks. However, existing benchmarks remain deficient in evaluation dimensions, response format, and scoring mechanisms, limiting their effectiveness in assessing such agents. This paper introduces Dr. Bench, a multidimensional evaluation framework tailored to DRAs and long-form report-style responses. The benchmark comprises 214 expert-curated challenging tasks across 10 broad domains, each accompanied by manually constructed reference bundles to support composite evaluation. This framework incorporates metrics for semantic quality, topical focus, and retrieval trustworthiness, enabling a comprehensive evaluation of long reports generated by DRAs. Extensive experimentation confirms the superior performance of mainstream DRAs over web-search-tool-augmented reasoning models, yet reveals considerable scope for further improvement. This study provides a robust foundation for capability assessment, architectural refinement, and paradigm advancement of DRAs. |
| title | Dr. Bench: A Multidimensional Evaluation for Deep Research Agents, from Answers to Reports |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2510.02190 |