DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911376314728448 |
|---|---|
| author | Wang, Yibo Wang, Lei Deng, Yue Wu, Keming Xiao, Yao Yao, Huanjin Kang, Liwei Ye, Hai Jing, Yongcheng Bing, Lidong |
| author_facet | Wang, Yibo Wang, Lei Deng, Yue Wu, Keming Xiao, Yao Yao, Huanjin Kang, Liwei Ye, Hai Jing, Yongcheng Bing, Lidong |
| contents | Deep research systems are widely used for multi-step web research, analysis, and cross-source synthesis, yet their evaluation remains challenging. Existing benchmarks often require annotation-intensive task construction, rely on static evaluation dimensions, or fail to reliably verify facts when citations are missing. To bridge these gaps, we introduce DeepResearchEval, an automated framework for deep research task construction and agentic evaluation. For task construction, we propose a persona-driven pipeline generating realistic, complex research tasks anchored in diverse user profiles, applying a two-stage filter Task Qualification and Search Necessity to retain only tasks requiring multi-source evidence integration and external retrieval. For evaluation, we propose an agentic pipeline with two components: an Adaptive Point-wise Quality Evaluation that dynamically derives task-specific evaluation dimensions, criteria, and weights conditioned on each generated task, and an Active Fact-Checking that autonomously extracts and verifies report statements via web search, even when citations are missing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_09688 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluation Wang, Yibo Wang, Lei Deng, Yue Wu, Keming Xiao, Yao Yao, Huanjin Kang, Liwei Ye, Hai Jing, Yongcheng Bing, Lidong Computation and Language Deep research systems are widely used for multi-step web research, analysis, and cross-source synthesis, yet their evaluation remains challenging. Existing benchmarks often require annotation-intensive task construction, rely on static evaluation dimensions, or fail to reliably verify facts when citations are missing. To bridge these gaps, we introduce DeepResearchEval, an automated framework for deep research task construction and agentic evaluation. For task construction, we propose a persona-driven pipeline generating realistic, complex research tasks anchored in diverse user profiles, applying a two-stage filter Task Qualification and Search Necessity to retain only tasks requiring multi-source evidence integration and external retrieval. For evaluation, we propose an agentic pipeline with two components: an Adaptive Point-wise Quality Evaluation that dynamically derives task-specific evaluation dimensions, criteria, and weights conditioned on each generated task, and an Active Fact-Checking that autonomously extracts and verifies report statements via web search, even when citations are missing. |
| title | DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.09688 |